ESP32_Serial_Swiss_Army_Knife/docs/https_idle_cleanup.md

Bounded ordinary HTTPS idle retention

Implemented 2026-09-08 on the existing 8D.11 key/diagnostic implementation. Host-tested and firmware-build verified. Subsequent scoped user report: the user accepted that idle cleanup worked. This supersedes the original target-pending status for that behavior only; no additional soak duration, individual checklist passes, memory reserve approval or broader 8D.11/M3 sign-off is inferred. The implementation evidence below remains historical and unchanged. No agent upload, erase, commit, SDK patch or generated-asset change.

Evidence and scope

User-authorized response to the admission capture: post-TLS occupancy 6/6, ordinary4 / serial2 / admin0, ordinary connection ages 50–74 seconds; admin ticket returned in 14 ms, no subsequent TLS/upgrade observed, ticket unconsumed. These ages measure connection lifetime, not time since the last request. They support investigating retained ordinary sockets, not claiming every captured socket was idle. Installed HTTPD excludes its listening fd from select while full with LRU disabled. New connections can therefore wait before TLS or upgrade even when ticket issuance was fast.

This policy releases expired idle ordinary HTTPS connections, not arbitrary ordinary requests to make room. Six sockets, 24 URI handlers, LRU disabled, existing receive/send timeouts (one second), TLS handshake timeout (five seconds), all task stacks/queue depths and browser code remain unchanged. No new task. UART0 recovery, independent USB access, serial broker ownership, tickets, authentication and both WebSocket transports are unchanged.

Usage and timeout semantics

• Automatically enabled for every successfully started HTTPS server, including when diagnostic tracing is disabled and either optional transport is unavailable. No new console command, setting or credential migration.
• 15 seconds of observed ordinary idle retention, checked by a one-second ESP timer that requests work on HTTPD. The three existing five-second browser status-poll intervals leave room for normal keepalive reuse instead of a TLS handshake per request.
• A new post-TLS connection that has sent no request gets the same full idle window. Time spent establishing TLS does not consume it.
• The first owner sweep observing a new connection or a changed successful-request completion marker starts a fresh window. The marker covers every ordinary route, including login, assets, status, typed settings, tickets and keepalive errors that return successfully—not just diagnostic wrappers. Failed requests are deleted by the normal SDK path instead.
• HTTPD pending bytes, TLS pending bytes or a readable TCP fd reset the observation window. Negative TLS pending results and select errors conservatively reset it too. No bytes are read or discarded by the probe.
• At the threshold, after current SDK classification and zero-time readiness checks, HTTPD calls shutdown(current_fd, SHUT_RDWR). A successful shutdown is latched; a failed call retries on the next probe. The SDK's subsequent read/delete path owns socket close, TLS destruction, diagnostic close notification and freeing the slot. The probe never calls close, overrides TLS cleanup, or queues a session-close pointer.
• WebSockets (serial and admin), closing WebSockets and async requests are exempt. The check uses actual SDK flags, not diagnostic metadata. Successful explicit 101/classification and request cleanup finish before the owner can sweep, so there is no ordinary-idle interval during upgrade admission.
• Under an available owner and timely successful work delivery, expect 15 seconds plus the initial observation delay (nominally up to one second), the next probe delay (nominally up to one second), and SDK read/cleanup latency. This is a conservative sampled idle policy, not a strict wall-clock timeout or admission SLA.

Deliberate limitations

1. Parsing an incomplete request, synchronous response sends, leftover-body purge and TLS handshakes serialize on HTTPD. The probe cannot interrupt them. Existing per-read/per-send timeouts remain; a peer trickling input can extend overall processing beyond one timeout. This change does not provide a slowloris deadline or solve TLS/memory/global-socket pressure.
2. Four continuously active ordinary connections plus two WebSockets can still fill all six slots. They are not evicted. Likewise, an admission attempt immediately after fresh saturation can hit the browser's existing 15-second timeout before the conservative window expires; the user may still need to retry. Older idle retained sockets are eligible on the next delivered probe.
3. As with any HTTP keepalive timeout, bytes arriving after the last readiness check can race a shutdown. Already executing HTTPD requests/responses are protected; future client intent cannot be predicted. A client may need a new connection. No new application-level mutation retry/replay is added. Response completion here means synchronous HTTPD completion, not proof of peer receipt/TCP acknowledgement.
4. httpd_queue_work with CONFIG_HTTPD_QUEUE_WORK_BLOCKING disabled uses loopback UDP. A reported queue error releases the reservation and retries next tick. A successful send is not an execution acknowledgement: an accepted-but-lost control message leaves one reservation pending, disables further probes, and requires successful HTTPS stop/restart to restore probing. There is no speculative reservation timeout: it could accumulate delayed callbacks and violate the one-probe bound. This is explicitly regression-tested, not hidden behind a hard retention guarantee. HTTPD blockage or queue loss also cannot be repaired by raising sockets/LRU/timeouts here.
5. A failed stop leaves cleanup detached and ownership retained for a later stop retry; it does not restart probes on a partially stopped server. Timer allocation/start failure gates HTTPS start with its error rather than silently starting without the policy. UART0/USB recovery is unaffected.

Exact installed SDK audit

All SDK references below are under ~/.platformio/packages/framework-espidf/components/; installed framework is ESP-IDF 5.5.0 (3.50500.0). Production private access remains solely in src/web_httpd_adapter.c, with the existing compile-time version guard extended to require idle-lifecycle re-audit.

• esp_http_server/src/httpd_main.c: httpd_server selects the listener only with capacity or LRU enabled. Control work runs before current data sessions, then accept. httpd_process_session skips async sessions and synchronously runs httpd_sess_process; errors delete the session before any subsequent owner work. Accept invokes TLS synchronously through open_fn.
• esp_http_server/src/httpd_sess.c: httpd_sess_process calls httpd_req_new, then httpd_req_delete, and only after both return success assigns session->lru_counter = ++hd->lru_counter. This happens even with LRU disabled. New sessions zero the slot, including the marker; the global counter resets when all sessions are deleted. Application calls to httpd_sess_update_lru_counter are confined to verified serial-WebSocket send work (web_serial_transport.c), which the sweep exempts. This marker is not a timestamp and not an fd-generation token.
• esp_http_server/src/httpd_parse.c: httpd_req_new synchronously parses and invokes the URI handler. httpd_req_delete drains any remaining body; cleanup clears hd_req_aux.sd and request pointers. A return from a diagnostic handler wrapper or a response-send call is earlier than this boundary. The sweep requires HTTPD's thread identity and no current hd_req_aux.sd, and skips for_async_req. Current ordinary handlers do not use async requests, out-of-owner sends, or unfinished chunked responses. Re-audit that contract if introduced.
• esp_http_server/include/esp_http_server.h has no global synchronous post-request-cleanup hook. Its event notifications are not such a hook: esp_http_server_dispatch_event posts to the event loop. HTTP_SERVER_EVENT_SENT_DATA is emitted by httpd_resp_send and per httpd_resp_send_chunk, before handler return/body purge/cleanup. Send/receive overrides belong to HTTPS and do not expose a safe completion hook. URI matching and error handlers likewise cannot supply an all-route post-cleanup boundary.
• esp_https_server/src/https_server.c: successful httpd_ssl_open stores the transport context and installs TLS send/recv/pending functions before synchronous HTTPD_SSL_USER_CB_SESS_CREATE. The application callback invalidates any old row for that fd even if the TLS pointer, socket-slot address and counter value were reused. If fd lookup unexpectedly fails, all observations reset conservatively. The close callback remains the diagnostic observer; SDK destruction remains intact. httpd_ssl_pending calls esp_tls_get_bytes_avail without consuming data; errors can post an existing SDK error event.
• httpd_sess_trigger_close resolves fd to a raw reusable sock_db *, then queues httpd_sess_close. Its zero-counter/LRU guard does not prove the same connection still occupies that slot. Not used by this policy. Direct owner shutdown retains the slot until SDK read cleanup and has no deferred fd/pointer argument that could later close a replacement.
• httpd_queue_work uses cs_send_to_ctrl_sock / sendto in src/util/ctrl_sock.c. The actual generated config leaves CONFIG_HTTPD_QUEUE_WORK_BLOCKING undefined/off, selecting the nonblocking queue mode. The idle initializer explicitly rejects builds with that blocking option on. Successful httpd_stop waits for THREAD_STOPPED, frees HTTPD and ends possible old callback execution; failed stop is not a retirement boundary.

Ownership and bounded storage

src/web_httpd_idle.{c,h} owns one persistent ESP timer, six static observation rows, lifecycle gate/generation and queued/submitting flags. It uses no request data, secret, dynamic per-connection allocation, payload buffer, new task or additional socket. Timer callback performs only short metadata locking and at most one queue submission; all private session access, readiness and shutdown run on HTTPD.

src/web_server.c serializes lifecycle. Prepare initializes rows before SSL startup; the TLS callback and sweep thereafter share the same owner. Attach publishes a nonzero, nonwrapping uintptr_t server generation, passed by value as opaque work argument—not a mutable shared descriptor or raw fd. At most one queued/executing probe is reserved; a separate submitting flag remains set until httpd_queue_work returns even if work already finished. That closes the callback-before-submit-return race.

Detach first prevents submissions, then waits at most one second for any submitting call to return. Fence timeout forbids SSL destruction, retaining the handle for retry. An already executing sweep may finish safely while stop waits for HTTPD. Only successful SDK stop retires a discarded queued reservation; restart gets a new generation even if the server handle is reused. A stale generation cannot sweep or clear a newer reservation. No counter wrap or generic off-owner session-list query is accepted.

src/web_httpd_adapter.{c,h} defines the six-row bound and 15-second policy and performs the version-pinned owner sweep. src/CMakeLists.txt adds only the new module. Diagnostics remains unchanged internally; server composes idle identity reset followed by existing diagnostic publication.

Validation and resource accounting

Commands executed successfully:

python3 tests/web_httpd_idle/run.py
python3 tests/web_admin_transport/server_lifecycle.py
python3 tests/web_diagnostics/run.py
python3 tests/web_cookie_auth/run.py --accounts
python3 tests/web_cookie_auth/run.py --admin
python3 tests/web_cookie_auth/run.py --settings
python3 tests/web_cookie_auth/run.py --serial-settings
python3 tests/web_admin_transport/run.py --tickets
python3 tests/web_session_store/run.py --serial
python3 tests/web_ui_session/run.py
python3 tests/admin_console_boundary/run.py
python3 tests/admin_console_boundary/accounts.py
python3 tests/admin_console_boundary/lifecycle.py
python3 tests/admin_ssh_policy/run.py
pio run
git diff --check

• Idle suite: 18 lifecycle groups plus SDK/source guards. Compiles complete production lifecycle module and production sweep, with installed SDK httpd_sess_process/httpd_req_delete. Uses real host socketpair, readiness, shutdown, EOF and fd reuse; TLS, parsing/response callbacks, timer/queue scheduling and session deletion are deterministic doubles. Tests cover full slots, never-used post-TLS idle, five-second polling, pending/readable/error input, slow synchronous handlers/purge, failed requests, upgrade/async/closing-WS exemption, exact fd/TLS/slot/counter reuse, queue failures, early callback return, submit-fence timeout, failed stop, discarded/stale generations, accepted-but-lost UDP work, diagnostic-independent enforcement and timer initialization/restart/exhaustion. No real TLS/ESP-IDF scheduler or target timing claim.
• Server lifecycle 18 groups, diagnostics 12+1, admin transport 25 / tickets 12, browser 68 plus renderer/CSP, and the listed auth/store/canonical console/account suites pass. The cookie harness required one missing <stdint.h> include after its private-layout double was expanded; fixed, all listed variants rerun successfully.
• Production build: 57.55 seconds, 99,316 B RAM / 1,708,981 B flash, +160 / +1,384 B versus the recorded diagnostic baseline (99,156 / 1,707,597). Final no-op confirmation pio run also passed in 3.13 seconds with identical sizes.
• Target object symbols: rows 144 B; module static symbols 167 B total before linker placement/alignment. Linked RAM delta need not equal the sum of new symbols. One persistent internal-memory ESP timer allocation: installed non-profiled layout 32 B before allocator overhead. Control UDP packet/mailbox transient allocation is SDK-owned, not included in static RAM. No task/stack/socket/queue-capacity increase.
• Target disassembly local frames (exclude callees): sweep 80 B, idle work/timer 32/32 B, prepare 64 B, attach/detach/stopped 32 B each, TLS idle callback 48 B, composed TLS callback 32 B. Actual HTTPD/timer stack margins, heap minima, TLS churn/latency and runtime reserves are unmeasured/unapproved.

Target checklist — not performed

1. Start with diagnostic capture disabled: establish two serial WebSockets and ordinary HTTPS fetches. Verify idle ordinary sockets disappear after the observation window while both serial clients/lease remain unchanged. Enable capture only as needed to compare close/open occupancy; ages remain connection ages, not idle timestamps.
2. Reproduce the original ordinary4/serial2 full-slot case; wait beyond the idle window, then issue/open admin. Record client ticket/TLS/upgrade timings and occupancy without recording tickets/cookies. Verify no repeat reload loop is needed for already-old idle saturation. Separately test fresh saturation and acknowledge the existing 15-second browser timeout limit.
3. Leave status polling active for several minutes: no five-second TLS reconnect churn; both WebSockets and binary serial data/broker isolation survive. Repeat Settings/account-key operations, large assets, login/logout and two browser contexts. Compare with polling paused to distinguish genuinely idle slots.
4. Slow incomplete headers/bodies, pipelined requests, slow response readers and slow/failed TLS handshakes: no probe-driven close of an executing response or admitted WebSocket. Record owner delays; do not infer an overall request deadline from the unchanged one-second receive timeout.
5. Repeated close/reopen/fd reuse, full-mix stop/start and certificate rotation via supported UART0/SSH/browser lifecycle paths. Confirm UART0 and USB remain available, queued work never affects replacement connections, failed-stop retries retain ownership, and no start allocates a second server.
6. Capture settled/loaded/post-cleanup internal/DMA/PSRAM values and HTTPD/ESP-timer minimum-free stack. Soak at the accepted full client mix. Investigate control queue loss separately if probing appears stuck; successful stop/start is the safe recovery, not an eviction/capacity increase.