# Bounded ordinary HTTPS idle retention

Implemented 2026-09-08 on the existing 8D.11 key/diagnostic implementation. Host-tested and firmware-build verified; **not target validated or signed off**. No upload, erase, commit, SDK patch or generated-asset change.

## Evidence and scope

User-authorized response to the admission capture: post-TLS occupancy **6/6, ordinary4 / serial2 / admin0**, ordinary connection ages **50–74 seconds**; admin ticket returned in **14 ms**, no subsequent TLS/upgrade observed, ticket unconsumed. These ages measure connection lifetime, not time since the last request. They support investigating retained ordinary sockets, not claiming every captured socket was idle. Installed HTTPD excludes its listening fd from `select` while full with LRU disabled. New connections can therefore wait before TLS or upgrade even when ticket issuance was fast.

This policy releases **expired idle ordinary HTTPS connections**, not arbitrary ordinary requests to make room. Six sockets, 24 URI handlers, LRU disabled, existing receive/send timeouts (one second), TLS handshake timeout (five seconds), all task stacks/queue depths and browser code remain unchanged. No new task. UART0 recovery, independent USB access, serial broker ownership, tickets, authentication and both WebSocket transports are unchanged.

## Usage and timeout semantics

- Automatically enabled for every successfully started HTTPS server, including when diagnostic tracing is disabled and either optional transport is unavailable. No new console command, setting or credential migration.
- **15 seconds of observed ordinary idle retention**, checked by a **one-second ESP timer** that requests work on HTTPD. The three existing five-second browser status-poll intervals leave room for normal keepalive reuse instead of a TLS handshake per request.
- A new post-TLS connection that has sent no request gets the same full idle window. Time spent establishing TLS does not consume it.
- The first owner sweep observing a new connection or a changed successful-request completion marker starts a fresh window. The marker covers every ordinary route, including login, assets, status, typed settings, tickets and keepalive errors that return successfully—not just diagnostic wrappers. Failed requests are deleted by the normal SDK path instead.
- HTTPD pending bytes, TLS pending bytes or a readable TCP fd reset the observation window. Negative TLS pending results and `select` errors conservatively reset it too. No bytes are read or discarded by the probe.
- At the threshold, after current SDK classification and zero-time readiness checks, HTTPD calls `shutdown(current_fd, SHUT_RDWR)`. A successful shutdown is latched; a failed call retries on the next probe. The SDK's subsequent read/delete path owns socket close, TLS destruction, diagnostic close notification and freeing the slot. The probe never calls `close`, overrides TLS cleanup, or queues a session-close pointer.
- **WebSockets (serial and admin), closing WebSockets and async requests are exempt.** The check uses actual SDK flags, not diagnostic metadata. Successful explicit 101/classification and request cleanup finish before the owner can sweep, so there is no ordinary-idle interval during upgrade admission.
- Under an available owner and timely successful work delivery, expect 15 seconds plus the initial observation delay (nominally up to one second), the next probe delay (nominally up to one second), and SDK read/cleanup latency. This is a conservative sampled idle policy, **not a strict wall-clock timeout or admission SLA**.

### Deliberate limitations

1. Parsing an incomplete request, synchronous response sends, leftover-body purge and TLS handshakes serialize on HTTPD. The probe cannot interrupt them. Existing per-read/per-send timeouts remain; a peer trickling input can extend overall processing beyond one timeout. This change does not provide a slowloris deadline or solve TLS/memory/global-socket pressure.
2. Four continuously active ordinary connections plus two WebSockets can still fill all six slots. They are not evicted. Likewise, an admission attempt immediately after fresh saturation can hit the browser's existing 15-second timeout before the conservative window expires; the user may still need to retry. Older idle retained sockets are eligible on the next delivered probe.
3. As with any HTTP keepalive timeout, bytes arriving **after** the last readiness check can race a shutdown. Already executing HTTPD requests/responses are protected; future client intent cannot be predicted. A client may need a new connection. No new application-level mutation retry/replay is added. Response completion here means synchronous HTTPD completion, not proof of peer receipt/TCP acknowledgement.
4. `httpd_queue_work` with `CONFIG_HTTPD_QUEUE_WORK_BLOCKING` disabled uses loopback UDP. A reported queue error releases the reservation and retries next tick. **A successful send is not an execution acknowledgement**: an accepted-but-lost control message leaves one reservation pending, disables further probes, and requires successful HTTPS stop/restart to restore probing. There is no speculative reservation timeout: it could accumulate delayed callbacks and violate the one-probe bound. This is explicitly regression-tested, not hidden behind a hard retention guarantee. HTTPD blockage or queue loss also cannot be repaired by raising sockets/LRU/timeouts here.
5. A failed stop leaves cleanup detached and ownership retained for a later stop retry; it does not restart probes on a partially stopped server. Timer allocation/start failure gates HTTPS start with its error rather than silently starting without the policy. UART0/USB recovery is unaffected.

## Exact installed SDK audit

All SDK references below are under `~/.platformio/packages/framework-espidf/components/`; installed framework is ESP-IDF **5.5.0** (`3.50500.0`). Production private access remains solely in `src/web_httpd_adapter.c`, with the existing compile-time version guard extended to require idle-lifecycle re-audit.

- `esp_http_server/src/httpd_main.c`: `httpd_server` selects the listener only with capacity or LRU enabled. Control work runs **before** current data sessions, then accept. `httpd_process_session` skips async sessions and synchronously runs `httpd_sess_process`; errors delete the session before any subsequent owner work. Accept invokes TLS synchronously through `open_fn`.
- `esp_http_server/src/httpd_sess.c`: `httpd_sess_process` calls `httpd_req_new`, then `httpd_req_delete`, and only after both return success assigns `session->lru_counter = ++hd->lru_counter`. This happens even with LRU disabled. New sessions zero the slot, including the marker; the global counter resets when all sessions are deleted. Application calls to `httpd_sess_update_lru_counter` are confined to verified serial-WebSocket send work (`web_serial_transport.c`), which the sweep exempts. This marker is not a timestamp and not an fd-generation token.
- `esp_http_server/src/httpd_parse.c`: `httpd_req_new` synchronously parses and invokes the URI handler. `httpd_req_delete` drains any remaining body; cleanup clears `hd_req_aux.sd` and request pointers. A return from a diagnostic handler wrapper or a response-send call is earlier than this boundary. The sweep requires HTTPD's thread identity and no current `hd_req_aux.sd`, and skips `for_async_req`. Current ordinary handlers do not use async requests, out-of-owner sends, or unfinished chunked responses. Re-audit that contract if introduced.
- `esp_http_server/include/esp_http_server.h` has no global synchronous post-request-cleanup hook. Its event notifications are not such a hook: `esp_http_server_dispatch_event` posts to the event loop. `HTTP_SERVER_EVENT_SENT_DATA` is emitted by `httpd_resp_send` and per `httpd_resp_send_chunk`, **before** handler return/body purge/cleanup. Send/receive overrides belong to HTTPS and do not expose a safe completion hook. URI matching and error handlers likewise cannot supply an all-route post-cleanup boundary.
- `esp_https_server/src/https_server.c`: successful `httpd_ssl_open` stores the transport context and installs TLS send/recv/pending functions before synchronous `HTTPD_SSL_USER_CB_SESS_CREATE`. The application callback invalidates any old row for that fd even if the TLS pointer, socket-slot address and counter value were reused. If fd lookup unexpectedly fails, all observations reset conservatively. The close callback remains the diagnostic observer; SDK destruction remains intact. `httpd_ssl_pending` calls `esp_tls_get_bytes_avail` without consuming data; errors can post an existing SDK error event.
- `httpd_sess_trigger_close` resolves fd to a raw reusable `sock_db *`, then queues `httpd_sess_close`. Its zero-counter/LRU guard does not prove the same connection still occupies that slot. **Not used by this policy.** Direct owner shutdown retains the slot until SDK read cleanup and has no deferred fd/pointer argument that could later close a replacement.
- `httpd_queue_work` uses `cs_send_to_ctrl_sock` / `sendto` in `src/util/ctrl_sock.c`. The actual generated config leaves `CONFIG_HTTPD_QUEUE_WORK_BLOCKING` undefined/off, selecting the nonblocking queue mode. The idle initializer explicitly rejects builds with that blocking option on. Successful `httpd_stop` waits for `THREAD_STOPPED`, frees HTTPD and ends possible old callback execution; failed stop is not a retirement boundary.

## Ownership and bounded storage

`src/web_httpd_idle.{c,h}` owns one persistent ESP timer, six static observation rows, lifecycle gate/generation and queued/submitting flags. It uses no request data, secret, dynamic per-connection allocation, payload buffer, new task or additional socket. Timer callback performs only short metadata locking and at most one queue submission; all private session access, readiness and shutdown run on HTTPD.

`src/web_server.c` serializes lifecycle. Prepare initializes rows **before** SSL startup; the TLS callback and sweep thereafter share the same owner. Attach publishes a nonzero, nonwrapping `uintptr_t` server generation, passed by value as opaque work argument—not a mutable shared descriptor or raw fd. At most one queued/executing probe is reserved; a separate submitting flag remains set until `httpd_queue_work` returns even if work already finished. That closes the callback-before-submit-return race.

Detach first prevents submissions, then waits at most one second for any submitting call to return. Fence timeout forbids SSL destruction, retaining the handle for retry. An already executing sweep may finish safely while stop waits for HTTPD. Only successful SDK stop retires a discarded queued reservation; restart gets a new generation even if the server handle is reused. A stale generation cannot sweep or clear a newer reservation. No counter wrap or generic off-owner session-list query is accepted.

`src/web_httpd_adapter.{c,h}` defines the six-row bound and 15-second policy and performs the version-pinned owner sweep. `src/CMakeLists.txt` adds only the new module. Diagnostics remains unchanged internally; server composes idle identity reset followed by existing diagnostic publication.

## Validation and resource accounting

Commands executed successfully:

```sh
python3 tests/web_httpd_idle/run.py
python3 tests/web_admin_transport/server_lifecycle.py
python3 tests/web_diagnostics/run.py
python3 tests/web_cookie_auth/run.py --accounts
python3 tests/web_cookie_auth/run.py --admin
python3 tests/web_cookie_auth/run.py --settings
python3 tests/web_cookie_auth/run.py --serial-settings
python3 tests/web_admin_transport/run.py --tickets
python3 tests/web_session_store/run.py --serial
python3 tests/web_ui_session/run.py
python3 tests/admin_console_boundary/run.py
python3 tests/admin_console_boundary/accounts.py
python3 tests/admin_console_boundary/lifecycle.py
python3 tests/admin_ssh_policy/run.py
pio run
git diff --check
```

- Idle suite: **18 lifecycle groups plus SDK/source guards**. Compiles complete production lifecycle module and production sweep, with installed SDK `httpd_sess_process`/`httpd_req_delete`. Uses real host `socketpair`, readiness, shutdown, EOF and fd reuse; TLS, parsing/response callbacks, timer/queue scheduling and session deletion are deterministic doubles. Tests cover full slots, never-used post-TLS idle, five-second polling, pending/readable/error input, slow synchronous handlers/purge, failed requests, upgrade/async/closing-WS exemption, exact fd/TLS/slot/counter reuse, queue failures, early callback return, submit-fence timeout, failed stop, discarded/stale generations, accepted-but-lost UDP work, diagnostic-independent enforcement and timer initialization/restart/exhaustion. No real TLS/ESP-IDF scheduler or target timing claim.
- Server lifecycle **18 groups**, diagnostics **12+1**, admin transport **25** / tickets **12**, browser **68 plus renderer/CSP**, and the listed auth/store/canonical console/account suites pass. The cookie harness required one missing `<stdint.h>` include after its private-layout double was expanded; fixed, all listed variants rerun successfully.
- Production build: **57.55 seconds**, **99,316 B RAM / 1,708,981 B flash**, **+160 / +1,384 B** versus the recorded diagnostic baseline (99,156 / 1,707,597). Final no-op confirmation `pio run` also passed in **3.13 seconds** with identical sizes.
- Target object symbols: rows **144 B**; module static symbols **167 B total before linker placement/alignment**. Linked RAM delta need not equal the sum of new symbols. One persistent internal-memory ESP timer allocation: installed non-profiled layout **32 B before allocator overhead**. Control UDP packet/mailbox transient allocation is SDK-owned, not included in static RAM. No task/stack/socket/queue-capacity increase.
- Target disassembly local frames (exclude callees): sweep **80 B**, idle work/timer **32/32 B**, prepare **64 B**, attach/detach/stopped **32 B each**, TLS idle callback **48 B**, composed TLS callback **32 B**. Actual HTTPD/timer stack margins, heap minima, TLS churn/latency and runtime reserves are **unmeasured/unapproved**.

## Target checklist — not performed

1. Start with diagnostic capture disabled: establish two serial WebSockets and ordinary HTTPS fetches. Verify idle ordinary sockets disappear after the observation window while both serial clients/lease remain unchanged. Enable capture only as needed to compare close/open occupancy; ages remain connection ages, not idle timestamps.
2. Reproduce the original ordinary4/serial2 full-slot case; wait beyond the idle window, then issue/open admin. Record client ticket/TLS/upgrade timings and occupancy without recording tickets/cookies. Verify no repeat reload loop is needed for already-old idle saturation. Separately test fresh saturation and acknowledge the existing 15-second browser timeout limit.
3. Leave status polling active for several minutes: no five-second TLS reconnect churn; both WebSockets and binary serial data/broker isolation survive. Repeat Settings/account-key operations, large assets, login/logout and two browser contexts. Compare with polling paused to distinguish genuinely idle slots.
4. Slow incomplete headers/bodies, pipelined requests, slow response readers and slow/failed TLS handshakes: no probe-driven close of an executing response or admitted WebSocket. Record owner delays; do not infer an overall request deadline from the unchanged one-second receive timeout.
5. Repeated close/reopen/fd reuse, full-mix stop/start and certificate rotation via supported UART0/SSH/browser lifecycle paths. Confirm UART0 and USB remain available, queued work never affects replacement connections, failed-stop retries retain ownership, and no start allocates a second server.
6. Capture settled/loaded/post-cleanup internal/DMA/PSRAM values and HTTPD/ESP-timer minimum-free stack. Soak at the accepted full client mix. Investigate control queue loss separately if probing appears stuck; successful stop/start is the safe recovery, not an eviction/capacity increase.