@@ -112,6 +112,8 @@ A WebSocket connection requires a one-time, principal-bound ticket with a maximu
`web_diagnostics` observes public synchronous HTTPS create/close callbacks without replacing socket/transport cleanup, and wraps only the four serial/admin ticket/upgrade handlers. Six always-maintained post-TLS metadata records supply a console-safe occupancy snapshot without querying HTTPD off-owner; an opt-in 32-entry numeric ring adds timing/heap/HTTPD stack samples. Firmware-lifetime connection sequences and capture epochs survive clear/restart and fence fd reuse/toggle races. No tasks/probes/event subscriptions; no authentication/request data retained. UART0/admin SSH commands never wait for HTTPD; browser policy remains unchanged. This is successful-TLS occupancy, not preaccept or failed/in-progress handshake instrumentation; exact limits/overhead in `docs/phase8d11_implementation.md`.
Ordinary HTTPS idle retention is independently enforced by `web_httpd_idle`: one persistent one-second ESP timer, at most one generation-qualified HTTPD work reservation and six owner-only rows. The private adapter observes IDF's all-route successful `req_new`/`req_delete` completion marker, checks actual WS/async flags and pending/readable input, and shuts down only the current expired ordinary fd after 15 seconds of observed idle. TLS-create resets reused-fd observations before diagnostic publication. No TLS cleanup override, LRU eviction, in-progress request interruption or diagnostics dependency. Stop fences submissions before destroying HTTPD; only successful stop retires queued state. Slow owner work and accepted-but-lost nonblocking UDP work preclude a hard wall-clock guarantee; loss stays bounded until successful restart. Timer preparation failure gates HTTPS start. SDK audit, tests and exact limits: `docs/https_idle_cleanup.md`.
Web serial initialization is failure-isolated from the base HTTPS service: if the transport cannot initialize, `web_server_init()` can still succeed and serve authenticated non-WebSocket routes.
`web_ui.c` contains authored index/application strings and response policy; it validates `/api/session` before connect/restore, adds explicit Sign out, and cancels stale work on 401/logout/page exit. `web_login_ui` is a standalone public page without protected-asset dependencies. Both authentication documents and app script are no-store. Its restrictive CSP contains a hard-coded hash of the inline loader, so those two must change atomically; preserve same-origin connections, no-referrer behavior, frame denial, and the existing cache policy. `web_assets_data.c` contains checked-in generated arrays for vendored compressed xterm assets and the logo. Normal builds compile these arrays directly; they do not regenerate assets.
@@ -4,6 +4,8 @@ This file is working memory. Update it during active work and before handoff; do
## Development state
- **User-authorized bounded ordinary HTTPS idle cleanup complete (2026-09-08), host-tested/build-verified; target pending:** Preserved existing 8D.11 key/diagnostic implementation (initial Git status was clean). User capture: post-TLS occupancy 6/6, ordinary4/serial2/admin0, ordinary connection ages 50–74 s; admin ticket returned in 14 ms, no TLS/upgrade observed and ticket unconsumed. Installed IDF stops selecting listener at capacity with LRU disabled. `web_httpd_idle.{c,h}` plus sole-private-boundary sweep in `web_httpd_adapter`: six owner-only rows, 15-second observed-idle window (three five-second status polls), one-second ESP timer/at most one queued owner probe, current fd shutdown (no reusable-pointer queued close), all-route SDK request-plus-purge completion marker and input readiness/WS/async checks, TLS-create fd reset, submit fence/nonwrapping restart generations. Diagnostics does not gate policy; socket/timeout/LRU/task/stack/asset settings unchanged. Implementation `pio run` PASS 57.55 s, final confirmation PASS 3.13 s; **99,316 B RAM / 1,708,981 B flash (+160/+1,384 versus diagnostic baseline)**. Rows144 B/module static167 B before placement; timer32 B internal heap before overhead; sweep local frame80 B, runtime margins unmeasured. Final idle **18 + SDK guards**, server **18**, diagnostics **12+1** PASS; cookie accounts/admin/settings/serial-settings, admin transport25/tickets12, store/serial, UI68+CSP and canonical console/account/key/lifecycle/policy suites PASS; diff check PASS. Cookie test-double missing `<stdint.h>` fixed and all variants rerun. Exact contracts/audit/tests/resources/target checklist: `docs/https_idle_cleanup.md`. Ordinary ages are connection ages, not proved idle duration. No hard wall-clock/admission guarantee: owner-blocking handshake/parser/send delays cleanup; continuously active ordinary sockets are not evicted. Reported queue errors retry, but accepted-and-lost nonblocking UDP work stays one reserved probe until successful stop/restart (tested), rather than accumulating unsafe delayed work. Failed stop remains detached/owned until retry. No hardware/upload/erase/commit, target acceptance or reserve approval; next is the documented target reproduction/soak, not capacity changes.
- **Authorized 8D.11 admission diagnostic slice complete (2026-09-08), host-tested/build-verified; target pending:**`web_diagnostics.{c,h}` uses public synchronous HTTPS create/close callbacks and four ticket/upgrade wrappers, six fixed metadata slots and a 32-record opt-in numeric ring. Console `web diagnostics enable|disable|show|clear` via UART0/admin SSH; browser policy unchanged. No close/open override, queue probe/task, capacity/timeout/log-level/assets/SDK change. Post-TLS owner-published occupancy only; TLS failures/in-progress handshakes and preaccept backlog remain invisible. Final `pio run` PASS **10.30 s, 99,156 B RAM / 1,707,597 B flash**, **+3,080/+3,912 B** versus recorded key-slice baseline. Ring/table target symbols 2,816/192 B; diagnostic handler/record/TLS callback/show local frames 144/144/160/528 B excluding callees, runtime margins unmeasured. PASS diagnostic **12+1**, server lifecycle **16**, canonical console lifecycle/policy/boundary, canonical account/key transactions, cookie/admin and cookie/accounts, admin transport **25**/tickets **12**, browser **68 plus renderer/CSP**, store/serial and diff check. Exact files/commands/contracts and reproduction checklist in `docs/phase8d11_implementation.md`. Existing key work preserved; no device/commit/upload/erase, timeout diagnosis, target acceptance or reserve approval claimed. Next is user target capture, not capacity changes or 8D.12.
- **8D.11 partial target evidence (2026-09-08):** User supplied 60-second fresh-boot/full-mix telemetry; no functional/sign-off claim. Full mix eventually established (two SSH public-key sessions, SSH writer + USB/two web observers, both admins), but last browser admission again required retries. Three TLS errors `-0x0050` verified against installed mbedTLS as NET_CONN_RESET, not allocation failure; cause unknown. Loaded internal/DMA free 32,596/24,840 B, lifetime minima 5,468/532 B, largest 22,528 B; per-region minima are conservative/non-simultaneous. Settled free is similar to prior 8D.10, minima much lower; no causal regression attribution. SSH stack minimum-free 18,468 boot/16,276 loaded B. Web send failure/close 1, expired tickets 3; SSH handshake/auth failures zero; no broker drop counters. Exact memory/counters/evidence limits in `docs/phase8d11_implementation.md`. Previously accepted admission issue recurred, not fixed; investigate timing/socket occupancy/allocation correlation before capacity changes. Key algorithms/new UI provenance, mutation checklist, HTTPD/dispatcher margins, cleanup/soak and target sign-off remain pending. Documentation-only update; no source/build/device action.
@@ -128,6 +128,8 @@ The following first-slice exclusions are historical and superseded by slice 2:
**HTTPD boundary:** `web_httpd_adapter` alone includes private ESP-IDF 5.5.0 structures. Public request getters expose only the first field, so the adapter validates bounded parsed headers/rejects duplicates. `/ws/serial` is an ordinary GET until authenticated transport admission explicitly sends 101 and installs the frame handler; automatic HTTPD upgrades happen before URI handlers. Preserve right-aligned unread pending bytes when wiping request memory. Two Set-Cookie calls append pointer-backed fields, whose distinct buffers must survive through send. HTTPD logs above ERROR are compiled out to prevent header/ticket exposure.
**Ordinary HTTPS idle lifecycle:** Keep six sockets and LRU disabled. `web_httpd_idle` queues at most one owner sweep each second; `web_httpd_adapter` alone reads the installed successful request-plus-purge completion marker. Fifteen seconds of observed ordinary idle (three normal five-second browser polls), current SDK WS/async exemption, pending/readable-input checks and synchronous TLS-create fd invalidation authorize current-owner `shutdown`, never queued `sock_db *` close. Do not use response events, diagnostic wrappers or connection age as the completion/idle boundary. All ordinary response work must finish synchronously or retain the SDK async exemption. Submission fencing precedes HTTPD stop; only successful stop retires queued state and admits a nonreused generation. No tracing dependency, forced per-response close, hard request deadline, arbitrary admission eviction or capacity increase. A reported queue failure retries; accepted-but-lost UDP work stays reserved until successful restart rather than accumulating potentially delayed probes. Exact safety/liveness limits and target checklist: `docs/https_idle_cleanup.md`.
**Consequence:** The version guard is not a source-hash guarantee. Re-audit layout, scratch/pending ownership, logging and handshake/frame dispatch on SDK changes; do not scatter private accesses through application code or assume host doubles prove real socket behavior. No SDK patch is currently applied. See `docs/phase8d3_implementation.md` for verification and target gates.
Implemented 2026-09-08 on the existing 8D.11 key/diagnostic implementation. Host-tested and firmware-build verified; **not target validated or signed off**. No upload, erase, commit, SDK patch or generated-asset change.
## Evidence and scope
User-authorized response to the admission capture: post-TLS occupancy **6/6, ordinary4 / serial2 / admin0**, ordinary connection ages **50–74 seconds**; admin ticket returned in **14 ms**, no subsequent TLS/upgrade observed, ticket unconsumed. These ages measure connection lifetime, not time since the last request. They support investigating retained ordinary sockets, not claiming every captured socket was idle. Installed HTTPD excludes its listening fd from `select` while full with LRU disabled. New connections can therefore wait before TLS or upgrade even when ticket issuance was fast.
This policy releases **expired idle ordinary HTTPS connections**, not arbitrary ordinary requests to make room. Six sockets, 24 URI handlers, LRU disabled, existing receive/send timeouts (one second), TLS handshake timeout (five seconds), all task stacks/queue depths and browser code remain unchanged. No new task. UART0 recovery, independent USB access, serial broker ownership, tickets, authentication and both WebSocket transports are unchanged.
## Usage and timeout semantics
- Automatically enabled for every successfully started HTTPS server, including when diagnostic tracing is disabled and either optional transport is unavailable. No new console command, setting or credential migration.
- **15 seconds of observed ordinary idle retention**, checked by a **one-second ESP timer** that requests work on HTTPD. The three existing five-second browser status-poll intervals leave room for normal keepalive reuse instead of a TLS handshake per request.
- A new post-TLS connection that has sent no request gets the same full idle window. Time spent establishing TLS does not consume it.
- The first owner sweep observing a new connection or a changed successful-request completion marker starts a fresh window. The marker covers every ordinary route, including login, assets, status, typed settings, tickets and keepalive errors that return successfully—not just diagnostic wrappers. Failed requests are deleted by the normal SDK path instead.
- HTTPD pending bytes, TLS pending bytes or a readable TCP fd reset the observation window. Negative TLS pending results and `select` errors conservatively reset it too. No bytes are read or discarded by the probe.
- At the threshold, after current SDK classification and zero-time readiness checks, HTTPD calls `shutdown(current_fd, SHUT_RDWR)`. A successful shutdown is latched; a failed call retries on the next probe. The SDK's subsequent read/delete path owns socket close, TLS destruction, diagnostic close notification and freeing the slot. The probe never calls `close`, overrides TLS cleanup, or queues a session-close pointer.
- **WebSockets (serial and admin), closing WebSockets and async requests are exempt.** The check uses actual SDK flags, not diagnostic metadata. Successful explicit 101/classification and request cleanup finish before the owner can sweep, so there is no ordinary-idle interval during upgrade admission.
- Under an available owner and timely successful work delivery, expect 15 seconds plus the initial observation delay (nominally up to one second), the next probe delay (nominally up to one second), and SDK read/cleanup latency. This is a conservative sampled idle policy, **not a strict wall-clock timeout or admission SLA**.
### Deliberate limitations
1. Parsing an incomplete request, synchronous response sends, leftover-body purge and TLS handshakes serialize on HTTPD. The probe cannot interrupt them. Existing per-read/per-send timeouts remain; a peer trickling input can extend overall processing beyond one timeout. This change does not provide a slowloris deadline or solve TLS/memory/global-socket pressure.
2. Four continuously active ordinary connections plus two WebSockets can still fill all six slots. They are not evicted. Likewise, an admission attempt immediately after fresh saturation can hit the browser's existing 15-second timeout before the conservative window expires; the user may still need to retry. Older idle retained sockets are eligible on the next delivered probe.
3. As with any HTTP keepalive timeout, bytes arriving **after** the last readiness check can race a shutdown. Already executing HTTPD requests/responses are protected; future client intent cannot be predicted. A client may need a new connection. No new application-level mutation retry/replay is added. Response completion here means synchronous HTTPD completion, not proof of peer receipt/TCP acknowledgement.
4. `httpd_queue_work` with `CONFIG_HTTPD_QUEUE_WORK_BLOCKING` disabled uses loopback UDP. A reported queue error releases the reservation and retries next tick. **A successful send is not an execution acknowledgement**: an accepted-but-lost control message leaves one reservation pending, disables further probes, and requires successful HTTPS stop/restart to restore probing. There is no speculative reservation timeout: it could accumulate delayed callbacks and violate the one-probe bound. This is explicitly regression-tested, not hidden behind a hard retention guarantee. HTTPD blockage or queue loss also cannot be repaired by raising sockets/LRU/timeouts here.
5. A failed stop leaves cleanup detached and ownership retained for a later stop retry; it does not restart probes on a partially stopped server. Timer allocation/start failure gates HTTPS start with its error rather than silently starting without the policy. UART0/USB recovery is unaffected.
## Exact installed SDK audit
All SDK references below are under `~/.platformio/packages/framework-espidf/components/`; installed framework is ESP-IDF **5.5.0** (`3.50500.0`). Production private access remains solely in `src/web_httpd_adapter.c`, with the existing compile-time version guard extended to require idle-lifecycle re-audit.
- `esp_http_server/src/httpd_main.c`: `httpd_server` selects the listener only with capacity or LRU enabled. Control work runs **before** current data sessions, then accept. `httpd_process_session` skips async sessions and synchronously runs `httpd_sess_process`; errors delete the session before any subsequent owner work. Accept invokes TLS synchronously through `open_fn`.
- `esp_http_server/src/httpd_sess.c`: `httpd_sess_process` calls `httpd_req_new`, then `httpd_req_delete`, and only after both return success assigns `session->lru_counter = ++hd->lru_counter`. This happens even with LRU disabled. New sessions zero the slot, including the marker; the global counter resets when all sessions are deleted. Application calls to `httpd_sess_update_lru_counter` are confined to verified serial-WebSocket send work (`web_serial_transport.c`), which the sweep exempts. This marker is not a timestamp and not an fd-generation token.
- `esp_http_server/src/httpd_parse.c`: `httpd_req_new` synchronously parses and invokes the URI handler. `httpd_req_delete` drains any remaining body; cleanup clears `hd_req_aux.sd` and request pointers. A return from a diagnostic handler wrapper or a response-send call is earlier than this boundary. The sweep requires HTTPD's thread identity and no current `hd_req_aux.sd`, and skips `for_async_req`. Current ordinary handlers do not use async requests, out-of-owner sends, or unfinished chunked responses. Re-audit that contract if introduced.
- `esp_http_server/include/esp_http_server.h` has no global synchronous post-request-cleanup hook. Its event notifications are not such a hook: `esp_http_server_dispatch_event` posts to the event loop. `HTTP_SERVER_EVENT_SENT_DATA` is emitted by `httpd_resp_send` and per `httpd_resp_send_chunk`, **before** handler return/body purge/cleanup. Send/receive overrides belong to HTTPS and do not expose a safe completion hook. URI matching and error handlers likewise cannot supply an all-route post-cleanup boundary.
- `esp_https_server/src/https_server.c`: successful `httpd_ssl_open` stores the transport context and installs TLS send/recv/pending functions before synchronous `HTTPD_SSL_USER_CB_SESS_CREATE`. The application callback invalidates any old row for that fd even if the TLS pointer, socket-slot address and counter value were reused. If fd lookup unexpectedly fails, all observations reset conservatively. The close callback remains the diagnostic observer; SDK destruction remains intact. `httpd_ssl_pending` calls `esp_tls_get_bytes_avail` without consuming data; errors can post an existing SDK error event.
- `httpd_sess_trigger_close` resolves fd to a raw reusable `sock_db *`, then queues `httpd_sess_close`. Its zero-counter/LRU guard does not prove the same connection still occupies that slot. **Not used by this policy.** Direct owner shutdown retains the slot until SDK read cleanup and has no deferred fd/pointer argument that could later close a replacement.
- `httpd_queue_work` uses `cs_send_to_ctrl_sock` / `sendto` in `src/util/ctrl_sock.c`. The actual generated config leaves `CONFIG_HTTPD_QUEUE_WORK_BLOCKING` undefined/off, selecting the nonblocking queue mode. The idle initializer explicitly rejects builds with that blocking option on. Successful `httpd_stop` waits for `THREAD_STOPPED`, frees HTTPD and ends possible old callback execution; failed stop is not a retirement boundary.
## Ownership and bounded storage
`src/web_httpd_idle.{c,h}` owns one persistent ESP timer, six static observation rows, lifecycle gate/generation and queued/submitting flags. It uses no request data, secret, dynamic per-connection allocation, payload buffer, new task or additional socket. Timer callback performs only short metadata locking and at most one queue submission; all private session access, readiness and shutdown run on HTTPD.
`src/web_server.c` serializes lifecycle. Prepare initializes rows **before** SSL startup; the TLS callback and sweep thereafter share the same owner. Attach publishes a nonzero, nonwrapping `uintptr_t` server generation, passed by value as opaque work argument—not a mutable shared descriptor or raw fd. At most one queued/executing probe is reserved; a separate submitting flag remains set until `httpd_queue_work` returns even if work already finished. That closes the callback-before-submit-return race.
Detach first prevents submissions, then waits at most one second for any submitting call to return. Fence timeout forbids SSL destruction, retaining the handle for retry. An already executing sweep may finish safely while stop waits for HTTPD. Only successful SDK stop retires a discarded queued reservation; restart gets a new generation even if the server handle is reused. A stale generation cannot sweep or clear a newer reservation. No counter wrap or generic off-owner session-list query is accepted.
`src/web_httpd_adapter.{c,h}` defines the six-row bound and 15-second policy and performs the version-pinned owner sweep. `src/CMakeLists.txt` adds only the new module. Diagnostics remains unchanged internally; server composes idle identity reset followed by existing diagnostic publication.
- Idle suite: **18 lifecycle groups plus SDK/source guards**. Compiles complete production lifecycle module and production sweep, with installed SDK `httpd_sess_process`/`httpd_req_delete`. Uses real host `socketpair`, readiness, shutdown, EOF and fd reuse; TLS, parsing/response callbacks, timer/queue scheduling and session deletion are deterministic doubles. Tests cover full slots, never-used post-TLS idle, five-second polling, pending/readable/error input, slow synchronous handlers/purge, failed requests, upgrade/async/closing-WS exemption, exact fd/TLS/slot/counter reuse, queue failures, early callback return, submit-fence timeout, failed stop, discarded/stale generations, accepted-but-lost UDP work, diagnostic-independent enforcement and timer initialization/restart/exhaustion. No real TLS/ESP-IDF scheduler or target timing claim.
- Server lifecycle **18 groups**, diagnostics **12+1**, admin transport **25** / tickets **12**, browser **68 plus renderer/CSP**, and the listed auth/store/canonical console/account suites pass. The cookie harness required one missing `<stdint.h>` include after its private-layout double was expanded; fixed, all listed variants rerun successfully.
- Production build: **57.55 seconds**, **99,316 B RAM / 1,708,981 B flash**, **+160 / +1,384 B** versus the recorded diagnostic baseline (99,156 / 1,707,597). Final no-op confirmation `pio run` also passed in **3.13 seconds** with identical sizes.
- Target object symbols: rows **144 B**; module static symbols **167 B total before linker placement/alignment**. Linked RAM delta need not equal the sum of new symbols. One persistent internal-memory ESP timer allocation: installed non-profiled layout **32 B before allocator overhead**. Control UDP packet/mailbox transient allocation is SDK-owned, not included in static RAM. No task/stack/socket/queue-capacity increase.
- Target disassembly local frames (exclude callees): sweep **80 B**, idle work/timer **32/32 B**, prepare **64 B**, attach/detach/stopped **32 B each**, TLS idle callback **48 B**, composed TLS callback **32 B**. Actual HTTPD/timer stack margins, heap minima, TLS churn/latency and runtime reserves are **unmeasured/unapproved**.
## Target checklist — not performed
1. Start with diagnostic capture disabled: establish two serial WebSockets and ordinary HTTPS fetches. Verify idle ordinary sockets disappear after the observation window while both serial clients/lease remain unchanged. Enable capture only as needed to compare close/open occupancy; ages remain connection ages, not idle timestamps.
2. Reproduce the original ordinary4/serial2 full-slot case; wait beyond the idle window, then issue/open admin. Record client ticket/TLS/upgrade timings and occupancy without recording tickets/cookies. Verify no repeat reload loop is needed for already-old idle saturation. Separately test fresh saturation and acknowledge the existing 15-second browser timeout limit.
3. Leave status polling active for several minutes: no five-second TLS reconnect churn; both WebSockets and binary serial data/broker isolation survive. Repeat Settings/account-key operations, large assets, login/logout and two browser contexts. Compare with polling paused to distinguish genuinely idle slots.
4. Slow incomplete headers/bodies, pipelined requests, slow response readers and slow/failed TLS handshakes: no probe-driven close of an executing response or admitted WebSocket. Record owner delays; do not infer an overall request deadline from the unchanged one-second receive timeout.
5. Repeated close/reopen/fd reuse, full-mix stop/start and certificate rotation via supported UART0/SSH/browser lifecycle paths. Confirm UART0 and USB remain available, queued work never affects replacement connections, failed-stop retries retain ownership, and no start allocates a second server.
6. Capture settled/loaded/post-cleanup internal/DMA/PSRAM values and HTTPD/ESP-timer minimum-free stack. Soak at the accepted full client mix. Investigate control queue loss separately if probing appears stuck; successful stop/start is the safe recovery, not an eviction/capacity increase.
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
