Commander1024/ESP32_Serial_Swiss_Army_Knife
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Consolidate Phase 8 documentation

Browse Source 
Mark web administration complete, centralize current contracts and
acceptance evidence, and remove superseded slice records. Update
roadmap,
architecture notes, and test references without changing firmware
sources.
This commit is contained in:
Commander1024
2026-09-13 22:27:10 +02:00
parent 1608641d50
commit 91267b371e
39 changed files with 376 additions and 3489 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/https_idle_cleanup.md
+7 -30
View File
@@ -1,12 +1,12 @@
# Bounded ordinary HTTPS idle retention
Implemented 2026-09-08 on the existing 8D.11 key/diagnostic implementation. Host-tested and firmware-build verified. **Subsequent scoped user report:** the user accepted that idle cleanup worked. This supersedes the original target-pending status for that behavior only; no additional soak duration, individual checklist passes, memory reserve approval or broader 8D.11/M3 sign-off is inferred. The implementation evidence below remains historical and unchanged. No agent upload, erase, commit, SDK patch or generated-asset change.
Current owner/SDK contract. The user accepted idle cleanup as working; [overall acceptance](web_administration_acceptance.md) records current status and evidence limits. No individual unreported soak, fault or reserve result is inferred.
## Evidence and scope
User-authorized response to the admission capture: post-TLS occupancy **6/6, ordinary4 / serial2 / admin0**, ordinary connection ages **50–74 seconds**; admin ticket returned in **14 ms**, no subsequent TLS/upgrade observed, ticket unconsumed. These ages measure connection lifetime, not time since the last request. They support investigating retained ordinary sockets, not claiming every captured socket was idle. Installed HTTPD excludes its listening fd from `select` while full with LRU disabled. New connections can therefore wait before TLS or upgrade even when ticket issuance was fast.
This policy releases **expired idle ordinary HTTPS connections**, not arbitrary ordinary requests to make room. Six sockets, 24 URI handlers, LRU disabled, existing receive/send timeouts (one second), TLS handshake timeout (five seconds), all task stacks/queue depths and browser code remain unchanged. No new task. UART0 recovery, independent USB access, serial broker ownership, tickets, authentication and both WebSocket transports are unchanged.
This policy releases **expired idle ordinary HTTPS connections**, not arbitrary ordinary requests to make room. The policy uses the existing six sockets, LRU-disabled operation, one-second receive/send and five-second TLS-handshake timeouts without expanding task stacks or queue capacities. Current overall handler capacity is 39. No new task. UART0 recovery, independent USB access, serial broker ownership, tickets, authentication and both WebSocket transports are unchanged.
## Usage and timeout semantics
@@ -37,7 +37,7 @@ All SDK references below are under `~/.platformio/packages/framework-espidf/comp
- `esp_http_server/include/esp_http_server.h` has no global synchronous post-request-cleanup hook. Its event notifications are not such a hook: `esp_http_server_dispatch_event` posts to the event loop. `HTTP_SERVER_EVENT_SENT_DATA` is emitted by `httpd_resp_send` and per `httpd_resp_send_chunk`, **before** handler return/body purge/cleanup. Send/receive overrides belong to HTTPS and do not expose a safe completion hook. URI matching and error handlers likewise cannot supply an all-route post-cleanup boundary.
- `esp_https_server/src/https_server.c`: successful `httpd_ssl_open` stores the transport context and installs TLS send/recv/pending functions before synchronous `HTTPD_SSL_USER_CB_SESS_CREATE`. The application callback invalidates any old row for that fd even if the TLS pointer, socket-slot address and counter value were reused. If fd lookup unexpectedly fails, all observations reset conservatively. The close callback remains the diagnostic observer; SDK destruction remains intact. `httpd_ssl_pending` calls `esp_tls_get_bytes_avail` without consuming data; errors can post an existing SDK error event.
- `httpd_sess_trigger_close` resolves fd to a raw reusable `sock_db *`, then queues `httpd_sess_close`. Its zero-counter/LRU guard does not prove the same connection still occupies that slot. **Not used by this policy.** Direct owner shutdown retains the slot until SDK read cleanup and has no deferred fd/pointer argument that could later close a replacement.
- `httpd_queue_work` uses `cs_send_to_ctrl_sock` / `sendto` in `src/util/ctrl_sock.c`. The actual generated config leaves `CONFIG_HTTPD_QUEUE_WORK_BLOCKING` undefined/off, selecting the nonblocking queue mode. The idle initializer explicitly rejects builds with that blocking option on. Successful `httpd_stop` waits for `THREAD_STOPPED`, frees HTTPD and ends possible old callback execution; failed stop is not a retirement boundary.
- `httpd_queue_work` uses `cs_send_to_ctrl_sock` / `sendto` in `esp_http_server/src/util/ctrl_sock.c`. The actual generated config leaves `CONFIG_HTTPD_QUEUE_WORK_BLOCKING` undefined/off, selecting the nonblocking queue mode. The idle initializer explicitly rejects builds with that blocking option on. Successful `httpd_stop` waits for `THREAD_STOPPED`, frees HTTPD and ends possible old callback execution; failed stop is not a retirement boundary.
## Ownership and bounded storage
@@ -49,36 +49,13 @@ Detach first prevents submissions, then waits at most one second for any submitt
`src/web_httpd_adapter.{c,h}` defines the six-row bound and 15-second policy and performs the version-pinned owner sweep. `src/CMakeLists.txt` adds only the new module. Diagnostics remains unchanged internally; server composes idle identity reset followed by existing diagnostic publication.
## Validation and resource accounting
## Regression and resource limits
Commands executed successfully:
`tests/web_httpd_idle/run.py` exercises production lifecycle/sweep paths, installed SDK request cleanup, host socketpair/readiness/shutdown/fd reuse and deterministic TLS/timer/queue doubles. `tests/web_admin_transport/server_lifecycle.py` checks server composition. Historical host validation passed; these references do not claim a new run or real target scheduler/TLS timing.
```sh
python3 tests/web_httpd_idle/run.py
python3 tests/web_admin_transport/server_lifecycle.py
python3 tests/web_diagnostics/run.py
python3 tests/web_cookie_auth/run.py --accounts
python3 tests/web_cookie_auth/run.py --admin
python3 tests/web_cookie_auth/run.py --settings
python3 tests/web_cookie_auth/run.py --serial-settings
python3 tests/web_admin_transport/run.py --tickets
python3 tests/web_session_store/run.py --serial
python3 tests/web_ui_session/run.py
python3 tests/admin_console_boundary/run.py
python3 tests/admin_console_boundary/accounts.py
python3 tests/admin_console_boundary/lifecycle.py
python3 tests/admin_ssh_policy/run.py
pio run
git diff --check
```
Bounded storage: six observation rows (144 bytes), one persistent timer (installed non-profiled layout 32 bytes before allocator overhead), lifecycle/generation flags and one reserved probe. SDK control UDP/mailbox allocation is transient and separate. No per-connection payload/task/stack/socket growth. Actual HTTPD/timer margins and runtime overhead/reserve floors require target measurement, not host sizes. See [latest firmware resources](web_administration_acceptance.md#latest-firmware-evidence).
- Idle suite: **18 lifecycle groups plus SDK/source guards**. Compiles complete production lifecycle module and production sweep, with installed SDK `httpd_sess_process`/`httpd_req_delete`. Uses real host `socketpair`, readiness, shutdown, EOF and fd reuse; TLS, parsing/response callbacks, timer/queue scheduling and session deletion are deterministic doubles. Tests cover full slots, never-used post-TLS idle, five-second polling, pending/readable/error input, slow synchronous handlers/purge, failed requests, upgrade/async/closing-WS exemption, exact fd/TLS/slot/counter reuse, queue failures, early callback return, submit-fence timeout, failed stop, discarded/stale generations, accepted-but-lost UDP work, diagnostic-independent enforcement and timer initialization/restart/exhaustion. No real TLS/ESP-IDF scheduler or target timing claim.
- Server lifecycle **18 groups**, diagnostics **12+1**, admin transport **25** / tickets **12**, browser **68 plus renderer/CSP**, and the listed auth/store/canonical console/account suites pass. The cookie harness required one missing `<stdint.h>` include after its private-layout double was expanded; fixed, all listed variants rerun successfully.
- Production build: **57.55 seconds**, **99,316 B RAM / 1,708,981 B flash**, **+160 / +1,384 B** versus the recorded diagnostic baseline (99,156 / 1,707,597). Final no-op confirmation `pio run` also passed in **3.13 seconds** with identical sizes.
- Target object symbols: rows **144 B**; module static symbols **167 B total before linker placement/alignment**. Linked RAM delta need not equal the sum of new symbols. One persistent internal-memory ESP timer allocation: installed non-profiled layout **32 B before allocator overhead**. Control UDP packet/mailbox transient allocation is SDK-owned, not included in static RAM. No task/stack/socket/queue-capacity increase.
- Target disassembly local frames (exclude callees): sweep **80 B**, idle work/timer **32/32 B**, prepare **64 B**, attach/detach/stopped **32 B each**, TLS idle callback **48 B**, composed TLS callback **32 B**. Actual HTTPD/timer stack margins, heap minima, TLS churn/latency and runtime reserves are **unmeasured/unapproved**.
## Target checklist — not performed
## Regression procedure — not execution evidence
1. Start with diagnostic capture disabled: establish two serial WebSockets and ordinary HTTPS fetches. Verify idle ordinary sockets disappear after the observation window while both serial clients/lease remain unchanged. Enable capture only as needed to compare close/open occupancy; ages remain connection ages, not idle timestamps.
2. Reproduce the original ordinary4/serial2 full-slot case; wait beyond the idle window, then issue/open admin. Record client ticket/TLS/upgrade timings and occupancy without recording tickets/cookies. Verify no repeat reload loop is needed for already-old idle saturation. Separately test fresh saturation and acknowledge the existing 15-second browser timeout limit.