ESP32_Serial_Swiss_Army_Knife/docs/phase8d3_implementation.md

Phase 8D.3 — Live browser authentication cutover

Status (2026-09-05): Implemented / host-tested / build-verified; both-role login and mixed-client target samples received, M1 validation in progress. The Origin-null login blocker is resolved by user-confirmed successful login after the fix; no full M1 sign-off is implied. Resumed another agent's uncommitted implementation, verified it against installed ESP-IDF, fixed pending-buffer cleanup, and refreshed documentation. Earlier parser/login-renderer records are historical preparatory checkpoints. 8D.0–8D.2 user sign-offs stand; numeric reserve gates remain open. Do not start 8D.4 before M1 sign-off or an explicit user decision.

Successful post-fix target login and mixed-client sample (2026-09-05)

The user explicitly reports successful HTTPS login as both commander1024 (user) and admin after the Origin-mode correction. This supersedes the earlier login-blocker statements below. The following are user-provided sequential snapshots, not agent-executed tests or an atomic measurement. Exact flashed revision/hash and settled duration for this new sample were not supplied.

Heap	Settled boot free / minimum / largest (B)	Mixed load free / minimum / largest (B)
Internal 8-bit	71,204 / 66,752 / 31,744	33,868 / 13,756 / 25,600
Internal DMA	63,448 / 58,996 / 31,744	26,112 / 6,000 / 25,600
PSRAM	8,247,744 / 8,245,836 / 8,126,464	8,089,060 / 8,072,744 / 7,995,392
SSH stack minimum-free	18,464 B (20,480 B configured)	16,288 B

Settled boot: UART service stopped/owner idle at 115200 8N1/no flow; no broker clients or SSH sessions. USB initialized/attached but host closed/DTR false. HTTPS ready, sessions/challenges/tickets zero, all supplied web and SSH request/traffic/failure counters zero. mDNS initialized/announced as sak-1024.local, last error ESP_OK. The USB host's diagnostic 9600 coding does not configure UART1.

Mixed-client load: UART running at 115200 8N1/no flow. Four broker clients: web user 8 observer, user SSH 9 sole writer, USB 10 observer, web admin 27 observer. Public-key user/admin SSH both active (2/2); admin SSH has no broker client. Two password-authenticated web serial sessions and two cookie sessions, no outstanding challenges/tickets. USB host open/DTR/RTS asserted; diagnostic line coding 115200. RX available/TX pending zero at the serial snapshot; all four displayed broker pending/event queues zero. mDNS remains announced without errors.

• Two consecutive loaded memory samples are identical. This is short-term observation, not leak/soak/cleanup or reserve-floor validation. Lifetime minima include handshake/earlier activity; DMA overlaps internal heap. The 6,000 B DMA minimum leaves runtime reserve analysis important even though current DMA free is 26,112 B. No stack fault/watchdog or memory exhaustion is reported.
• SSH: two successful handshakes/auth attempts, no handshake/auth/I/O/session-revocation failures. Stream RX 73 accepted, zero rejected, TX 31,941 B. Broker initial writer request was denied once; later writer snapshot shows SSH owns the lease (no inconsistency inferred from cumulative counters). Admin command-running/output-pending fields were sampled while executing status commands, not proof of a stuck dispatcher.
• Web: 97 protected requests, 96 authenticated and one auth failure; four roots, 80 status, four tickets, eight assets; zero response errors. Four tickets issued/consumed, zero rejected/expired. Four serial connects/two disconnects; two currently active. RX one accepted frame/14 B with no rejection; TX 495 binary frames/43,738 B and 15 control frames/1,255 B. Writer requests four, grants one, denials three, releases one; no revocations. No reported send/queue/protocol/service-start/broker/connection failures.
• Cookie auth: six password checks, three invalid-credential results, zero throttle/capacity/CSRF-or-Origin rejections, one logout, two active sessions. Successful both-role login is explicit user confirmation; the counters also show logout/reconnection activity but do not establish five cycles, account-isolation coverage or logout acknowledgement delivery. The three invalid-credential results and one protected-request auth failure are retained without attributing a cause.

Reported admin-SSH empty-line issue: pressing Enter without text prints “Command is restricted to physical UART0.” Source trace identifies remote_command_allowed() classifying zero parsed arguments as a policy denial. Corrected that helper to allow empty input to reach IDF's normal quiet ESP_ERR_INVALID_ARG handling; existing current-admin/generation checks still run, and user bootstrap/user recover remain denied. No new dispatcher, route or 8D.4 refactor. python3 tests/admin_ssh_policy/run.py passes 15 policy cases with the production helper and installed IDF argument parser, including quoted restricted commands. pio run passes in 21.04 seconds, 95,508 B RAM / 1,625,725 B flash (+20 B flash versus Origin fix). This SSH fix is not yet target-tested and was not present in the user's sample. No upload/erase/commit performed.

Remaining: explicit M1 sign-off, repeated lifecycle/expiry/revocation/isolation and raw-client security checks, timed full-load soak and settled cleanup, numeric reserves and non-SSH owner stack margins. Do not reopen prior phase sign-offs or invent missing execution evidence.

First target sample and login blocker (historical, user-provided, 2026-09-05)

After clean boot and 60 seconds settled, the user reports:

Heap	Free	Minimum-free	Largest block
Internal 8-bit	69,004 B	66,752 B	31,744 B
Internal DMA	61,248 B	58,996 B	31,744 B
PSRAM	8,223,116 B	8,218,204 B	8,126,464 B

SSH has 0/2 sessions, all supplied error/traffic counters zero, configured stack 20,480 B and minimum-free 18,464 B. HTTPS is running/ready with no lifecycle/response errors, cookie sessions 0/4 and challenges 0/4; eight protected requests were unauthenticated, with zero password-verification attempts, CSRF/origin rejections, logouts, tickets or serial sockets at this sample point. mDNS announces sak-1024.local. UART1 is stopped/owner idle, configured 115200 8N1/no flow; no broker clients. USB is initialized/attached but host-open/DTR false, broker disconnected. Its reported 9600 host coding does not configure UART1. These snapshots precede the reported login attempts; they do not establish post-attempt counters. Exact flashed revision/hash was not supplied.

M1 is blocked: subsequent login attempts for a normal user and administrator both show “The sign-in challenge expired or the request was rejected. Please try again.” Browser console reports blocked inline scripts with two hashes different from the application's login script hash, denied favicon by default-src, and a denied file URL. No browser login, loaded-memory or full M1 acceptance is claimed. These errors do not establish memory exhaustion or invalid passwords.

The production-renderer suite was rerun and the exact shipped inline-script hash still matches its CSP (x70ID2kbifGBVYfh/pePTt5v/AVHkT7JVAV0LjT1wCo=). The displayed login message maps to HTTP 403 in the running script; the console's other hashes may be injected-script warnings, not a reason to broaden CSP. Request-stage/status and the bounded error code plus nonsecret Origin/Fetch Metadata are needed to isolate the rejection. No corrective firmware change has yet been made for this target report.

Confirmed Origin-null diagnosis and correction (2026-09-05)

Follow-up user evidence: /api/login-challenge returns 200, /api/login returns 403 with request Origin: null, Sec-Fetch-Site: same-origin, and the pre-login cookie present. web status reports ready, zero sessions/challenges/tickets, zero password-verification attempts and seven CSRF/origin rejections. This confirms the rejection occurs before password authentication; it is not evidence of wrong credentials. No secret values were requested or retained.

Cause: the authored fetch requests used non-CORS mode: 'same-origin' under Referrer-Policy: no-referrer; browser Origin-header serialization for these POSTs yields null. Corrected login fetch options to mode: 'cors' and the existing app API helper to use cors for POST (ticket/logout), retaining same-origin mode for app GETs. Fetch CORS mode is not permission for cross-origin service access: paths remain fixed same-origin endpoints, credentials remain same-origin, redirects remain rejected, CSP connect-src 'self' remains intact, and the server's strict Origin/CSRF checks/no-CORS-response policy are unchanged. The login script hash was updated atomically to eZO4pMDQx6SIaa5AFlMnuf0CD5JdGSWyi8lNVmCNPBQ=; existing app loader hash is unchanged because only its external app script changed.

Validation: login renderer/CSP eight Node groups, app nine Node groups, and cookie-policy suite all pass. Node guards assert CORS mode for every mutation (including logout), fixed endpoint destinations and no manually supplied Origin. They do not synthesize real browser Origin headers; Firefox/target retest is still required. pio run passed in 14.30 seconds, 95,508 B RAM / 1,625,705 B flash (RAM unchanged, flash +16 B versus the preceding live build). No upload/erase. Retest both roles, serial Connect/Disconnect/reconnect and Sign out; expect login POST Origin https://sak-1024.local (or the actual direct-IP origin), not null. M1 remains blocked until confirmed on target; other CSP warnings were not loosened or assumed resolved.

Delivered behavior

• web_cookie_auth.{c,h} replaces Basic authentication/cache completely. Both roles use /login and the same serial/status application. No admin shell/settings routes were added. Previously cached Basic headers do not authorize a request.
• Four digest-only authenticated sessions retain the existing store's copied principal, canonical-origin binding, CSRF state, non-reused ID and one-hour absolute lifetime. Traffic/polling does not renew expiry. Failure to initialize authentication prevents HTTPS start; UART0/USB/SSH implementations remain unchanged.
• Four 120-second pre-login challenges contain only token/origin digests, CSRF state and expiry. Explicit login bootstrap requires X-Login-Bootstrap: 1; a matching live challenge is reused without extending its lifetime or resetting its cookie. Credential submissions consume the challenge, including wrong passwords. A global fixed window permits five password verifications per 60 seconds, including successes. Further attempts return 429 with Retry-After; no HTTPD sleep or per-IP/account table.
• Session and pre-login cookies use __Host-sak-session / __Host-sak-prelogin, Secure; HttpOnly; SameSite=Strict; Path=/, explicit Max-Age 3600/120 and no Domain. A consumed challenge expires its cookie; successful login additionally sets a fresh session cookie. Login with a current authenticated cookie returns 409; account switching requires logout.
• Mutations require canonical same-origin HTTPS Origin and CSRF; upgrade requires Origin and matching cookie/session/ticket. Host case and optional default port 443 normalize; non-443 ports, malformed authorities and IPv6 literals are rejected. Direct-IP and mDNS names remain distinct cookie origins. Cross-site/same-site Fetch Metadata requests are rejected (same-origin/none accepted); absent Origin is permitted only on read/bootstrap requests after Host validation.
• Exactly username/password string fields, maximum 512-byte login JSON, decoded 16/64-byte limits. Unknown/duplicate fields, NUL and malformed Unicode fail. Header/body/request scratch is wiped; rejected unread bodies close instead of invoking HTTPD's automatic body drain. Login reads have a three-second application deadline plus existing socket wait bounds. API authentication responses are at most 512 bytes; safe username JSON encoding is explicit.
• Full live session/challenge/ticket tables reject with 503 and Retry-After 5; serial-ticket earliest-expiry eviction is removed. Expired/stale tickets are reclaimed without database calls under the transport lock. Existing two serial sockets, one-writer broker model and binary protocol are unchanged.
• Logout invalidates its originating session before acknowledgement and requests only its ticket/socket cleanup. Account mutation/revocation continues to invalidate all affected account sessions, not unrelated accounts. Mint/consume/admission/input and existing periodic owner checks remain authoritative if notification fails. Zero session ID no longer falls back to Basic.
• Browser validates /api/session before initial connect/reconnect/restore; stores CSRF only in memory; adds Sign out and visible absolute expiry. 401 cancels work/closes local serial/navigates once to /login; explicit Disconnect still pauses reconnect without ending login. 403 mutation failures require explicit retry; capacity/backoff and network errors are not bad credentials. Lost logout response is reconciled with session status rather than claiming success. Pending fetch/socket callbacks are generation-guarded. Both authored inline scripts have exact CSP hashes; generated assets were not regenerated.

Route boundary

Route	Policy
GET /login	Public standalone no-store login page, no protected assets
GET /api/login-challenge	Validated Host, bootstrap header, Fetch Metadata and any supplied Origin
POST /api/login	Strict Origin, pre-login cookie/CSRF, bounded JSON and throttle
GET /api/session	Current cookie session; username/role/CSRF/remaining seconds only
POST /api/logout	Current session, strict Origin/CSRF, empty body
GET /	Current session; unauthenticated navigation gets 303 /login
GET five /assets/ routes; GET /api/status	Current session; unauthenticated gets 401, not login HTML
POST /api/ws-ticket	Current session, strict Origin/CSRF, empty body
GET /ws/serial?ticket=...	Cookie/Origin authorization and ticket/principal/broker admission before explicit 101

No CORS/preflight compatibility or Basic fallback. Query strings outside the exact serial-ticket route and wrong methods are rejected. Error routes have bounded no-store responses. The login document itself also rejects malformed/duplicate cookies; manually corrupted cookies may require clearing those site cookies, unlike ordinary expired well-formed cookies.

Verified HTTPD boundary and maintenance risk

The delivered solution is not the previously proposed SDK patch. web_httpd_adapter.{c,h} alone includes installed HTTPD private headers. src/CMakeLists.txt supplies private include paths; the adapter fails compilation unless ESP_IDF_VERSION == 5.5.0. No installed SDK source was changed and no full component was vendored.

Verified under /home/mscholz/.platformio/packages/framework-espidf/components/esp_http_server/:

• src/httpd_parse.c, httpd_req_get_hdr_value_len/str: return the first matching header only. Parsed fields occupy NUL-separated scratch, not a raw CRLF block. Adapter walks that bounded storage and rejects all duplicate field names, case-insensitively, plus malformed fields, control characters, Transfer-Encoding and Expect. This is stricter than general HTTP acceptance, deliberately fail-closed. Public getters are called only after validation and with terminator capacity.
• src/httpd_txrx.c, httpd_resp_set_hdr: appends pointers, does not replace an earlier same-name field. Sending emits each entry; login retains its two cookie values until send returns. Success uses exactly six of eight additional-header slots. Tests extract the installed getters and append function rather than inventing their behavior.
• src/httpd_uri.c: routes marked is_websocket=true send 101 before their handler. The application's serial URI is deliberately registered as an ordinary GET. After cookie/Origin checks, transport consumes the matching ticket and completes currentness/broker admission, then adapter calls httpd_ws_respond_server_handshake() and installs the existing transport frame handler. Failed pre-admission never sends 101; handshake/admission failure uses existing reserved-slot/broker cleanup. Tests stub the handshake send: real on-wire integration remains a target gate.
• src/httpd_txrx.c, httpd_unrecv/httpd_recv_pending: pending bytes are right-aligned. The inherited adapter incorrectly wiped the unread suffix. This continuation fixes cleanup to wipe the consumed prefix while preserving unread bytes at the end, or wipe everything when closing. Regression exercises all 0–128 pending lengths and partial reads through the installed reader function. This prevents corruption of pipelined requests/early serial frames; it is not a claim of real socket execution.
• HTTPD DEBUG logs include headers, and URI warnings can include ticket queries. HTTPD is compiled with LOG_LOCAL_LEVEL=ESP_LOG_ERROR; ERROR sites were inspected for secret-bearing content. This deliberately removes HTTPD warning/debug diagnostics regardless of runtime log-level changes. Application count-only authentication telemetry remains available via web status/web counters.

Private layout, frame dispatch and scratch ownership must be re-audited for an SDK update, including same-version local source patches (the guard checks the version, not source hashes). Do not distribute private-structure access into other application modules. Wiping reduces request lifetime, not all TLS/allocator/browser copies; do not claim resistance to RAM extraction.

Resource accounting

Final pio run passed in 17.62 seconds after the cleanup fix:

Metric	8D.2 / preparatory baseline	Live 8D.3	Increment
Linked static RAM	95,260 B	95,508 B	+248 B
Reported program flash	1,601,925 B	1,625,689 B	+23,764 B

Cumulative versus recorded 8D.0 build (94,532 / 1,599,973 B): +976 B RAM / +25,716 B flash. These are linked sizes, not runtime headroom.

• Target object symbol accounting: challenges 576 B (144 × 4), counters 32 B, lock 8 B, epoch 8 B, window 8 B, attempts 4 B, ready 1 B: 637 B before placement padding. Removed Basic cache/key/readiness offset most of this; final link delta includes alignment/other changes. Existing session store remains present.
• No new application task, task-stack size change, module heap allocation, queue, TLS buffer, accepted socket or lwIP descriptor limit. HTTPD URI capacity rises 9 → 14, with five additional dynamically allocated handler records; HTTPD error handlers use its existing table. Six HTTPS clients and two web serial slots remain unchanged. LRU purge remains enabled; retained-serial admission protection is still an M2 concern.
• Auth request locals include 513 B body/response scratch, 180 B cookie header, token/CSRF/canonical buffers, copied session/principal/challenge/credentials; cookie parsing has nested 1025 B header scratch. No task-stack reserve is inferred from source locals or static link size. Existing HTTPD stack is 10,240 B; real worst-case stack/TLS/PBKDF2/fragmentation measurements remain pending.
• Existing xterm/logo data unchanged. Login page and enlarged authored app are now actually linked; their dormant-preparation flash numbers were not their live cost. Header slots remain eight; login success six, login renderer five.

Executed validation

All ran successfully in this continuation:

python3 tests/web_cookie_auth/run.py
python3 tests/web_auth_parse/run.py
python3 tests/web_login_ui/run.py
python3 tests/web_ui_session/run.py
python3 tests/web_session_store/run.py --serial
pio run

• Cookie policy suite compiles production store/parser/policy/adapter with OpenSSL SHA-256 and deterministic database/HTTPD doubles. Covers fragmented reads, challenge reuse/consumption/expiry/capacity, session-specific logout, throttle, duplicate headers/cookies, methods/Origin/CSRF/Fetch Metadata, Basic denial, currentness, failures/stop race, cookie header budget and explicit upgrade state. Installed IDF getter/setter/pending-reader functions are extracted verbatim. It does not execute the full IDF parser, TLS, URI dispatcher, network handshake or real tasks.
• Parser suite: 268 cases. Login renderer: production C failure/header checks and eight Node groups. Serial app: production C resource/header/CSP checks and nine Node groups. Node VM DOM/fetch doubles are not a real browser/CSP/bfcache test.
• Serial integration mode includes store public-API tests plus transport binding/isolation/races and no Basic/no live-ticket eviction. No sanitizer pass is claimed.

No upload, erase, commit, branch change or target/browser exercise was performed. The preceding agent's changes were preserved except the focused pending-buffer fix/tests; its unrecorded executions are not evidence here.

M1 target acceptance handoff — stop before 8D.4

Use the complete M1 contract/checklist and user acceptance matrix. At minimum:

1. Keep UART0 attached. Record flashed revision/configuration and settled-boot memory, web status, web counters, broker clients, ssh status. Confirm native USB and both SSH roles survive HTTPS stop/start and authentication failures.
2. Test both roles, fresh and previously Basic-authenticated profiles, direct IP and mDNS. Wrong credentials, refresh/back, expiry/reboot, sign out/account switch and lost logout response must remain usable. Verify actual secure cookie attributes and CSP; never include raw cookies/CSRF/tickets/passwords in shared evidence.
3. Five login/serial-disconnect/reconnect/logout cycles per role; five HTTPS stop/start cycles. Check session-specific logout across two sessions of the same account, and account password/role/key changes/deletion/recreation via UART0 while unrelated sessions survive.
4. Challenge/session/ticket capacity without eviction; bounded throttle and retry. Raw-client missing/malformed/duplicate Origin/Host/Cookie/CSRF/content-type/framing tests. Verify an unauthorized or mismatched-ticket upgrade gets no 101; validate actual frame routing, early/pending bytes and close cleanup after admitted upgrades. These are especially important for the private adapter.
5. Fifteen-minute full-client mix at 115200 baud (USB, two web serial clients, user SSH and admin SSH), then 60-second cleanup. Record internal/DMA/PSRAM free/minimum/largest block plus SSH stack margins at boot/login/serial/load/cleanup. Check binary integrity, writer isolation, drops and watchdogs. Measure the planned ≤1-second expiry/revocation detection target under contention separately from socket-close delivery.
6. Numeric reserve floors and non-SSH owner-stack instrumentation remain pending. Obtain explicit M1 sign-off before adding the browser admin shell. Do not equate host tests/build success with target acceptance.