ESP32_Serial_Swiss_Army_Knife/docs/phase8d_plan.md

Phase 8D — Incremental web administration plan

Status: planned; no implementation or validation is claimed by this document.

This is the execution plan for roadmap Phase 8D. The roadmap retains the feature/security requirements; this document defines small work units, dependencies, and release gates. The administration test matrix remains the final acceptance checklist.

Why this phase is split

The previous all-in-one attempt was rolled back from devel. The user reports a roughly +10k/-1k-line change, repeated agent/time/context interruptions, incomplete validation, broken browser login, and severe memory pressure. These are reported symptoms, not a diagnosed root cause. The separate experimental branch is not the implementation baseline: do not merge/cherry-pick it wholesale or copy its abstractions without a separately scoped review.

Deliver three independently useful milestones before attempting feature completeness:

1. M1: reliable browser login/logout and the existing serial UI — 8D.0–8D.3.
2. M2: browser admin shell alongside uninterrupted serial access — 8D.4–8D.7.
3. M3: guided administration, one settings domain at a time — 8D.8–8D.21.

8D.22 is final integration acceptance, not the first time anyone builds or tries the firmware. Stopping after M1 or M2 is valid incremental delivery, but does not mean all of Phase 8D is complete.

Work-unit rules

• One numbered chunk per implementation request. Do not interpret “continue Phase 8D” as permission to implement all remaining chunks. Select the first unblocked chunk and state its scope before editing.
• Each chunk should fit one normal agent session, including review, a bounded build, focused validation, and handoff. Plan for roughly 60–90 minutes of implementation and reserve at least the final third for validation/documentation. These are scope limits, not runtime guarantees; split further before coding if the estimate does not fit.
• Aim for a few hundred changed authored lines in a small source set. An expected change above roughly 600–800 authored lines, more than one new task, or several independent subsystem changes triggers a scope review and another split. This is not an incentive to compress code, omit tests, or hide generated changes.
• Declare allowed files, behavior changes, explicit exclusions, resource deltas, and acceptance checks first. Source sets below are starting points, not permission to refactor every listed module. New files must have a narrow responsibility justified by that chunk.
• Build the smallest complete increment. Internal preparatory chunks may leave unused interfaces, but must not expose half-protected routes. Do not ship an intermediate cookie-authentication route without CSRF/origin protection, currentness checks, expiry, and logout cleanup.
• Reuse subsystem APIs, the existing HTTPD ownership model, the canonical command dispatcher, and authored UI assets. Do not introduce a generic web framework, CLI-over-HTTP endpoint, second command registry, second dispatcher, task-per-request model, or speculative settings infrastructure.
• Do not rename/extract the entire admin_ssh_console module just to give it a generic name. Adapt the smallest necessary boundary and retain existing SSH callers. Do not regenerate src/web_assets_data.* or update vendored xterm dependencies as incidental work.
• Add focused counters and checks with the feature that needs them, not as a final observability project. New tests must be local/bounded and document their exact command; this repository currently has no automated host test command. Avoid scaffolding a general test platform as part of implementation.
• Run pio run with a finite tool timeout. A timeout is incomplete validation, not a pass; record it and stop rather than starting overlapping/repeated build jobs. Fix only failures caused by the chunk.
• End with a reviewed diff and handoff: implemented behavior, build result, memory delta, checks actually run, hardware checks pending, and exact next chunk. Keep docs/agent/current-state.md current. Never include credentials, cookies, CSRF values, tickets, or verifier material in evidence.
• A chunk is implemented / build-verified / target-verified as separate states. If device access is unavailable, provide the small manual checklist and mark it blocked for target validation. Do not proceed past an M1/M2 gate, or stack further runtime-changing chunks on an unverified predecessor, without an explicit user decision. Never silently promote documented tests to passed tests.
• No automatic commits, branch changes, uploads, erase, NVS migration, or imports from the abandoned branch. The user chooses commit/revert checkpoints; preserve independently reviewable diffs.

Baseline and resource gates

8D.0 must record the actual baseline revision and configuration before resource budgets become acceptance criteria. The current memory notes report 94,532 bytes linked RAM and 1,599,765 bytes flash for an mDNS-enabled build; these are historical reference values, not a fresh measurement or sufficient runtime headroom.

For every chunk that changes allocation, concurrency, or routes:

• Inventory static internal RAM, normal and worst-case internal heap, PSRAM, task count/stack sizes, request/response scratch buffers, queue depth, HTTPD handlers, sockets/TLS connections, and session/ticket capacity. Record limits before increasing any of them. Cookie sessions, serial sockets, admin sockets, and outstanding tickets are different resources.
• Use UART0 memory to record internal/PSRAM free, minimum-free, and largest-block values at the same defined workload points: settled boot, login, serial connected, admin connected when available, full supported concurrent workload, and after repeated close/logout/reconnect. Compare both each chunk and cumulative growth against the baseline. Collect relevant task stack high-water marks where available; explicitly record missing instrumentation.
• Set numeric internal-heap/largest-block/stack reserve floors and per-chunk incremental budgets from baseline measurements and actual allocation sizes in 8D.0. No invented “safe free heap” constant and no percentage-of-total-RAM substitute. If the floor cannot be measured, mark the gate pending rather than guessing.
• Account for two simultaneous browser WebSockets when serial and admin coexist, plus HTTPS requests and existing SSH connections. Capacity rejection must be explicit and must not silently evict an active serial client/writer to admit an admin socket. Do not merely raise HTTPD/lwIP limits until a page happens to work.
• Prefer bounded PSRAM payload storage only where cache-disable/lifetime constraints allow it; retain required internal control structures/stacks. Record fallback behavior: opportunistic internal fallback must not consume the recovery reserve unnoticed. Optional web-admin allocation failure must leave current serial, UART0, USB, and SSH paths usable.
• Do not accept a monotonic heap leak, declining largest-block trend, watchdog/stack fault, unexplained reserve-floor violation, or unrecoverable login failure. Stop and fix/split the current chunk instead of borrowing from future budgets. Supported baud rates and client capacities must not be silently reduced to pass.

Always-on regression smoke check

After each runtime-changing chunk: build; boot and read UART0 status/memory; log in through the currently supported browser authentication path; open serial, explicitly disconnect and reconnect; exercise native USB UART1 access and existing user/admin SSH routes. Add the chunk-specific checks below. Use a fixed small repetition count selected in 8D.0 for routine lifecycle checks and a longer bounded soak at milestone gates; record actual counts and durations, not just “stable.”

M1 — Authentication without changing the rest of the UI

8D.0 — Baseline, browser contract, and resource budget

Scope: Documentation and measurement only. Read the relevant web_server, web_serial_transport, web_ui, user-principal, and memory-reporting paths. Confirm current login/serial operation on rolled-back devel; no investigation of the experimental branch is required.

Deliver: A short baseline record, measured resource table, supported concurrency/socket budget, and exact M1 browser contract: public login assets; protected routes; session/ticket capacities and lifetimes; absolute/idle expiry rules; capacity rejection; cookie renewal; pre-login CSRF bootstrap; strict origin policy; no-store responses; logout-versus-account-wide revocation; and browser error handling. Preserve mDNS and direct-IP access as separate host-only cookie origins. Choose the simplest bounded policy, with no Basic compatibility path by default. Explicitly list the few authentication request/response fields rather than designing all future settings APIs.

Gate: Existing admin and user browser login, serial data, USB, and SSH work; baseline pio run and target memory evidence are recorded. If the rolled-back baseline already fails login, diagnose that in a separate task before changing authentication. If target evidence is unavailable, the budget and runtime gate remain pending.

8D.1 — Bounded server-side session primitives, not yet browser-facing

Start in: src/web_server.{c,h}, src/user_database.h, src/secure_random.h; add a narrowly owned session module only if needed.

Scope: Fixed-capacity session issue/lookup/expiry/invalidation with digest-only token storage, copied principal and authentication-generation binding, CSRF state, and secret-free capacity/expiry/invalidation counters. Decide ownership/locking explicitly because future console revocation is not necessarily on the HTTPD task. Wipe transient secrets and use existing secure randomness. No login page, HTTP auth cutover, admin route, new permanent task, or settings work.

Gate: Focused local/component checks for lifecycle, capacity, slot reuse, stale principals, and failed initialization; bounded storage accounting and build. Existing Basic-auth behavior remains unchanged. If no executable harness is practical, distinguish code review from target execution and carry the missing checks into 8D.3; do not claim unused code was exercised by a boot test.

8D.2 — Bind existing serial tickets and sockets to a web-session identity

Start in: src/web_serial_transport.{c,h}, src/web_server.{c,h}, the session module from 8D.1, and the existing revocation call sites in src/user_console.c.

Scope: Add generation-safe originating web-session identity to serial tickets/slots, specific-session cleanup, and account-wide invalidation hooks. Keep web-session identity distinct from account authentication generation and transport slot generation. Preserve authoritative currentness checks when best-effort notification fails. Keep the current Basic path working until the atomic cutover; no public cookie-auth route yet.

Gate: Build and existing serial regression. Exercise session-specific versus account-wide cleanup through focused checks where available: one session's logout must not disconnect another session for the same unchanged account; account mutation must invalidate all affected account sessions/tickets. Stale cleanup cannot close a reused slot. Record any dormant paths that require the next chunk's browser validation.

8D.3 — Atomic login/logout cutover with the unchanged serial application

Start in: src/web_server.{c,h}, src/web_ui.{c,h}, session primitives, and the session-bound serial integration.

Scope: Minimal same-origin login page, session status, explicit logout, bounded login throttling, and cookie-based authorization of the existing app/status/ticket routes. Use a host-only __Host- cookie with Secure, HttpOnly, SameSite=Strict, Path=/, no Domain, and explicit lifetime. Enforce the agreed pre-login CSRF protection plus strict origin checks on login, and session CSRF plus strict origin checks on every authenticated mutation including logout and ticket issuance. Keep passwords/tokens transient, preserve security headers/CSP, update the loader hash atomically if it changes, and handle JSON/text safely. Remove browser Basic challenges/cache authorization so stale credentials cannot bypass logout. No admin shell, settings, visual redesign, or new serial protocol.

Gate — M1 (mandatory target/browser pause): Both roles log in; incorrect credentials give a usable error; logout and account switching need no browser credential-cache reset. Refresh/reboot/expiry return to a usable login; no redirect loop or missing login asset. Test fresh and previously Basic-authenticated browser profiles, direct IP and mDNS where available, bounded capacity/backoff, CSRF/origin rejection, current-session logout cleanup, password/role/key changes and deletion/recreation via UART0, and unrelated-session/account isolation. Existing serial data, writer ownership, and explicit reconnect still work. Record memory before/during/after repeated login/serial/logout and simultaneous SSH/USB operation. Stop here for user confirmation before M2.

If 8D.3 exceeds the work-unit limit, first split out inert login-page rendering or private request-parsing helpers. Do not split the live security cutover into an insecure intermediate deployment.

M2 — Reuse the admin shell, then expose it

8D.4 — Small transport-neutral console boundary

Start in: src/admin_ssh_console.{c,h}, src/console_input.{c,h}, and only the necessary src/ssh_transport.c callers.

Scope: Introduce the minimal transport identity/output/lifecycle boundary needed by a future web frontend. Retain the single dispatcher, fixed queue, line editing/history/completion/prompts, currentness checks, generation tokens, and existing SSH API compatibility where practical. Do not duplicate per-SSH buffers for hypothetical web capacity or rename the whole module. No web endpoint or second dispatcher/task.

Gate: UART0/admin-SSH serialization, hidden prompts, completion/history, deferred actions, disconnect/revocation with queued work, and slow-output behavior regressions pass. Memory delta is explained before adding browser slots. This refactor must stand alone and leave behavior unchanged.

8D.5 — Bounded admin WebSocket backend, no normal UI entry yet

Start in: The console boundary from 8D.4, src/web_server.{c,h}, and a small web-admin transport adapter as justified. Reuse existing HTTPD scheduling patterns without mixing admin data into the serial transport.

Scope: Admin-only, short-lived single-use tickets bound to both current web session and principal; bounded console admission/input/output; session expiry/logout/revocation cleanup. HTTPD owns socket work and the dispatcher owns command execution. No broker client for this route. An absent UI is not authorization: every ticket, upgrade, and sensitive operation is checked on the server. For self-affecting web actions not safely supported yet, explicitly reject before side effects and list the temporary restrictions for 8D.7.

Gate: Focused authenticated test-client or temporary local development-page checks (not a shipped debug endpoint): user-role rejection, admin command/output, prompt/backpressure, stale ticket/slot rejection, cleanup, and concurrent UART0/admin SSH. Admission failure does not remove the serial writer. Build and measure actual socket/console-slot cost. No generic HTTP command runner.

8D.6 — Browser terminal selector and serial-lease preservation

Start in: src/web_ui.{c,h}, using the completed serial/admin protocols.

Scope: Add admin-only Serial/Admin selection and separate bounded terminal state; no new settings. Keep serial connected and drained/observed while hidden. Keep writer/observer identity and Request control/Release control visible in both modes. Mode changes route displayed output and keyboard input only. Closing the admin route or shell exit leaves serial intact; explicit serial Disconnect retains its documented cleanup/reconnect behavior. Bound scrollback and avoid leaking browser listeners/sockets across switches.

Gate: Repeated switches preserve the same broker client ID and writer ID; background serial remains observed, with any bounded overflow visible rather than silently stopping observation. Ordinary users retain serial-only navigation and server-side denial. Exercise hidden prompts, line editing/history/completion, resize, admin-route reconnect, full logout, and expired-session UI. Measure simultaneous two-WebSocket plus HTTPS request headroom; no eviction to make the selector work.

8D.7 — Web-shell lifecycle parity and M2 acceptance

Start in: The console deferred-control boundary, web lifecycle owner, and only the affected command handlers.

Scope: Close the explicit 8D.5 restrictions for self-terminating web-shell operations (including HTTPS stop/identity changes and reboot where supported by the canonical registry). Reuse bounded deferred acknowledgement/close semantics; never claim application-buffer draining proves browser receipt. Prevent further input during pending actions. Preserve UART0-only bootstrap/recovery and safe policy for one-time self-generated credentials. No typed danger-zone API yet. If this requires several distinct owner changes, split them before implementation and keep unimplemented actions explicitly rejected.

Gate — M2 (mandatory target/browser pause): Browser/UART0/admin-SSH commands serialize with no output or hidden-prompt crossover; queued work is discarded after logout/revocation/slot reuse. Deferred operations acknowledge or fail within bounds and reconnect behaves as documented. Run simultaneous USB, serial WebSocket, admin WebSocket, user SSH, admin SSH, UART0 and UART1 traffic; verify lease retention, recovery availability, and measured memory/stack floors. Web-admin initialization/admission failure must leave M1 serial login and existing non-web paths usable. Stop for user confirmation before settings.

M3 — Typed settings, one domain per chunk

Every row is a separate implementation request, not a batch. Add only the endpoints and UI needed for that row. All typed routes require current admin authorization, bounded bodies/responses, secret-safe encoding, and the established CSRF/origin/no-store policy. Do not send constructed command strings to esp_console_run().

Typed operations must preserve subsystem owner/lock/persistence contracts and coexist safely with console operations. The current admin_command_gate is a narrow user-command wrapper, not an existing global typed-operation serializer; do not assume it solves concurrency. Specify per-domain serialization and committed-mutation notification before adding writes. Reuse a small common mechanism only when a concrete second caller needs it.

Chunk	Bounded deliverable and starting source set	Focused acceptance gate / exclusions
8D.8 — Read-only settings entry and Serial page	Minimal admin Settings navigation, common bounded error handling, and a typed serial snapshot; start in web_server, web_ui, serial_service.h, serial_config.h.	Normal users are denied by server, unknown/oversized input fails safely, serial values match UART0. No mutations, schema generator, empty placeholder pages, or all-subsystem snapshot.
8D.9 — Serial edits and persistence	Typed framing/lifecycle and explicit apply/save/load/default/reset controls through existing serial APIs.	Invalid framing is rejected; working versus persisted state and stop/reconfigure data-discard effects are explicit. Writer ownership semantics stay unchanged; compare CLI/browser edits and reboot persistence. No quick popover yet.
8D.10 — Accounts and passwords	User list/create/delete/role/password workflows, final-admin protection, one-time generated-password handling; start in user_database, existing user_console mutation/revocation behavior, and web handlers/UI.	No raw database export, verifier fields, or UART0 recovery endpoints. Commit/invalidation behavior matches CLI, unrelated users remain connected, own-account changes have safe reconnect/credential-delivery semantics, secret fields are cleared after use. If CRUD and password UX exceed budget, split at read-only/role/delete versus create/password before editing.
8D.11 — SSH authorized keys	List fingerprints and add/delete/clear supported public keys through bounded user APIs.	Ed25519/P-256 import, maximum supported length, malformed input, duplicates, targeted revocation, and unchanged SSH authentication behavior. No private-key upload/export or host-identity management.
8D.12 — Network settings without secret mutation	Secret-free STA/AP/profile and mDNS settings, non-secret edits, and explicit persistence through wifi_manager, wifi_config, mdns_service/mdns_config.	Responses never serialize saved PSKs; validate working/persisted semantics, live hostname changes, and behavior after connection loss. No new manager/task or Wi-Fi blob migration. Split mDNS into a follow-up if needed.
8D.13 — Wi-Fi secrets and connection controls	Explicit password replacement/clear semantics, bounded transient input, profile selection/reconnect and AP policy actions using manager-owned operations.	Preserve existing secrets when fields are omitted; never prefill saved secrets; document apply/save and likely connection loss; reconnect via STA/AP and verify UART0/USB recovery. No background secret fetch or general credential export.
8D.14 — Display settings	Typed local display configuration and explicit persistence via local_ui_config/public UI APIs.	Validate limits, save/reboot, absent-display behavior, and concurrent buttons/CLI edits. No I2C ownership changes or electrical diagnostics UI.
8D.15 — Bounded network diagnostics	Secret-free network status and a narrowly bounded diagnostic workflow through existing network facilities.	Diagnostic start/result/cancel/expiry behavior and concurrent CLI use are bounded; callbacks do not format/send HTTP directly. No unbounded result/history buffer or new generic jobs framework. Split asynchronous ping from read-only status if necessary.
8D.16 — Broker client visibility and writer transfer	Admin-only detailed client snapshot plus explicit confirmed writer assignment using existing broker APIs; smallest broker change only if authoritative generation-safe validation is missing.	Stale/disconnected/reused target fails without changing the current lease; exactly one writer; normal users cannot obtain management details or transfer. Test concurrent USB/SSH/browser requests. No transfer on page open or selection alone.
8D.17 — Serial/Wi-Fi quick popovers	UI-only reuse of completed typed endpoints, with full-page links and shared validation; start in web_ui.	Hover, focus, click/tap parity, Escape/outside-click dismissal, no mutation on opening, explicit apply/save, no secret exposure. No duplicate backend or new settings scope.
8D.18 — Client/writer contextual dialogs	Reuse 8D.16 for live client popover and confirmed Active writer dialog.	Accessible pointer/keyboard/touch paths; refresh preserves explicit selection safely; stale confirmation is rejected visibly; normal users retain only ordinary status. No new writer policy.
8D.19 — Ordinary service/session controls	Typed service status and targeted disconnect/start/stop controls, excluding actions that cut off the invoking HTTPS session; start in existing service APIs and generation-safe snapshots.	Explicit scope/confirmation, stale target rejection, owner-safe execution and failure isolation. Stop/start and disconnect do not clear settings/identities. Split by service if more than one owner adaptation is necessary.
8D.20 — Self-affecting service actions and reboot	Confirmed typed operations for connection-losing HTTPS/Wi-Fi/service actions and reboot, reusing lifecycle behavior established in 8D.7/8D.13.	Acknowledgement/pending-action handling is bounded, no duplicate action on UI retry, connection loss is explained, recovery and reconnect work. No new unbounded queue or certificate/key rotation yet.
8D.21 — Security/danger-zone settings	Carefully separated HTTPS/SSH identity rotation/reset and any explicitly retained recovery-secret operation through existing security APIs. Enumerate allowed operations first; split HTTPS and SSH work into separate requests if both need owner changes.	Confirmation, secret-safe one-time responses, no routine private-key export, expected trust/fingerprint changes, bounded self-disconnect, no accidental configuration wipe. Bootstrap/unavailable-database recovery remain UART0-only. NVS encryption, secure boot, OTA, and new factory-reset semantics stay out of scope.

Dependencies: 8D.8 establishes only the minimal typed-request/UI pattern. 8D.9 follows 8D.8; 8D.11 follows 8D.10; 8D.13 follows 8D.12; 8D.17 follows 8D.9/8D.13; 8D.18 follows 8D.16; 8D.20 follows 8D.7/8D.13/8D.19; 8D.21 follows 8D.20. Independent domains can be reordered by the user, but should not be implemented concurrently against shared web_server/web_ui files.

8D.22 — Final integration acceptance and documentation

No new feature work. Run the complete Phase 8D acceptance matrix, including the accumulated per-chunk regression checks and a bounded concurrent-transport soak at supported workloads. Record the exact revision/configuration, browser(s), client mix, baud rate, test duration/cycle counts, linked RAM/flash, runtime free/minimum/largest internal heap and PSRAM, relevant stack margins, and queue/drop observations. Compare against 8D.0 and milestone measurements. Known high-load serial drops are not permission for new unbounded blocking, hidden lease loss, login failure, or resource exhaustion.

Verify optional web-session/admin/settings initialization failures preserve UART0 and native USB; admin-only failure must not disable otherwise working serial web access. Check logout/expiry/revocation across every added route, and verify no retained Basic path or direct typed endpoint bypasses policy. Review all routine snapshots/logs/DOM status for secret exposure and confirm changes did not regenerate unrelated vendor assets.

Update the roadmap and user/command documentation to distinguish completed features, explicit restrictions, build results, and tests actually passed. Update durable agent architecture/code-map notes only for implemented ownership/contracts. If a check fails, open a bounded repair chunk and rerun affected checks; do not append features or declare the whole phase done with unrecorded failures.

Progress and next-request template

Initial progress: 8D.0–8D.22 all planned. Record incremental results in docs/agent/current-state.md, retaining the baseline and cumulative resource measurements as work proceeds. This planning-only change has not measured the target or run firmware acceptance tests.

Suggested next request:

Work on Phase 8D.0 only, following docs/phase8d_plan.md. Establish the rolled-back baseline, the minimal M1 authentication/browser contract, and the resource budget. Do not implement authentication, import the experimental branch, or proceed to 8D.1. Report missing hardware evidence as pending and stop with the exact validation checklist.

For later chunks:

Work on Phase 8D.N only. First verify its prerequisites and last target-validation checkpoint. State allowed files, exclusions, resource budget, and focused checks. Split the chunk if it does not fit one session with validation time reserved. Build, report actual versus pending validation and memory deltas, update the handoff, and stop; do not continue to the next chunk.