ESP32_Serial_Swiss_Army_Knife/docs/phase8d9_implementation.md

Phase 8D.9 — Serial edits and persistence

State and scope (2026-09-07)

Implemented, reviewed, host-tested and firmware-build verified; target/browser validation and phase signoff remain pending. Resumed the previous agent's uncommitted implementation at the user's request. Continuation was explicitly authorized despite pending 8D.8 target validation; it does not sign off 8D.8 or 8D.9. M2 acceptance stands. No later domains, quick popover, generated assets, upload, erase, or commit actions.

UX refinement (2026-09-08)

User reported Apply succeeds but manual Check Result hides the editor without refreshing the displayed configuration. User approved automatic completion/refresh and removal of routine disruption popups. Implemented in src/web_ui.c, with no backend/protocol change:

• Actions submit once, show applying/pending and keep settings visible with conflicting controls disabled and an explicit stale-snapshot message.
• After acknowledged submission, result checks wait one second initially and between pending replies, capped at 10 GET attempts and a separate 15-second overall monotonic deadline including session revalidation. The deadline aborts an in-flight check; late callbacks are fenced. Browser timer scheduling is not a hard real-time guarantee.
• Known terminal results, including failures, automatically refresh working values while retaining outcome/uncertainty messages. Refresh failure leaves existing values visible and explicitly stale; Refresh remains available. The completion refresh has its own existing per-request bounds, outside the auto-check deadline.
• Errors/exhaustion/lost acknowledgements stop automatic checking and retain Check Result for manual recovery. Navigation/pagehide/logout/identity changes cancel checks without auto-resumption; they do not cancel backend work. No automatic POST retry.
• Apply/Save/Load/Defaults/Start/Stop no longer show confirmation dialogs. Reset alone confirms overwriting saved NVS configuration. Inline RAM/NVS/discard semantics remain.
• Review found repeated selection of the current Settings tab cancelled requests/polling; selecting the current view is now a no-op, with a regression covering submission, inter-check delay, in-flight checks and completion refresh.

Final parent validation: python3 tests/web_ui_session/run.py 35 behavior groups plus renderer/header/failure/exact CSP checks, pio run PASS 10.91 s, git --no-pager diff --check PASS. Build 95,708 B RAM / 1,668,837 B flash, 0 / +2,112 B versus the original 8D.9 build below (+128 / +14,308 B versus 8D.8). Static build accounting is not runtime/stack telemetry. Tests model DOM/fetch/timers/WebSockets; new flow not target-validated. No upload, erase, asset regeneration or commit. Original validation below remains historical evidence, not a claim that backend suites were rerun for this UI-only refinement.

Contract

• Existing admin-only Settings/Serial now has typed baud, data bits, parity, stop bits, flow control, DTR and RTS-threshold drafts, plus explicit Apply, Start, Stop, Save, Load, Defaults and Reset controls. Server authorization, not UI visibility, enforces admin access.
• POST /api/settings/serial-operation requires current cookie/principal, admin role, canonical Origin and CSRF. JSON is at most 256 bytes and four receive calls, with exact action-only or complete Apply schema; unknown/duplicate fields, escapes, nesting, invalid framing and out-of-range values reject. Failed unread bodies close rather than contaminate the next request.
• HTTPD admits only an operation ID to the existing four-entry administration queue with zero wait. web_serial_settings owns one static session-bound pending/result slot; the existing 12 KiB dispatcher executes typed serial APIs, never CLI text or HTTPD-owned socket operations. Queue/slot exhaustion returns 503 with Retry-After; no new task or generic job framework.
• Execution checks originating cookie/principal/admin currentness and the 30-second queue-admission deadline before side effects. This deadline is checked on dequeue: a blocked dispatcher retains the slot, and the deadline is neither an execution timeout nor a slot-release timer. Work already admitted to serial/NVS APIs may finish after logout/revocation. IDs do not wrap/reuse; stale queued IDs cannot execute replacement work.
• Bodyless GET /api/settings/serial-operation exposes only the current login's retained {id, action, state} (96-byte response buffer). States are idle, pending, ok, failed, cancelled, loaded_defaults and rollback_failed. A subsequent admitted operation, including another session's, can replace a completed result. This is neither durable history nor an idempotent retry API.
• Result checking is single-flight and automatically bounded after acknowledgement, with manual recovery as detailed above; mutations are never automatically retried. Lost acknowledgements and replaced results retain uncertainty warnings across further reads, Refresh and navigation. Cancellation/navigation is not a promise that admitted work stopped. Inspect current working settings and CLI storage before deciding to retry.
• Settings navigation preserves both terminal sockets, hidden output, broker client and writer lease. The operation itself may stop/reconfigure UART1 and discard serial-service RX/TX/task-pending bytes; already-fanned broker output and writer ownership remain unchanged. Open USB can restart a stopped service.

Working versus persisted state

Action	Canonical behavior
Apply	Validate and replace all working fields; running service stop/restart and best-effort rollback use existing service API. Overwrites intervening CLI edits. No NVS write.
Start / Stop	Existing service lifecycle APIs; no persistence or broker lease change.
Save	Persist device working configuration at execution, not the browser draft.
Load	Load and apply stored config; absent/incompatible storage applies defaults and reports loaded_defaults, without writing NVS.
Defaults	Apply defaults to RAM only.
Reset	Capture previous working config, apply defaults, persist defaults through canonical reset API; on persistence failure attempt runtime rollback, reporting rollback failure distinctly.

Refresh replaces the browser draft. Snapshots do not expose a saved-config copy or claim a dirty/persisted comparison. There is no compare-and-swap protection against intervening CLI edits.

Integration and continuation fixes

• src/web_serial_settings.{c,h}: parser, bounded operation slot, typed execution, result protocol.
• src/admin_ssh_console.{c,h}: typed ID in existing queue union and dispatcher; no console slot consumed.
• src/web_cookie_auth.{c,h}: bounded-JSON mutation authorization while preserving bodyless policies for existing endpoints.
• src/web_httpd_adapter.{c,h}: existing failure-safe optional registration generalized to exact GET/POST. src/web_server.c registers result GET first and mutation POST second; partial failure cannot leave a mutation-only endpoint. Optional failure preserves both transports. SDK-private boundary remains pinned to IDF 5.5.0; inherited non-Settings registration audit remains open.
• src/web_ui.c: explicit typed controls, Reset-only confirmation, bounded automatic completion/refresh and manual result recovery and session/navigation fencing. Review fixed uncertainty warnings disappearing after later result checks/Refresh/navigation.
• /api/status now uses the zero-wait serial snapshot rather than blocking config acquisition. Review fixed a separate live running read mixing moments with the snapshot. When unavailable, running is JSON null and config_available is false; consumers must not interpret that as an authoritative stopped state.
• Backend review found no further confirmed runtime defects; added parser/security/queue/currentness/race regressions and documented deadline/result limitations. Existing parent/user edits in docs/phase8d8_implementation.md were preserved.

Validation performed

Final parent sequential run, all passed:

python3 tests/web_cookie_auth/run.py --serial-settings
python3 tests/web_cookie_auth/run.py --settings
python3 tests/web_ui_session/run.py
python3 tests/admin_console_boundary/run.py
python3 tests/web_admin_transport/server_lifecycle.py
pio run
git --no-pager diff --check

• Serial: 10 groups plus shared auth/store regressions, including 288 valid framing/range combinations, 256-byte/four-read boundaries, strict rejection, no HTTPD serial/NVS execution, session isolation, stale IDs, simulated concurrent reads/submissions, canonical CLI/API ordering, failures/reset rollback, account revocation and cookie expiry before the queue deadline.
• Settings/status: 6 groups plus shared regressions. Status serial projection compiles production acquisition/format/arguments, verifies consistent snapshot across changed live state and null on unavailability; it is not the full multi-subsystem handler.
• UI: 27 behavior groups plus production C renderer/header/failure and exact loader CSP hash checks. DOM/fetch/WebSockets are modeled, not on-wire integration.
• Console boundary: FIFO/zero-wait typed admission, full queue preserves all four UART requests; existing console/certificate/SSH-adapter regressions.
• Server: 13 lifecycle groups, including optional GET/POST failure and recovery, six sockets/no LRU.

Additional passing runs during continuation: parent transport run.py --tickets (25 transport/12 ticket groups), session store run.py --serial; backend review agent cookie --admin, console accounts.py and lifecycle.py. These use deterministic RTOS/UART/NVS/transport doubles, not real hardware scheduling, flash failures or rollback. No sanitizer run or target pass claimed.

Resource accounting

Final pio run: 23.73 s; 95,708 B RAM / 1,666,725 B flash. Delta against recorded 8D.8: +128 B RAM / +12,196 B flash; against 8D.0: +1,176 / +66,752 B. Build totals are static/linker accounting, not runtime free heap or stack margins.

Two additional optional handlers (GET and POST at one path) raise the configured URI budget from 17 to 19. Six HTTPS sockets/no LRU, two serial sockets/one admin socket, existing queue depth, tasks and stack allocations are unchanged. Additional route descriptors/names/table pointers consume runtime heap beyond static RAM; exact allocator overhead and handler/dispatcher high-water marks have not been measured. The bounded request/result buffers are 256/96 bytes. No generated asset changes; loader hash remains unchanged and verified. Numeric memory reserves/stack margins remain unapproved.

Pending target checklist / next step

1. Flash only by explicit user action/authorization; compare browser Refresh with UART0 serial config/status and confirm ordinary users cannot access settings/mutations.
2. Compare every browser action with CLI semantics, valid framing boundaries and invalid requests. Verify Apply/Defaults versus Save/Load/Reset across reboot; confirm explicit Refresh draft replacement and concurrent CLI overwrite warning.
3. Exercise start/stop/reconfigure under USB + SSH + both browser serial observers + browser/SSH admin. Confirm expected serial-service discards, unchanged broker writer/client identity, isolated output and USB restart behavior; do not mistake expected discards for a regression.
4. Hold dispatcher in a prompt, submit once, observe bounded busy behavior and dequeue deadline cancellation. Test logout/account revocation while queued, expiry, navigation, lost acknowledgement, replacement by another tab/login, and manual uncertain-result recovery without blind resubmission.
5. Where safely injectable, verify NVS/load/apply/rollback failures and physical port fault behavior. Host doubles are not evidence of hardware rollback success.
6. Capture settled boot/full-mix/post-cleanup internal/DMA/PSRAM free/min/largest and dispatcher/HTTPD stack high-water data; test slow/fragmented body requests alongside serial traffic. Existing intermittent supported-mix admission issue remains accepted nonblocking and unresolved.
7. Obtain explicit 8D.8/8D.9 target acceptance as appropriate. Stop before separately requested 8D.10. M2 is not reopened, deferred browser owner/credential restrictions remain, and no memory reserve approval is inferred.