ESP32_Serial_Swiss_Army_Knife/docs/agent/design-decisions.md

Durable design constraints and decisions

Only constraints supported by implementation or current project documentation belong here. When original rationale is unknown, the entry describes the observable constraint without inventing intent.

Display configuration has an owner reservation separate from button activity

Decision: local_status_ui owns a nonwrapping configuration generation and a zero-wait reservation shared by typed Display and CLI/legacy Apply. NVS runs outside timing critical sections; Save stabilizes selected bytes and Reset commits defaults before RAM publication. Load retains canonical fallback. Buttons/diagnostic holds update activity, not configuration generation.

Consequence: Compare before mutation, never overwrite intervening CLI edits, and do not make physical panel presence a configuration prerequisite. A successful config API is not proof of display IO. Owner and persistence contract.

Typed Network edits preserve manager ownership and current secret bytes

Decision: HTTPD uses zero-wait secret-free projections and an ID-only dispatcher. Wi-Fi compare/merge/whole-candidate validation and queue-before-publication preserve omitted secrets and reject stale changes; its manager alone owns radio/reannouncement. mDNS generation/persistence is independent. Save stabilizes selected RAM; Wi-Fi Load is stored-only, not fallback-secret generation.

Consequence: Keep SSIDs reversible bytes, omitted/Replace/disabled-STA Clear distinct, AP clear denied, and Next profile separate from editor selection. mDNS may report RAM applied but reannouncement not queued, without rollback. A one-second queued-secret timer is not hard cancellation; accepted is not online/DNS. Recovery and no automatic replay are correctness requirements. Network contract.

One broker mediates all production serial transports

Decision: USB CDC, WebSocket, and role-user SSH access UART1 through session_broker; transports do not independently own the serial service.

Rationale/evidence: The broker is initialized after the serial service and all transport implementations connect broker clients. It is the normal serial RX consumer and TX gate. Project documentation requires one writer and multiple observers.

Consequence for future changes: New serial transports must become broker clients. Do not bypass writer checks or consume serial_service RX directly. serial_service_start() is not idempotent, so admission code must reconcile check/start races as the existing transports do. Broker paths enter serial-service APIs while holding the broker mutex; preserve that lock order and do not call back into the broker while holding the serial state mutex. Preserve binary transparency and avoid in-band ownership control.

Relevant files: src/session_broker.{h,c}, src/serial_service.{h,c}, src/usb_cdc_transport.c, src/web_serial_transport.c, src/ssh_transport.c

Slow clients are isolated by bounded per-client storage

Decision: UART RX is drained and copied into independent bounded broker output streams; a full observer loses only its own copy.

Rationale/evidence: session_broker accounts per-client dropped bytes instead of blocking fan-out. The roadmap records slow-client isolation as a project-wide constraint.

Consequence for future changes: Do not replace fan-out with a blocking shared queue. Any added transport must tolerate partial/no-progress reads and expose drop/backpressure counters.

Throughput observation and controlled experiments: The initial diagnostic baseline used CPU160MHz; a CPU240MHz-only experiment reduced but did not eliminate browser queue overflow. Combining binary WebSocket header/payload into one bounded session-override send eliminated reported drops, and the user signed off230400-baud full-client-mix operation after returning to160MHz. Retain the combined send, not the frequency increase; evidence and limits are in current-state.md. Preserve scheduling/priorities and 4096/512-byte broker/web buffers while gathering per-client HWM/drop attribution and independent opt-in web binary-TX timing. Fixed-slot epoch/generation-fenced aggregates avoid stale attribution; no new runtime allocations. Clear preserves queued data and seeds broker HWM; disconnected rows disappear while global discard counts remain. Callback timestamps precede the transport lock; synchronous send return is not peer receipt. Completion-to-read intervals include broker/mutex/control work and possible idle, even when the first read is nonempty; never label them pure scheduling latency or proof of backlog at completion. Compare enabled/disabled target captures before drawing overhead conclusions. Contracts and reproduction: throughput diagnostics.

Relevant files: src/session_broker.c, src/session_broker.h, docs/roadmap.md

Physical UART ownership and logical writer ownership remain separate

Decision: rs232_port_owner controls whether diagnostics or the serial service may manipulate UART/MAX3243 hardware; the broker separately controls which connected client may write.

Rationale/evidence: The code has explicit NONE, PHASE0, SERVICE, and FAULT hardware states plus broker client/writer IDs.

Consequence for future changes: A writer lease never authorizes direct UART/GPIO access. Active hardware tests must claim PHASE0; the production service must claim SERVICE. Boot-time static-safe GPIO setup and service-owned static-mode restoration are explicit exceptions to this cooperative gate. Ambiguous cleanup must keep the transceiver safe and require reboot rather than clearing fault casually.

Relevant files: src/rs232_port_owner.{h,c}, src/rs232_hw_test.c, src/serial_service.c, src/session_broker.c

Resource IDs are generation-safe

Decision: Broker/SSH/WebSocket slots, originating web sessions, queued admin operations and account principals carry distinct generation/identity fences. Browser cookie-session identity is not interchangeable with account identity. Invalidate session records before socket cleanup and retain authoritative currentness checks even when notifications fail.

Consequence: Never turn selected transport IDs into arbitrary broker IDs/fds, rebase a stale confirmation or wrap a published token. Exhausted broker/SSH slots retire; operation/reservation IDs do not reuse; saturated service/lease generations fence ABA without disabling canonical recovery. Reboot invalidates old logins. Validate/reserve at the owner immediately before effects, not snapshot-check/unlock/unconditional mutation. Broker and service contracts.

Confirmed writer transfer compares a lease version inside the broker lock

Decision: Compare the connected nonreused target and separate lease generation inside the force-writer lock. A client ID alone cannot fence release/reacquire ABA. Lease generation saturates, survives counter clear and advances before advisory event delivery, potentially twice for force transfer.

Consequence: Saturation rejects typed assignment but preserves ordinary request/release/disconnect and recovery force. Accepted serial TX is not recalled. Live contextual refresh must retain the selected versions and sticky stale/absence state until explicit reselection. Wrap and UI contract.

UART0 is the physical recovery authority

Decision: UART0 remains independent of UART1 and networking. The first administrator is created with normal user add on UART0; explicit unavailable-user-database recovery to empty is UART0-only and refuses healthy storage. No bootstrap command/API remains.

Rationale/evidence: main.c configures UART0 separately; command policy and user handlers deny these operations remotely. README/roadmap identify UART0 as the trusted recovery console.

Consequence for future changes: Network failures or credential corruption must not remove UART0 recovery. Do not expose unauthenticated first-admin provisioning or recovery through web or admin SSH without an explicit security redesign.

Relevant files: src/main.c, src/admin_ssh_console.c, src/user_console.c, docs/roadmap.md

Admin SSH and user SSH are different routes

Decision: A role-user SSH session becomes a broker serial client. A role-admin session enters the administration console and never obtains a broker client/writer lease.

Rationale/evidence: Role routing is explicit after SSH authentication. The administrative shell is intended for command execution, not multiplexed serial data.

Consequence for future changes: Do not silently give administrators both streams or infer that higher privilege means UART1 ownership. A route-switch feature would require explicit protocol, lifecycle, and authorization design.

Relevant files: src/ssh_transport.c, src/admin_ssh_console.{h,c}, src/session_broker.c

One dispatcher executes the canonical command registry

Decision: UART0 and admin SSH submit complete lines to one fixed queue; one task is the sole caller of esp_console_run().

Rationale/evidence: The implementation treats ESP-IDF console execution as non-reentrant and removes the need for separate remote command implementations.

Consequence for future changes: Register one canonical handler rather than creating a second SSH dispatcher. Long commands/prompts block all administration, so keep handlers bounded or explicitly asynchronous. Preserve output routing and remote principal checks.

Relevant files: src/admin_ssh_console.c, src/main.c, src/console_input.c, all src/*_console.c

Selected self-affecting admin SSH actions use bounded deferred control

Decision: SSH self-close/reboot/lifecycle/host-key actions use existing bounded application-drain control; browser shell stop/reboot and typed certificate handoff use owner adapters, not command replay. Drain is not peer receipt. Crypto/NVS browser rotation runs on the existing dispatcher, not the small control stack; executing slots remain reserved across self-detach.

HTTPS typed ACK: Successful synchronous send return precedes one nonreused-ID HTTPD callback to the existing dispatcher. No captured request/fd/reusable operation pointer or lifecycle wait on HTTPD. Lost accepted work retains one reservation until callback or successful destruction; never release speculatively on timeout. Original-login/current-admin/dequeue deadline checks precede admission. Reserved restart may deliberately invalidate that login; subsequent revocation is not cancellation. Reboot invokes canonical esp_restart() outside locks without self-console cleanup.

Identity replacement: Compare/reserve service then task-bound nonreused identity token, shared by canonical/direct callers. Crypto/NVS run outside security locks. HTTPS commits before stop/start; SSH stops before commit/restart. Failed SSH stop skips mutation/start; persistence failure can follow client disconnection. Committed material is never rolled back on restart failure. SSH owner retains context until all slots retire; start rejects orphan handles. Public fingerprint projections do not authorize mutation, and stored/served HTTPS identity can differ after failed stop.

Consequence: Explicitly communicate partial effects, trusted UART0 fingerprint verification and fresh HTTPS login after restart. Preserve CLI recovery/reset but do not add browser Reset/export as if it were ordinary rotation. Native USB is independent UART1, not administration or uninterrupted reboot. Lifecycle/identity contract.

Authentication uses copied principals and fail-safe currentness checks

Decision: Network sessions retain secret-free copied principals. Database/account ID/auth generation and originating web-session identity must be current at admission, before sensitive input and during reconciliation. Best-effort target notifications supplement, never replace, these checks and cannot roll back committed mutations.

Consequence: Shared remote-console slots require transport-qualified tokens and immutable owner adapters. Validate owner currentness outside console locks, then recheck identity. Owner-side HTTPD/SSH IO and generation-safe cleanup remain mandatory; session liveness checks do not cancel executing handlers. Browser-shell permissions are parsed and narrower than typed Settings. Authentication, console policy.

Typed serial mutations share the administration dispatcher

Decision: Typed domains queue IDs to the existing serialized dispatcher, never CLI strings or secrets. One original-login slot per domain and a nonreused ID fence stale work; session/deadline checks precede canonical admission. Results are replaceable observations, not durable history/idempotency.

Consequence: No automatic mutation replay, including after navigation, timeout or logout. Ordinary deadlines limit dequeue admission, not execution. Accounts/Network queued-secret timers wipe only non-executing inputs; locals wipe after admitted work returns. Explicit RAM/NVS/reset semantics and partial-effect uncertainty must match each canonical owner. API bounds and lifetime.

Typed account selection is checked inside the database mutation lock

Decision: Target username/account ID/auth generation compare occurs inside the canonical mutation lock for role/delete/password/key changes. HTTPD uses compact zero-wait secret-free projections, not the blocking CLI snapshot. Successful changes notify only the target's web/SSH sessions, including self.

Consequence: Separate generated-value delivery from mutation and retain no retrieval history; context-bound saved acknowledgement is UX, not receipt proof. Self-revocation can deny results, so 401/disconnect cannot mean success or cancellation. Key slots are stable and sparse, fingerprint-only on output; import shares canonical validation. Final-admin invariants and UART0 provisioning/recovery remain. Accounts contract.

Browser authentication has a narrow version-pinned HTTPD boundary

Decision: web_httpd_adapter alone accesses private IDF 5.5.0 parsed-header/session state. Reject duplicate/ambiguous headers; defer 101 until cookie/Origin/ticket/transport admission; wipe consumed scratch while preserving unread bytes. Stage optional Settings descriptor/name allocations before publication. Compile header/ticket debug logging out.

Consequence: Re-audit SDK assumptions on upgrade; never patch around Origin null by weakening same-origin policy. Browser authentication POST uses CORS mode with fixed same-origin URLs/credentials because no-referrer non-CORS POST can serialize Origin as null. Digest-only cookie/challenge sessions replace Basic without fallback or live-record eviction. CSP loader hashes and authored scripts change atomically. Navigation preserves terminals/lease, while session-identity changes require a clean document before showing retained buffers. Authentication and terminal contracts.

Security material and configuration use bounded, versioned NVS records

Decision: Application settings, users, and identities use separate fixed/versioned NVS blobs. Serial, Wi-Fi, mDNS-hostname, and local-UI working edits are RAM-only until explicitly saved. User mutations and HTTPS/SSH identity changes commit directly as part of the operation. Invalid ordinary configuration generally selects RAM defaults without erasing storage; malformed security material fails closed and needs explicit reset.

Rationale/evidence: Serial, Wi-Fi, local UI, web security, users, and SSH security each validate schema/size and own their namespace. User/security mutations build and validate candidate state before committing it; security modules avoid silently replacing an established identity. The live user database remains internal while its 5,360-byte candidate is a persistent PSRAM-preferred allocation with internal fallback and is wiped after every transaction.

Consequence for future changes: Add schema versions and transactional candidate validation. Do not overwrite unknown records automatically; provide explicit migration/reset behavior. Preserve the distinct persistence contracts: explicit save/load/default/reset for working configuration and per-blob commit-before-live-install for user and identity mutation. Keep candidate ownership mutex-local and wipe/free it on initialization or recovery failure. Recheck external-buffer staging in the flash/NVS implementation when upgrading from the pinned ESP-IDF 5.5 baseline. Legacy credential synchronization and reconciliation are removed. Missing user storage commits empty; valid user v1 bytes remain compatible, with private v1_admin_marker derived from admin count, not a public bootstrap contract. HTTPS v1 (1,392 bytes) migrates through a private validated reader to TLS-only v2 (1,340 bytes), preserving exact DER/fingerprint/generation and committing before publication. Failures fail closed without fallback regeneration or overwriting rejected records. See legacy compatibility.

Relevant files: src/serial_config.c, src/wifi_config.c, src/mdns_config.c, src/mdns_service.c, src/local_ui_config.c, src/web_security.c, src/user_database.c, src/ssh_security.c

NVS is persistence, not a physical security boundary

Decision: The current firmware stores Wi-Fi credentials and TLS/SSH private keys in unencrypted application NVS. The reserved NVS-key partition does not enable encryption.

Rationale/evidence: partitions.csv, README security notes, and current code show no NVS-encryption setup. Original rationale for deferring encryption is outside the implementation; the observable limitation is explicit.

Consequence for future changes: Do not claim resistance to flash extraction. Logical NVS replacement can leave old plaintext credentials in flash and is not secure erasure; no factory erase is required by this cleanup. Older v1-only firmware cannot read v2 HTTPS material. Avoid increasing stored secret exposure. Enabling encryption requires migration/recovery planning, not just changing the partition table.

Relevant files: partitions.csv, README.md, src/web_security.c, src/ssh_security.c, src/wifi_config.c

Wi-Fi callbacks enqueue; the manager owns policy

Decision: ESP event callbacks copy bounded event data into the Wi-Fi manager queue. A permanent manager task performs driver operations, profile/AP policy, deadlines, reconciliation, and station mDNS announcement transitions. mDNS initializes at most once, remains allocated across transient disconnects while its component handlers withdraw/re-enable the STA interface, and treats failure as nonfatal.

Rationale/evidence: Callback paths avoid blocking, NVS, and policy work. Manager deadlines consult authoritative driver/netif state so dropped events are recoverable.

Consequence for future changes: Keep callbacks short and nonblocking. Add state transitions to the manager rather than directly invoking Wi-Fi policy from consoles, UI, or callbacks. Preserve queue-drop observability.

Relevant files: src/wifi_manager.{h,c}, src/wifi_config.{h,c}, src/mdns_service.{h,c}, src/mdns_config.{h,c}

Optional local UI cannot become a core dependency

Decision: The OLED/display may fail without stopping serial, UART0, USB, or networking. The UI consumes copied snapshots and calls public APIs; it never parses CLI output or joins the broker.

Rationale/evidence: main.c logs display failures and continues. local_status_ui collects snapshots before display frames and exposes limited confirmed controls.

Consequence for future changes: Keep OLED/I2C work bounded and outside service locks. Do not put credentials or core ownership into UI state. A missing display must remain nonfatal.

Relevant files: src/main.c, src/local_display.{h,c}, src/local_status_ui.c, src/local_ui_config.c

Hardware and library access has designated owners

Decision: The serial task owns UART1 while active, local_display owns I2C/framebuffer access, the SSH owner task owns runtime wolfSSH contexts/calls after caller-side library initialization, and the console dispatcher alone runs registered commands.

Rationale/evidence: These constraints are enforced by module structure, mutex/task assertions, and transport indirection. Original rationale varies; the observable effect is serialized library/hardware access.

Consequence for future changes: Cross-task requests should use existing queues/public APIs. Do not make post-initialization wolfSSH calls, mutate display frames, or run console handlers from arbitrary tasks.

Relevant files: src/serial_service.c, src/local_display.c, src/ssh_transport.c, src/admin_ssh_console.c

Software cryptography settings are a validated concurrency workaround

Decision: wolfSSL ESP32 AES/SHA acceleration is disabled, and HTTPS uses software AES for PSRAM-backed TLS records. Internal task stacks are retained where cache-disable safety matters.

Rationale/evidence: Root CMakeLists.txt disables wolfSSL hardware crypto. The roadmap reports a reproduced watchdog stall involving mbedTLS external-RAM hardware-AES DMA, uncoordinated mbedTLS/wolfSSL hardware locks, and a successful software-crypto concurrency retest; no standalone execution record is checked in.

Consequence for future changes: Do not remove these definitions as a performance cleanup. Any re-enablement needs target-hardware concurrency testing with simultaneous USB, WebSocket, SSH, and serial traffic plus watchdog/stack telemetry.

Relevant files: CMakeLists.txt, src/CMakeLists.txt, docs/roadmap.md, relevant sdkconfig.defaults crypto settings

Embedded web assets are checked-in generated artifacts

Decision: Vendored xterm assets are compressed and embedded ahead of the normal firmware build; src/web_assets_data.c is compiled directly.

Rationale/evidence: src/CMakeLists.txt lists generated data as a source, and web_assets/SOURCES.md documents pinned versions, hashes, and deterministic gzip inputs.

Consequence for future changes: Edit authored web UI separately. Changes to its inline bootstrap loader must update the hard-coded CSP hash atomically and preserve the response security policy. When dependency assets change, follow the documented provenance/generation process and review generated diffs; do not hand-edit arrays or regenerate assets during unrelated work.

Relevant files: web_assets/SOURCES.md, web_assets/generate_embedded_assets.py, src/web_assets_data.{h,c}, src/web_ui.c