ESP32_Serial_Swiss_Army_Knife/docs/roadmap.md

# Implementation roadmap

This document tracks the implementation and hardware-validation plan for the ESP32 Serial Swiss Army Knife. It describes intended sequencing rather than a release schedule. A phase is complete only when its implementation, documentation, build validation, and relevant hardware tests have passed.

## Status legend

- **Complete** — implemented and validated on the target hardware.
- **In progress** — implementation or validation is actively underway, but the overall phase is not complete.
- **Implemented; validation pending** — code is present and builds, but the current implementation still needs the listed hardware checks.
- **Planned** — accepted project direction, not yet implemented.
- **Under evaluation** — useful candidate whose feasibility, security, or resource cost must be measured before it becomes a commitment.

## Project-wide design constraints

These constraints apply across all phases:

- UART0 remains the trusted physical administration and recovery console.
- UART1 and the MAX3243 provide the single physical RS-232 data port.
- All serial transports use the central session broker: one writer, multiple observers.
- A slow or failed client must not block UART reception or unrelated clients.
- Serial data paths remain binary-transparent; transports must not reserve in-band control sequences.
- Network serial access must be authenticated and encrypted. No unauthenticated plaintext TCP serial service is planned.
- Secrets must not be exposed by routine status output, command completion, or logs.
- Resource use must remain bounded and observable on the ESP32-S3 N16R8 target.
- New work must preserve USB CDC and UART0 recovery when Wi-Fi or a network service fails.
- Project code remains `GPL-3.0-only`; third-party license notices and compatibility must be preserved.

## Phase overview

| Phase | Scope | Status |
|---|---|---|
| 0 | RS-232 hardware characterization and diagnostics | **Complete** |
| 1 | Persistent UART1 serial-service foundation | **Complete** |
| 2 | Transport-neutral session broker | **Complete** |
| 3 | Native USB CDC-ACM serial transport | **Complete** |
| 4 | Wi-Fi station/AP foundation and network diagnostics | **Complete** |
| 5A | Authenticated HTTPS administration foundation | **Complete** |
| 5B | Offline xterm.js WebSocket serial terminal | **Complete** |
| 6 | Authenticated SSH serial transport | **Complete** |
| 7 | Local display and button interface | **Complete** |
| 8 | Role-based users and administrative access | **In progress (8A–8C complete; integrated web administration 8D planned)** |
| 9 | Security and production hardening | **Planned** |
| 10 | Authenticated, rollback-capable OTA | **Planned** |
| 11 | BLE serial transport and provisioning evaluation | **Planned** |
| 12 | Advanced network integration | **Under evaluation** |
| 13 | Optional filesystem-backed features | **Under evaluation** |

## Completed phases

### Phase 0 — RS-232 hardware characterization

Implemented and hardware-validated:

- Central board pin profile for the ESP32-S3 N16R8 and MAX3243 breakout.
- MAX3243 enable, static driver, receiver, valid-voltage, and modem-signal checks.
- UART loopback across supported framing and baud-rate combinations.
- Hardware CTS transmit gating and RTS receive-backpressure tests.
- Exclusive RS-232 port ownership and fault-safe cleanup.
- RGB status indication for diagnostic state.
- Phase 0 hardware diagnostics, including signal status, are isolated under the `debug` CLI submenu.

Detailed procedures are in [Electrical tests](electrical_tests.md) and [Hardware wiring](wiring.md).

### Phase 1 — Serial-service foundation

Implemented and hardware-validated:

- Persistent, versioned UART configuration with safe defaults.
- UART1 lifecycle management and exclusive ownership.
- Nonblocking RX and TX software streams.
- Baud rate, data bits, parity, stop bits, RTS/CTS, DTR policy, and RTS threshold controls.
- Runtime counters and safe restart after working-configuration changes.
- Explicit save/load/default/reset behavior; edits are not silently persisted.

### Phase 2 — Session broker

Implemented and hardware-validated:

- Up to eight generation-safe broker clients.
- Exactly one writer lease with multiple observers.
- Bounded per-client output and event queues.
- Slow-observer isolation and drop accounting.
- Writer grant, release, denial, revocation, and forced administrative reassignment.
- DTR `on-connect` integration.
- Partial-transfer and retryable no-progress semantics for transports.
- Broker payload storage placed preferentially in PSRAM while FreeRTOS control structures remain in internal RAM.

### Phase 3 — Native USB CDC-ACM

Implemented and hardware-validated:

- Native ESP32-S3 USB CDC-ACM serial transport.
- Automatic broker connection when the host opens the port with DTR asserted.
- Opportunistic writer acquisition with observer fallback.
- Binary-transparent bidirectional serial data.
- Host line-coding visibility for diagnostics; UART1 remains under explicit serial configuration control.
- Disconnect cleanup, bounded buffering, counters, and UART0 lifecycle controls.

### Phase 4 — Wi-Fi foundation

Implemented and hardware-validated:

- Four persistent station profiles with priority ordering.
- WPA2/WPA3 mixed mode and WPA3-SAE-required mode.
- Explicit no-echo secret entry through UART0.
- AP policies `off`, `fallback`, and `always`.
- Random persistent fallback-AP password and MAC-derived default SSID.
- Bounded profile attempts, DHCP deadlines, failover, and retry backoff.
- Root and `wifi` aliases for `ping`, `nslookup`, and `traceroute`.
- Wi-Fi/lwIP payload allocation configured to prefer PSRAM with capacities pinned in `sdkconfig.defaults`.

Post-validation enhancements implemented; regression validation pending:

- Edits to disabled station profiles are staged without restarting the active radio; enable-state changes and edits affecting enabled radio policy apply asynchronously.
- Configurable STA-only `sak-<suffix>.local` mDNS hostname with an independent versioned NVS record, live hostname updates, automatic withdrawal/restoration across STA IPv4 transitions, and nonfatal failure isolation. The responder initializes at most once and keeps its task stack internal while general metadata prefers PSRAM.

A WPA3-required profile correctly rejects an AP that does not advertise the required Protected Management Frame capability; mixed mode is the compatible choice for such an AP.

### Phase 5A — Authenticated HTTPS foundation

Implemented and hardware-validated:

- HTTPS-only service on TCP port 443 using ESP-IDF mbedTLS.
- Persistent random shared `admin` credential, later retained only as migration/recovery material by Phase 8.
- Persistent device-specific ECDSA P-256 certificate and private key.
- Physical-console credential retrieval, rotation, certificate inspection, rotation, and recovery.
- Explicit failure behavior that preserves UART0, USB, serial, and Wi-Fi recovery paths.
- Credential rotation revocation hooks shared with WebSocket and SSH sessions.

### Phase 5B — Offline browser terminal

Implemented and hardware-validated:

- Vendored xterm.js and FitAddon assets; no Internet or CDN dependency.
- CSP-compatible page structure with scripts served as separate resources.
- Authenticated status and one-time WebSocket-ticket APIs.
- Binary WebSocket serial frames and broker writer/observer integration.
- Two bounded browser-terminal sessions.
- Writer request/release controls and role indication.
- Bounded TLS/socket behavior and slow-observer isolation.
- Viewport-constrained terminal fitting without recursive page growth.
- Validation with multiple clients, ANSI colors, advanced escape sequences, and full-screen terminal software.

Post-validation enhancement implemented; browser regression validation pending: the combined Connect/Disconnect control explicitly closes the terminal WebSocket and pauses automatic reconnect until Connect is selected.

### Phase 6 — Authenticated SSH serial transport

Implemented and target-hardware validated:

- wolfSSH on TCP port 22 with a separate persistent ECDSA P-256 host key.
- Two bounded interactive shell/PTY sessions; no `exec`, file transfer, forwarding, or subsystems.
- Authenticated, binary-transparent broker forwarding with opportunistic writer acquisition and observer fallback.
- Bounded handshake attempts/deadlines, receive work, buffers, and a single wolfSSH owner task pinned to CPU 1.
- UART0 lifecycle, session, counter, and host-key administration.
- PSRAM-preferred allocations with internal fallback and internal task stacks.
- Bounded broker/WebSocket work, software mbedTLS AES, and software wolfCrypt AES/SHA to avoid the validated shared-hardware-crypto/PSRAM watchdog failure while retaining ESP-IDF SHA/MPI acceleration.
- Concurrent USB, WebSocket, and SSH operation validated with USB and SSH alternately holding the writer lease.

### Phase 7 — Local display and buttons

Implemented and target-hardware validated:

- Optional 128×64 SSD1315-compatible I²C OLED on GPIO11/12 and active-low buttons on GPIO10/13/14; assignments remain centralized in `board_pins.h`.
- Display service with a static framebuffer, bounded dirty-page commits, and separate yellow status rows 0–15 plus blue content rows 16–63.
- Low-priority status/control task built from copied public snapshots; no service lock is held across I²C and the UI never becomes a broker client.
- Overview, RS-232, broker, and network/service pages with no credential or key material.
- Confirmation-protected local controls for service lifecycle, Wi-Fi reconnect/profile rotation, writer release, display off, and reboot; the UI cannot assign a writer or edit secrets/configuration.
- Persistent dim/off settings, bounded boot animation, consumed wake presses, stuck-button quarantine, and one-shot confirmations.
- Missing, disconnected, or unresponsive display hardware remains nonfatal and can recover through one bounded reprobe.
- Concurrent serial, UART0, USB, WebSocket, SSH, and injected display/button fault behavior validated. See [Electrical tests](electrical_tests.md) and [Hardware wiring](wiring.md).

## Current and planned phases

Phase 8A through 8C are complete and target-hardware validated; integrated web administration in 8D remains planned. Later work remains planned or under evaluation. Optional features must not weaken the completed serial and recovery paths.

### Phase 8 — Role-based users and administrative access

Replace the single shared network credential with a bounded, persistent user system. Both roles can use the authenticated web serial/status interface. Over SSH, `user` routes to the broker-backed serial stream while `admin` routes exclusively to the administration shell and receives no broker client or writer lease. UART0 remains the physical recovery and bootstrap authority.

Implementation sequence:

1. **Phase 8A — User database and UART0 administration — Complete**
   - Versioned NVS database for up to eight `user`/`admin` accounts, random account IDs and authentication generations, final-admin protection, and three Ed25519/P-256 keys per account.
   - Salted PBKDF2-HMAC-SHA256 password verifiers, bounded no-echo entry, one-time generated passwords, and secret-free account/key status.
   - Transactional migration of the legacy credential to an unprivileged account, explicit UART0 bootstrap, and UART0-only recovery that replaces only the unavailable user blob.
   - Migration, bootstrap, CRUD, persistence, and command basics target-hardware validated; the full fault-injection matrix remains regression coverage. NVS remains unencrypted and offline guessing remains possible.
2. **Phase 8B — Role-aware HTTPS and SSH authentication — Complete**
   - HTTPS Basic plus SSH password/public-key authentication use the common database and copied secret-free principals; unavailable user storage fails closed.
   - One-time principal-bound WebSocket tickets and ongoing principal-currentness checks prevent stale admission or input.
   - Account mutations request targeted WebSocket/SSH revocation; authentication generations provide fail-safe invalidation while unrelated accounts remain connected.
   - Legacy credentials remain migration/recovery-only after bootstrap. Password/key login, ticket behavior, targeted revocation, recovery, and concurrent transport operation are target-hardware validated.
3. **Phase 8C — SSH administrative shell — Complete**
   - Authenticated `admin` SSH shell sessions route to a bounded administration worker and never create a broker client or acquire a serial writer lease. Normal `user` sessions retain the existing broker-backed serial stream.
   - UART0 and admin SSH now submit complete lines to one fixed-length request queue. A single dispatcher task is the sole caller of ESP-IDF's non-reentrant `esp_console_run()` and therefore executes the same registered command handlers for both entry routes. The former separately implemented reduced SSH command dispatcher has been removed.
   - The worker uses fixed per-session command/input and output buffers. Queue records contain copied secret-free principals and generation-tagged session tokens; late work is discarded after disconnect, slot reuse, role change, password/key mutation, or deletion. Task-local standard streams route canonical handler output into the applicable bounded SSH ring, and only the SSH owner task calls wolfSSH APIs.
   - Transport-neutral bounded prompts now support interactive user passwords/keys and Wi-Fi secrets over admin SSH without exposing hidden input or allowing another command while a prompt is active. Ping callbacks enqueue typed bounded events and the dispatcher alone formats their output. Four-entry per-session history and whole-line Tab completion are RAM-only and wiped on disconnect.
   - Authenticated administrators receive the operational registry, including recovery-secret display, HTTPS material rotation/reset, reboot, ping, and SSH lifecycle/session/host-key mutation. Self-terminating reboot and SSH actions are deferred until acknowledgement output drains, block further shell input, and execute through existing synchronous owner APIs from a separate bounded control task. Initial `user bootstrap` and explicit `user recover --force` remain physical-UART0 operations; admin SSH also rejects generating a replacement password for its own account.
   - `ssh sessions` and `ssh counters` identify broker versus admin-console routes, worker command state, queued admin output, admission failures, and input backpressure. `exit` and Ctrl+D on an empty command line request bounded deferred self-disconnect after best-effort application-buffer draining. Admin sessions are checked for a current `admin` principal before command execution and during the active-session reconciliation.
   - Keep SFTP, SCP, `exec`, forwarding, subsystems, and unauthenticated shells disabled.
   - Target-hardware validation passed for route separation, history/Tab editing, interactive visible/hidden prompts, output/backpressure, generated and entered user/password/key management including the longest ECDSA P-256 import, ping event routing, deferred reboot/SSH lifecycle drain behavior, bootstrap/recovery rejection, targeted self/other-user revocation during queued work, UART0/SSH administration serialization, and concurrent USB/WebSocket/user-SSH/admin-SSH operation. Stress at 460800 baud with SSH and WebSocket clients caused substantial expected packet drops and slower display controls, but did not exhaust memory or require lowering the supported baud-rate range.
4. **Phase 8D — Integrated web administration — Planned**
   - Begin with integrated authentication: replace browser-facing HTTP Basic authentication with a same-origin HTTPS login page, explicit logout, and bounded opaque server-side sessions. Store only a digest of each random session token with a copied secret-free principal, expiry, CSRF state, and authentication-generation binding. Send the raw token only in a host-only `__Host-` cookie with `Secure`, `HttpOnly`, `SameSite=Strict`, `Path=/`, no `Domain`, and an explicit lifetime; never retain passwords, Basic headers, raw tokens, verifiers, or SSH-key blobs in snapshots or logs.
   - Make logout invalidate the current server-side session, expire its cookie, close that session's serial and administrative WebSockets, and redirect to login. Password/role/key mutation, deletion, recreation, and explicit revocation invalidate the affected account's web sessions and tickets without disturbing unrelated accounts. Require the CSRF token plus strict same-origin checks for every state-changing request, including logout, and rate-limit login attempts with bounded secret-free accounting.
   - Add an admin-only **Serial terminal**/**Admin shell** selector. The administrative route uses a short-lived, single-use, admin-principal-bound ticket and a bounded WebSocket frontend for the same serialized command registry used by UART0 and admin SSH; it is not a generic HTTP command-execution endpoint. Normal users retain the existing serial interface and cannot mint, upgrade, or invoke administrative routes. Remote policy still rejects physical-only `user bootstrap` and `user recover --force`.
   - Switching the visible terminal between Serial and Admin changes only the displayed terminal route. It must not disconnect the browser's serial broker client, release its writer lease, or silently stop serial observation. Keep the writer/observer badge plus Request control/Release control visible in both modes so an administrator knows the retained state and cannot unintentionally lose the lease to another client. Explicit Disconnect, logout, revocation, session expiry, or an explicit release/transfer operation still performs normal broker cleanup.
   - Add an admin-only Settings area backed by typed, bounded subsystem APIs rather than generated CLI strings. Cover user/password/role/SSH-key management, serial configuration and persistence, Wi-Fi profiles/AP policy/secrets, service and session controls, display settings, network diagnostics, and carefully separated security/danger-zone operations. The admin shell provides full remote operational parity and the settings pages provide guided high-frequency workflows; unusual electrical/debug operations may remain shell-only.
   - Add contextual admin quick settings to the existing status cards. Hover, keyboard focus, or click on **Serial** opens a popover with current framing/lifecycle state, safe common edits, apply/save semantics, and a link to full Serial settings. The **Wi-Fi** card similarly exposes connection/profile controls and a link to full Network settings without revealing saved secrets by default. Touch and keyboard users must receive the same functionality as pointer hover, with Escape/outside-click dismissal and no action triggered merely by opening a popover.
   - Hover, focus, or click on **Broker clients** opens a live, secret-free client list with IDs, transport, writer/observer state, and bounded queue/drop information. Activating **Active writer** opens an admin-only transfer dialog listing current eligible clients; transfer is an explicit confirmed generation-safe administrative assignment, never a side effect of opening or hovering. Stale/disconnected targets fail visibly without changing the current lease. Normal users may retain ordinary aggregate status but receive neither client-management details nor mutation controls.
   - Preserve strict CSP, no-referrer/frame-denial policy, no-store responses for login/session/admin material, secret-safe JSON encoding, one-time generated-password display, and bounded request/response bodies. Keep UART0 recovery, native USB UART1 access, and existing SSH behavior available if web sessions or administration cannot initialize.
   - Add secret-free counters/snapshots for session capacity, expiry, login failure/backoff, logout, invalidation, CSRF/origin rejection, admin-console admission/backpressure, typed-API failure, and writer-transfer races. Compatibility HTTP Basic, if retained temporarily during migration, must be explicit, separately constrained, and unable to bypass logout or revocation.

Completion requires login/logout and account switching without browser credential-cache dependence; stale-session, CSRF, origin, capacity, expiry, and revocation tests; hidden and server-rejected normal-user administration; shared admin-console serialization and backpressure; terminal switching that demonstrably preserves the browser broker client and writer lease; accessible Serial/Wi-Fi/client/writer popovers; generation-safe explicit writer transfer; typed settings and secret-handling tests; concurrent USB/WebSocket/user-SSH/admin-SSH/browser-admin operation; and continued UART0 recovery.

### Phase 9 — Security and production hardening

Harden stored credentials, boot integrity, update trust, failure diagnostics, and dependency maintenance. This phase must be staged carefully because some ESP32 eFuse operations are irreversible.

Planned work:

- Define development, test, and production security profiles.
- Enable and validate NVS encryption for Wi-Fi, HTTPS, and SSH material.
- Evaluate flash encryption and PSRAM encryption for the target hardware and threat model.
- Enable secure boot with a documented signing-key and recovery process.
- Define secret-safe core-dump handling; do not expose credentials or session material in diagnostics.
- Audit key and session-material zeroization across application, mbedTLS, wolfSSL, and wolfSSH allocations.
- Add authentication throttling, lockout/backoff, and security-event counters where appropriate.
- Review TLS/SSH algorithms, certificate policy, password policy, and browser security headers.
- Review current wolfSSL/wolfSSH and ESP-IDF security releases instead of treating integration versions as permanent pins.
- Recheck all third-party licenses and preserve source/license notices.
- Document provisioning, key rotation, factory reset, backup, recovery, and device decommissioning.

Security features must be validated first on expendable hardware before any irreversible production configuration is recommended.

### Phase 10 — Authenticated OTA and rollback

Use the existing dual 4 MiB application slots and `otadata` partition to add safe firmware updates.

Planned work:

- Accept only authenticated, integrity-checked firmware images compatible with the selected secure-boot policy.
- Support an update path that does not expose a plaintext management endpoint.
- Stream downloads with bounded RAM use and explicit progress/error reporting.
- Preserve serial, Wi-Fi, HTTPS, and SSH configuration across successful updates.
- Mark new firmware pending until a post-boot health check succeeds.
- Roll back automatically after failed boot or health confirmation.
- Define power-loss behavior for every update stage.
- Keep UART0 recovery and wired re-flashing documented and functional.
- Add version/compatibility checks and reject accidental downgrade unless an explicit policy allows it.

OTA is complete only after successful update, interrupted-update, invalid-image, rollback, and configuration-persistence tests on hardware.

### Phase 11 — BLE

Evaluate and, if resource limits permit, add BLE as another bounded transport or provisioning path.

Planned work:

- Decide whether BLE is primarily a serial transport, a provisioning interface, or both.
- Require authenticated pairing/bonding appropriate to the device threat model.
- Integrate serial access through the existing broker rather than bypassing writer ownership.
- Preserve binary transparency through explicit framing and MTU-aware partial transfers.
- Bound connections, queues, retries, and advertising behavior.
- Measure coexistence with 2.4 GHz Wi-Fi, HTTPS, WebSocket, SSH, USB CDC, and UART service.
- Store bond/provisioning data under the hardened secret-storage policy.
- Provide UART0 controls and a physical recovery path for clearing BLE state.

BLE remains subordinate to stable Wi-Fi and serial operation; it should be omitted if coexistence or memory costs cannot be bounded acceptably.

### Phase 12 — Advanced network integration

These features are candidates, not current commitments:

- Additional DNS-SD service advertisement and any certificate-name integration beyond the implemented configurable STA hostname.
- Enterprise Wi-Fi support, subject to credential-storage and certificate-validation design.
- IPv6 behavior and diagnostics beyond the current basic support.
- WireGuard feasibility evaluation, including RAM, CPU, licensing, key storage, routing, and recovery impact.

The device is not intended to become a general-purpose router. Captive-portal interception, unauthenticated DNS redirection, NAPT, and a plaintext serial listener remain out of scope unless the project requirements are explicitly revised.

### Phase 13 — Optional filesystem-backed features

The `storage` partition is reserved but not currently mounted. Possible uses must be justified individually:

- LittleFS-backed web assets to decouple large static files from the application image.
- Bounded diagnostic logs with explicit retention and secret-redaction rules.
- Import/export of non-secret configuration.
- Optional user files needed by later administration features.

Before enabling storage, define corruption recovery, wear limits, quotas, atomic update behavior, compatibility across OTA slots, and whether encryption is required. Serial operation and recovery must not depend on a mountable filesystem.

## Cross-phase release gates

Every phase should satisfy the following before being marked complete:

1. A clean release build for the ESP32-S3 N16R8 target.
2. No new compiler or project diagnostics attributable to the change.
3. Focused automated tests where practical and documented hardware tests where hardware behavior is involved.
4. UART0, native USB CDC, and previously completed network transports still operate.
5. Broker writer/observer semantics and binary transparency remain intact.
6. Failure, disconnect, stop/start, reboot, and stale-session cleanup paths are exercised.
7. Runtime memory, stack, socket, and queue bounds are measured under maximum supported concurrency.
8. Secrets remain absent from ordinary status output and logs.
9. [Command reference](command_reference.md), [Hardware wiring](wiring.md), electrical tests, and this roadmap are updated as applicable.
10. New dependencies receive security, maintenance, and GPL-3.0 compatibility review.

## Explicitly deferred work

The following are not implemented merely because flash partitions or library support exist:

- NVS, flash, or PSRAM encryption.
- Secure boot or production eFuse provisioning.
- OTA download, image confirmation, or rollback policy.
- Core-dump collection or secret-safe core-dump processing.
- Filesystem mounting.

- SFTP, SCP, SSH `exec`, forwarding, or subsystems.
- General routing, NAPT, captive-portal interception, or unauthenticated serial access.

Deferring these features is intentional: each changes the security model, recovery behavior, or bounded-resource assumptions and therefore requires its own design and validation gate.