Add SSH host identity rotation controls

docs/phase8d21_implementation.md

@@ -1,6 +1,67 @@
-# 8D.21 — HTTPS-first security settings
+# 8D.21 — HTTPS and SSH host identity settings
 
-## Status (2026-09-13)
+## Current aggregate status (2026-09-13)
+
+**The explicitly authorized HTTPS and remaining SSH identity slices are implemented end-to-end and host/build verified.** This completes the allowed implementation scope of 8D.21, not target acceptance, runtime reserve approval, M3, or 8D.22. The HTTPS slice's independent-review history below is preserved. The SSH slice has now been independently reviewed with no confirmed actionable defects. Reviewer checked installed wolfSSH key copying/wiping, retained-context cleanup, reservation/lock ordering, failure effects, public metadata and auth/UI scope; independently reran SSH security5, retention/management, cookie SSH7+shared, UI158+renderer/HTML/CSP, console boundary/lifecycle/policy and HTTPS lifecycle/security17 PASS. Parent final `pio run` confirmation PASS7.15s,100,556 B RAM/1,828,573 B flash; diff check PASS. Target trust renewal, real scheduling, NVS power-loss and runtime margins remain unverified.
+
+The SSH continuation began with a clean working tree and preserved the completed HTTPS shared reservations, 8D.19 SSH controller/routes, earlier UI work and 160 MHz throughput configuration. No upload, erase, commit, branch, assets regeneration, SDK/dependency/configuration changes, or new tasks/timers/queues/routes were performed.
+
+### SSH delivered workflow and exclusions
+
+Admin Settings → SSH now adds stored host-key OpenSSH SHA-256 fingerprint, identity generation, service generation and the sole canonical supported algorithm, `ecdsa-sha2-nistp256` (ECDSA P-256 / `nistp256`). This is a public metadata projection, not private/public key download or proof of the identity a peer actually served. No algorithm selector or invented host-key algorithm was added; user authorized-key algorithms and authentication are unrelated and unchanged.
+
+**Rotate SSH host identity…** confirms the exact old fingerprint and both generations before any asynchronous session validation. It explicitly warns that all SSH sessions, including subsequently admitted sessions, close; already executing administration may finish; stopped ordinary rotation stays stopped; persistence and restart may fail after partial effects. Verify the NEW fingerprint using trusted UART0 **`ssh host-key info` before accepting changed `known_hosts` trust**. Do not blindly remove warnings. HTTPS remains accessible and its login/browser terminals are not deliberately stopped; Wi-Fi, users, UART0 administration and native USB UART1 access are unchanged. USB is not an administrative recovery console.
+
+Reset audit: healthy `ssh reset --force` duplicates key replacement, additionally starts stopped SSH and permits unavailable/corrupt-material recovery. Preserve that canonical CLI/deferred-SSH behavior; do not add a duplicate browser Reset or new recovery endpoint. Full21 excludes user authorized-key work, key/certificate export, passwords/recovery secrets, unavailable-database recovery, factory erase/configuration wipe, OTA, NVS encryption and secure boot. Existing browser-shell SSH policy remains unchanged; typed SSH controls do not enable deprecated/restricted shell commands.
+
+### SSH owner and material lifetime contract
+
+- `ssh_transport_replace_identity(service_generation, identity_generation, reset, &committed)` is the shared off-HTTPD owner. Both nonzero generations select conditional rotation; both zero retain canonical semantics; reset plus conditional generations is invalid. The canonical `ssh_transport_replace_host_key()` wrapper and existing UART0/deferred admin-SSH paths use it. Conditional stopped rotation is supported without starting SSH.
+- Take the existing command mutex with zero wait, compare current initialized/service generation/transition/cleanup state, then reserve identity **before stop or crypto/NVS**. Canonical rotate/reset also now fail busy rather than waiting on another identity owner. Ordinary start/stop retain their existing command-mutex waits. The combined command mutex spans stop, replacement and conditional restart, without releasing an interleaving gap.
+- `ssh_security_reserve_identity()` uses the security mutex only for short admission. Direct canonical `ssh_security_rotate()` / `ssh_security_reset()` share this reservation. Nonreused uint32 tokens are task-owner-bound; only the reserving task can replace once and release; stale tokens cannot release or reuse a later reservation. Failed expected identity comparison has no service side effect. Init cannot publish unavailable material over a live reservation.
+- After both reservations, advance saturated service generation; retain established **stop → generate/commit/publish → conditional restart** ordering. Crypto and NVS run outside the security mutex and all spinlocks; the service command mutex remains held. Read-only public metadata/DER copies can take their short security mutex without a crypto-held lock. Existing startup initialization retains its mutex semantics.
+- Failed stop/timeout skips identity mutation and **never attempts another start**. Pending owner work is not cancelled by timeout. Failed cleanup retains canonical recovery. Generation/RNG/NVS failure after successful stop leaves live identity unchanged, but SSH clients have already disconnected; if previously running, make the established best-effort restart using unchanged material. That restart can itself fail. No blanket “failure means no effect” claim.
+- Commit precedes publication and old-private-material wipe. `committed=true` remains true if the subsequent restart fails; no rollback. Identity generations change only after successful commit, saturating without wrap. Service generation advances on admitted replacement and admitted lifecycle transitions, including failures, independently of counter clear. Reservation exhaustion blocks future identity mutation until reboot; reboot invalidates old browser logins.
+- Only the existing SSH owner task invokes runtime wolfSSH/context/socket operations. Installed `wolfSSH_CTX_UsePrivateKey_buffer()` delegates to `wolfSSH_ProcessBuffer()`; its ASN.1 path allocates and copies input DER (`internal.c` 2122–2129) before `SetHostPrivateKey`. The production caller wipes its bounded stack DER on every return path, so a running context does not borrow `s_material` or stack bytes. No SDK/library edit was needed.
+- `start_runtime()` now rejects retained context/listener/non-free slots rather than overwriting orphan handles. `stop_runtime()` retains the context until all slots retire; `process_slots()` frees a retained context on the owner, outside the spinlock, before clearing cleanup admission. The prior `cleanup_slot()` already frees its wolfSSH object before a possible broker-disconnect failure: this review did **not** demonstrate a preexisting UAF. The change enforces the explicitly requested stronger retained-context invariant. Library destructor private-key wiping was source-inspected, not exercised by a real wolfSSH target test.
+- Existing SSH self-affecting command drain/deferred-control behavior is preserved. No new dispatcher/task or command-string replay. The dispatcher does not execute its own SSH-shell host rotation synchronously; the existing control owner performs it after the handler/drain boundary. Existing authentication roles, shell-request policy, exact session generations and broker routing are unchanged.
+
+### SSH HTTP/UI bounds and result semantics
+
+Reuse GET `/api/settings/ssh` and GET/POST `/api/settings/ssh-operation`, current-admin cookie/Origin/CSRF/no-store protections, existing ID-only dispatcher and single original-login slot. No HTTPD crypto, NVS, lifecycle wait or wolfSSH call. Existing 256-byte/four-receive input, 768-byte snapshot and 96-byte result bounds remain unchanged; 39 total handlers/six sockets and unchanged queue depth/item capacity.
+
+Service requests retain exactly `action`, `generation`, `target`. Rotation requires exactly four fields, e.g. `{"action":"rotate","generation":7,"target":0,"identity_generation":3}`; unknown/duplicate/escaped fields, unsupported action, nonzero rotation target, missing/zero/saturated identity generation, coercion, malformed or oversized bodies reject. Snapshot adds `identity_generation`, `algorithm`, `fingerprint`, `rotatable` to the existing four fields. Fingerprint is unpadded OpenSSH `SHA256:` base64. Security metadata is atomically copied under a zero-wait lock, with no private material. Service and identity observations are separate, not a cross-owner atomic authorization; execution compares/reserves both. Unavailable identity yields generation0/empty fingerprint/rotatable false without removing ordinary service controls.
+
+Queue admission and response follow the existing SSH post-before-execute pattern, **not** the HTTPS self-cutting ACK gate: SSH rotation does not stop the invoking HTTPD/login. A lost response does not cancel queued work. Original-login/current-admin/30-second dequeue deadline checks precede combined owner admission; revocation after admission is not cancellation. Completed duplicate IDs are inert, IDs do not wrap, results are replaceable and login-isolated, not durable/idempotent history. Rotation errors conservatively report `failed`, even for rejected owner admission, because later failures can have partial effects; ordinary service conflict reporting is unchanged.
+
+The existing controller keeps 15-second whole-request bounds, captured confirmation values, single-flight/pending gates, manual Check Operation Result then Refresh, navigation/late-response fencing and no mutation retry or restore. Terminal errors/results explicitly warn that SSH may have disconnected despite persistence failure, or a key may be persisted despite restart failure. HTTPS does not require fresh login due to this SSH operation. No new polling timer or separate controller.
+
+### SSH validation and resources
+
+Commands actually run, all PASS after the described harness fixes:
+
+- `python3 tests/ssh_management/security.py`: five groups, full production security + real host mbedTLS and NVS fault doubles, plus extracted exact production combined owner functions. Covers P-256 generation/validation/copy/reload, RNG/NVS open/set/commit faults and unchanged stored/live bytes, commit-before-publication, stale admission before effects, competing canonical/direct owners during crypto, task/token ownership/reuse/one-shot/exhaustion, postcommit restart failure/no rollback, zero-wait metadata, malformed storage and canonical reset recovery. NVS handle closure and candidate wipes checked. No real power-loss/RTOS scheduling claims.
+- `python3 tests/ssh_management/runtime.py`: exact production runtime start/stop/process-slot functions; deterministic retained-resource doubles prove failed-stop retention, no orphan overwrite/start, owner-only final retirement and listener-failure cleanup. Not a real wolfSSH allocator/socket test.
+- `python3 tests/ssh_management/run.py`: five groups, existing published snapshot/session-close/ABA/retired-ID/timeout/exhaustion tests plus combined identity-owner comparison, busy reservation, failed-stop no start/mutation, persistence recovery and stopped/reset semantics.
+- `python3 tests/web_cookie_auth/run.py --ssh`: seven SSH groups plus shared auth; strict rotation fields/generations, dispatcher-only single execution, revoked queued rotation, login isolation, metadata bounds, existing Origin/CSRF/session/receive/expiry/deadline/lost-response tests. Uses owner doubles, not crypto.
+- `python3 tests/web_ui_session/run.py`: **158 groups** plus production C rendering/HTML/CSP checks. Adds rotation confirmation/fingerprint/both generations, exact request, all-SSH/trust/UART0/partial-effect warnings, duplicate suppression, HTTPS terminal isolation and malformed/unavailable identity metadata; all prior domains remain green.
+- Broad PASS: `tests/admin_console_boundary/run.py`, `tests/admin_console_boundary/accounts.py`, `tests/admin_console_boundary/lifecycle.py`, `tests/admin_ssh_policy/run.py`, `tests/web_admin_transport/server_lifecycle.py` (44+2 integrated real HTTPS security groups), `tests/web_security/run.py` (17), all cookie variants (`--admin`, `--settings`, `--serial-settings`, `--accounts`, `--network`, `--display`, `--broker`, `--lifecycle`, `--ssh`), `tests/web_admin_transport/run.py --tickets`, `tests/web_session_store/run.py --serial`, `tests/web_serial_performance/run.py`, `tests/web_httpd_idle/run.py`, `tests/web_auth_parse/run.py`, `tests/web_network_settings/run.py`, `tests/session_broker_diagnostics/run.py`, `tests/web_diagnostics/run.py`, `tests/web_login_ui/run.py` (all invoked with `python3`).
+- Initial compile caught an enum-type comparison in the new API action; fixed with the module's unsigned action value. Cookie harness needed real `-lmbedcrypto` for base64; integrated owner harness needed its extracted generation constant. These were corrected and affected suites rerun successfully.
+- Baseline `pio run`: PASS7.20s, **100,532 B RAM / 1,825,073 B flash**. Final production build after retained-context changes: PASS21.98s, **100,556 B RAM / 1,828,573 B flash**, SSH slice delta **+24 B / +3,500 B**; aggregate21 delta versus pre-HTTPS100,508/1,821,505 is **+48 B / +7,068 B**. These are linked static/flash counts, not heap or stack reserves.
+- Final confirmation `pio run` PASS7.07s at identical100,556/1,828,573 B; final SSH security/runtime/management/cookie/UI158 and console accounts/lifecycle reruns PASS. No independent-review attribution is implied by these same-agent reruns.
+- Defaults, active sdkconfig and generated sdkconfig.h explicitly checked: CPU **160 MHz**. Board-banner240MHz is not the configured CPU clock. Combined binary WebSocket send path/config/assets untouched. `git diff --check` PASS.
+
+### Pending SSH parent and target gates
+
+Independent parent review is required after this implementation; no independent review was performed or fabricated. Review combined-owner lock ordering, command/control self-deferral, direct-security exclusion, retained-context recovery and actual DER-copy/destructor semantics, plus API/UI partial-effect contracts. No target acceptance is claimed.
+
+On device: compare stored fingerprint with trusted UART0 `ssh host-key info` and actual peer host key; confirm changed-known_hosts verification; rotate with active admin/user SSH and concurrent HTTPS/USB/full mix; verify all SSH disconnect while HTTPS stays accessible; test stopped rotation/reset recovery, service restart/CLI interleavings, broker cleanup failures, reboot persistence and realistic NVS/power-loss faults. Measure dispatcher/control/SSH/HTTPD stack high-water, heap/internal/DMA minima and throughput at160MHz. Confirm unchanged account roles/authorized keys and ordinary/restricted shell behavior. Hardware trust, timing, power-loss, real-wolfSSH allocation/failure behavior and runtime margins remain unmeasured. No new phase is authorized by this record.
+
+---
+
+## Historical HTTPS slice status (2026-09-13)
+
+The following record preserves the completed HTTPS slice and its independent-review evidence. Its statements that SSH/full21 remain unimplemented are historical and superseded by the aggregate status above; its target-pending limits still apply.
 
 **Chosen HTTPS slice implemented end-to-end, host-tested and build-verified.**
 Independent review complete with no confirmed actionable findings; target validation/sign-off remains pending. Full