Add HTTPS identity rotation support

docs/phase8d21_implementation.md

@@ -0,0 +1,233 @@
+# 8D.21 — HTTPS-first security settings
+
+## Status (2026-09-13)
+
+**Chosen HTTPS slice implemented end-to-end, host-tested and build-verified.**
+Independent review complete with no confirmed actionable findings; target validation/sign-off remains pending. Full
+8D.21 remains incomplete: SSH identity work requires its separately requested
+owner/scope audit and implementation. No SSH identity changes were made here.
+
+The previous agent delivered only an audit and baseline, modifying this document,
+`docs/agent/current-state.md` and `docs/phase8d_plan.md`. Those were the only initial
+worktree changes in this continuation. This implementation supersedes their
+NOT-implemented status, not prior user sign-offs or resource follow-ups.
+
+Independent reviewer checked shared service/identity reservations, all canonical mutation callers, installed-IDF TLS material copies, commit/failure semantics, auth/ACK and public metadata. Reviewer reran security17, lifecycle44+2 integration groups, cookie lifecycle8+shared, UI/renderer/HTML/CSP, dispatcher/console lifecycle and diff checks PASS. Parent final `pio run` confirmation PASS6.98s,100,532 B RAM/1,825,073 B flash; diff check PASS. No target trust/reset/NVS power-loss or runtime-margin validation inferred.
+
+## Delivered scope and user workflow
+
+- Admin Settings → **HTTPS / Reboot** now also shows the **stored** HTTPS
+  certificate's SHA-256 fingerprint, identity generation and service generation.
+  This is public metadata behind current-admin authorization, not an unauthenticated
+  endpoint, certificate download, private-key export or proof of the served identity.
+- **Rotate HTTPS identity…** confirms the old fingerprint and both generations,
+  persistence, changed browser trust and loss of all web logins/both browser terminal
+  routes, including clients admitted before execution. Save browser drafts first.
+- Verify the NEW fingerprint using trusted UART0 `web certificate info` before
+  accepting changed browser trust. A browser certificate warning alone is not
+  trusted verification. Reload and sign in freshly; never automatically retry.
+- A new identity may already be persisted even if stop/start fails. A retained
+  HTTPD can still serve the old certificate; compare stored and served identity
+  explicitly and recover using canonical `web stop` / `web start` when necessary.
+- SSH identity, accounts, Wi-Fi, serial configuration and other saved settings
+  are unchanged. UART0 administration and native USB UART1 access are independent
+  of this rotation. USB is not an administrative console. The preexisting reboot
+  action still interrupts the entire device; rotation does not reboot it.
+- No duplicate browser Reset for healthy TLS material. Existing CLI
+  `web reset --force` retains TLS-only replacement/recovery and starts a stopped
+  HTTPS service; ordinary CLI rotation leaves a stopped service stopped. Neither
+  touches users. Unavailable-user-database recovery stays UART0-only. No browser
+  unavailable-material recovery, bootstrap, retained recovery secret, new password
+  feature, configuration wipe, factory reset, OTA, encryption or secure boot.
+
+## Ownership and admission contract
+
+### Combined service/security owner
+
+`web_server_replace_identity(service_generation, identity_generation, reset,
+&committed)` is the shared off-HTTPD owner entry point. Both nonzero generations
+select conditional healthy-running rotation; both zero select existing canonical
+CLI semantics. Reset cannot be combined with conditional generations.
+
+1. Conditional calls avoid the blocking legacy initializer and take the existing
+   server mutex with zero wait. Reject stale/saturated service generation, absent
+   server, transition or failed cleanup **before any identity mutation**.
+2. Set `s_transitioning` under the server mutex, then release it. This reservation
+   excludes canonical start/stop/restart/reboot and competing CLI identity work.
+3. Reserve the expected identity using `web_security_reserve_identity()`. Its
+   zero-wait security mutex check excludes stale/unavailable/saturated identity,
+   existing reservation and exhausted reservation IDs. Canonical direct
+   `web_security_rotate_certificate()` / `web_security_reset_all()` acquire the
+   same reservation, so they cannot bypass an admitted combined operation.
+4. After both reservations, advance the saturated service generation. Generate a
+   local candidate and commit its bounded existing NVS blob without holding either
+   mutex or a spinlock. Publish only after successful commit, wiping the old key.
+   The security mutex protects short state admission/copies/publication, not crypto
+   or flash. Initialization/migration retains its existing startup locking.
+5. Keep the identity reservation through canonical reserved stop/start. Start can
+   copy the committed TLS pair under the ordinary security mutex without deadlock.
+   Running rotation/reset restarts HTTPS; stopped rotation remains stopped and
+   stopped reset starts HTTPS. No unreserved stop-to-start gap is introduced.
+6. Release the nonreused identity token on return. Only its reserving owner may
+   replace once and release it; old tokens cannot replace/release a new reservation.
+
+Identity generation advances only on committed replacement, retaining the existing
+persisted UINT32_MAX fail-closed rule. Service generation advances on admitted
+replacement and on each admitted stop/start, including failed attempts, saturating
+at UINT32_MAX without wrapping. Failed identity admission does not advance it;
+failed crypto/storage after combined admission can advance service generation
+without changing identity or stopping HTTPD. Reservation IDs never reuse; exhaustion
+denies identity mutations until reboot, while existing canonical service recovery
+remains available. Reboot invalidates all old browser sessions.
+
+Public snapshot copies use separate zero-wait service/security observations, not
+one cross-owner atomic view. They authorize nothing; any intervening service or
+identity change is rejected by the combined execution-time reservations. Unavailable
+identity metadata yields generation0/empty fingerprint/rotatable false without
+removing ordinary lifecycle controls.
+
+### Failures, revocation and console safety
+
+- Generation, RNG, NVS open/set/commit errors before publication leave the live
+  identity unchanged and do not stop HTTPD or revoke its logins. Host NVS faults
+  also verify unchanged stored bytes; doubles do not prove power-loss behavior.
+- `committed=true` is irreversible publication even when stop/start returns an
+  error. Stop failure skips start and retains canonical cleanup ownership. Failed
+  start follows the existing retained-handle cleanup rules. No rollback to the old
+  identity occurs. CLI reports the precommit/postcommit distinction; typed results
+  deliberately retain conservative failure/unknown-outcome language.
+- Stop invokes canonical `web_cookie_auth_stop()` and transport detach/stopped
+  paths, invalidating all web sessions/tickets and closing browser routes. No
+  unrelated account or SSH revocation is added. Later logout/revocation is not
+  cancellation of admitted crypto/commit/restart.
+- `web_console.c` rotate/reset wrappers and `web_admin_transport.c` deferred
+  browser-shell rotation call the same combined owner, not separate commit/stop/start.
+  Browser shell retains its existing parsed force policy and drain/dispatcher mask;
+  its acknowledgement is still the bounded application-drain heuristic, not HTTP ACK.
+- Inspected `web_admin_transport_detach()`: it waits only for queue submitters, not
+  console execution. `admin_ssh_console_close()` marks an executing slot inactive
+  without waiting/wiping it; dispatcher return retires that slot. No self-console
+  cleanup wait, command-string replay, HTTPD self-stop or lock-held HTTPD wait.
+
+## Bounded HTTP and UI contracts
+
+Reuse the three existing optional routes and single lifecycle operation slot:
+
+- `GET /api/settings/lifecycle`: seven fields, **320-byte response buffer**:
+  `generation`, `running`, `transitioning`, `controllable`, `identity_generation`,
+  `fingerprint` (64 lowercase hex digits, SHA-256), `rotatable`.
+- `POST /api/settings/lifecycle-operation`: `stop|restart|reboot` still require
+  exactly action + service generation. `rotate` requires exactly
+  `{"action":"rotate","generation":7,"identity_generation":11}`.
+  No reset, extra fields, duplicate fields, escaped/coerced values, zero/saturated
+  generations or trailing bytes. Existing **256-byte/four-receive** bound retained.
+- `GET /api/settings/lifecycle-operation`: existing **96-byte** login-isolated
+  `id/action/state` result. `failed` can mean rejected admission OR partial effects;
+  no falsely precise success, cancellation, rollback or durable-history guarantee.
+
+Current cookie/principal/admin checks protect all routes; mutations additionally
+require existing JSON/Origin/CSRF policy. The operation keeps the original login,
+not merely the account. Reuse8D.20's send-return → nonreused-ID HTTPD callback →
+existing dispatcher handoff: no crypto, persistence or lifecycle on HTTPD and no
+request/fd/operation pointer captured. Two-second ACK and post-validation30-second
+dequeue deadlines remain admission bounds, not peer receipt or execution deadlines.
+One accepted-but-lost callback reservation survives until callback arrival or
+successful HTTPD destruction; failed destruction cannot accumulate new callbacks.
+
+`web_ui.c` extends the existing lifecycle view/controller, not a parallel Security
+controller. It validates the complete bounded public schema, captures both versions
+before confirmation/session validation, and shares single-flight/pending gates with
+all lifecycle actions. Identity unavailable/exhausted disables rotation. Explicit
+manual Check Result/Refresh,15-second whole-request deadline, late-result/context
+fencing and original-login rules remain. Unknown/lost ACK cannot clear the duplicate
+gate by adopting another result; navigation/pagehide does not replay or restore a
+mutation. No generated secret, secret input or retained retry material is introduced.
+No generated assets or loader/CSP changes; existing HTML/CSP tests pass.
+
+## Validation performed
+
+All commands below passed in this continuation; no hardware/runtime pass inferred.
+
+- `python3 tests/web_security/run.py`: **17 production-security groups** with real
+  host mbedTLS, including exact NVS open/set/commit transactional failures, real
+  fingerprint/key changes, generation exhaustion, public-only projection, zero-wait
+  contention, one-use/nonreused tokens and competing canonical writers during real
+  crypto outside locks. Existing migration/invalid storage/recovery tests retained.
+- `python3 tests/web_admin_transport/server_lifecycle.py`: **44 orchestration/ACK
+  groups plus2 integrated owner/security groups**. The second executable links the
+  extracted production server owner to the complete production security implementation
+  and real mbedTLS, with NVS/HTTPD/scheduler doubles. Tests real generation-time RNG
+  and NVS open/set/commit failures, exact unchanged identity/storage before commit,
+  committed identity surviving stop/start failures, stale service/identity rejection,
+  and canonical/direct-security exclusion during crypto AND stop/start. The44 groups
+  additionally cover CLI stopped rotation/reset, saturated/stale/busy/currentness
+  admission, every detach failure, original ACK sequencing and lifecycle recovery.
+- `python3 tests/web_cookie_auth/run.py --lifecycle`: **8 extended lifecycle groups
+  plus shared auth tests**, actual handlers/parser/auth/session store; rotate's exact
+  third generation, malformed/truncated inputs, bounded fingerprint-only projection,
+  all four actions off-handler/off-callback, login invalidation after admission,
+  queue/send/lost/late callback/request storage reuse and deadline/wrap regressions.
+- `python3 tests/web_ui_session/run.py`: **156 browser behavior groups plus C-renderer,
+  seven-view HTML and exact CSP checks**. Three new groups cover fingerprint/version
+  confirmation, missing/invalid generation/fingerprint/secret-field rejection,
+  unavailable identity, shared pending gate, partial-effect warning, role loss,
+  lost ACK, expired login and no replay/restore. Existing lifecycle timing and
+  navigation tests remain. Not real-browser geometry or TLS target validation.
+- `python3 tests/admin_console_boundary/run.py` and `lifecycle.py`: PASS, including
+  deferred browser certificate dispatcher/executing-slot/self-detach safety and
+  typed dispatcher queue isolation. `python3 tests/web_admin_transport/run.py`:
+  **25 groups**; its owner dependency double now represents the combined API.
+- `python3 tests/web_cookie_auth/run.py` with no option and each of `--settings`,
+  `--serial-settings`, `--accounts`, `--network`, `--display`, `--broker`, `--ssh`,
+  `--lifecycle`, `--admin`: PASS.
+- `python3 tests/<suite>/run.py` for `admin_ssh_policy`, `web_auth_parse`,
+  `web_session_store`, `web_httpd_idle`, `web_serial_performance`, `web_diagnostics`,
+  `web_login_ui`, `ssh_management`, `web_network_settings`,
+  `session_broker_diagnostics`, `web_security`, `web_admin_tickets`: PASS.
+  `python3 tests/web_session_store/run.py --serial`: PASS.
+- Final combined owner/integration suite rerun after adding generation-time RNG and
+  stop/start reservation interleavings: PASS. The initial integration-fixture compile
+  found an unused copied test counter; fixture extraction was corrected, not compiler
+  warnings disabled. No production test failure was hidden.
+- `git --no-pager diff --check`: PASS. Final worktree inspection shows only the
+  intended source/tests/docs; no assets, configuration or device-operation changes.
+
+## Build, resources and preserved work
+
+- Prior audit baseline: `pio run` PASS7.17s, **100,508 B RAM / 1,821,505 B flash**.
+  It was not redundantly rerun before implementation.
+- Intermediate implementation build: PASS21.92s,100,532 /1,824,889 B.
+- Final production `pio run`: **PASS22.78s,100,532 B RAM /1,825,073 B flash**.
+  Final delta against audited baseline: **+24 B RAM /+3,568 B flash**.
+- Still **39 handlers/six sockets**, same existing lifecycle slot/dispatcher item
+  and queue depth, no new task/timer/stack size/session/ticket/heap allocation/schema.
+  Lifecycle snapshot buffer grows128→320 bytes; identity-operation slot adds one
+  uint32 field; security adds bounded token/one-use state. The existing local TLS
+  candidate is retained, not another persistent private-material allocation.
+- Defaults, active sdkconfig and generated sdkconfig.h independently checked:
+  **CPU160MHz**. No configuration changes. Combined binary WebSocket send,
+  signed-off throughput, serial broker isolation and native USB path unchanged.
+- No assets regeneration, SDK/dependency changes, device actions, upload, erase,
+  commit or new branch. User/prior-agent documentation work preserved and updated.
+  Runtime HTTPD/dispatcher stack margins and heap minima are not inferred from
+  linked RAM or host sizeof; the larger snapshot handler frame needs target checks.
+
+## Pending parent/target gates and remaining phase
+
+- [ ] Independent parent review of combined ownership, direct canonical exclusions,
+      snapshots/ACK/auth/parser/UI and failure/uncertainty contracts.
+- [ ] On target, verify old/new stored and served fingerprints, trusted UART0
+      verification, browser trust renewal, fresh login and persisted identity after
+      a normal reboot (no automatic reboot in this feature).
+- [ ] Verify canonical CLI reset with unavailable TLS material and stopped-service
+      rotation/reset semantics, including low-memory/cleanup failure recovery.
+- [ ] Real-client lost response, timeout, two-admin concurrency, stale confirmations,
+      changed role/login and cleanup behavior; no mutation replay.
+- [ ] Full mix at160MHz (two web serial, browser admin, SSH serial/admin and USB),
+      unchanged binary transparency/writer isolation, UART0 recovery/native USB
+      availability through HTTPS rotation, HTTPD/dispatcher stack and heap minima.
+- [ ] User target sign-off. No reserve approval, full M3 or full8D.21 claim.
+
+**HTTPS slice has no remaining implementation item identified here.** SSH identity
+metadata/rotation/reset remains outside this slice and requires a separate request;
+no speculative SSH owner design or implementation is included.