ESP32_Serial_Swiss_Army_Knife/docs/phase8d21_implementation.md

# 8D.21 — HTTPS-first security settings

## Status (2026-09-13)

**Chosen HTTPS slice implemented end-to-end, host-tested and build-verified.**
Independent review complete with no confirmed actionable findings; target validation/sign-off remains pending. Full
8D.21 remains incomplete: SSH identity work requires its separately requested
owner/scope audit and implementation. No SSH identity changes were made here.

The previous agent delivered only an audit and baseline, modifying this document,
`docs/agent/current-state.md` and `docs/phase8d_plan.md`. Those were the only initial
worktree changes in this continuation. This implementation supersedes their
NOT-implemented status, not prior user sign-offs or resource follow-ups.

Independent reviewer checked shared service/identity reservations, all canonical mutation callers, installed-IDF TLS material copies, commit/failure semantics, auth/ACK and public metadata. Reviewer reran security17, lifecycle44+2 integration groups, cookie lifecycle8+shared, UI/renderer/HTML/CSP, dispatcher/console lifecycle and diff checks PASS. Parent final `pio run` confirmation PASS6.98s,100,532 B RAM/1,825,073 B flash; diff check PASS. No target trust/reset/NVS power-loss or runtime-margin validation inferred.

## Delivered scope and user workflow

- Admin Settings → **HTTPS / Reboot** now also shows the **stored** HTTPS
  certificate's SHA-256 fingerprint, identity generation and service generation.
  This is public metadata behind current-admin authorization, not an unauthenticated
  endpoint, certificate download, private-key export or proof of the served identity.
- **Rotate HTTPS identity…** confirms the old fingerprint and both generations,
  persistence, changed browser trust and loss of all web logins/both browser terminal
  routes, including clients admitted before execution. Save browser drafts first.
- Verify the NEW fingerprint using trusted UART0 `web certificate info` before
  accepting changed browser trust. A browser certificate warning alone is not
  trusted verification. Reload and sign in freshly; never automatically retry.
- A new identity may already be persisted even if stop/start fails. A retained
  HTTPD can still serve the old certificate; compare stored and served identity
  explicitly and recover using canonical `web stop` / `web start` when necessary.
- SSH identity, accounts, Wi-Fi, serial configuration and other saved settings
  are unchanged. UART0 administration and native USB UART1 access are independent
  of this rotation. USB is not an administrative console. The preexisting reboot
  action still interrupts the entire device; rotation does not reboot it.
- No duplicate browser Reset for healthy TLS material. Existing CLI
  `web reset --force` retains TLS-only replacement/recovery and starts a stopped
  HTTPS service; ordinary CLI rotation leaves a stopped service stopped. Neither
  touches users. Unavailable-user-database recovery stays UART0-only. No browser
  unavailable-material recovery, bootstrap, retained recovery secret, new password
  feature, configuration wipe, factory reset, OTA, encryption or secure boot.

## Ownership and admission contract

### Combined service/security owner

`web_server_replace_identity(service_generation, identity_generation, reset,
&committed)` is the shared off-HTTPD owner entry point. Both nonzero generations
select conditional healthy-running rotation; both zero select existing canonical
CLI semantics. Reset cannot be combined with conditional generations.

1. Conditional calls avoid the blocking legacy initializer and take the existing
   server mutex with zero wait. Reject stale/saturated service generation, absent
   server, transition or failed cleanup **before any identity mutation**.
2. Set `s_transitioning` under the server mutex, then release it. This reservation
   excludes canonical start/stop/restart/reboot and competing CLI identity work.
3. Reserve the expected identity using `web_security_reserve_identity()`. Its
   zero-wait security mutex check excludes stale/unavailable/saturated identity,
   existing reservation and exhausted reservation IDs. Canonical direct
   `web_security_rotate_certificate()` / `web_security_reset_all()` acquire the
   same reservation, so they cannot bypass an admitted combined operation.
4. After both reservations, advance the saturated service generation. Generate a
   local candidate and commit its bounded existing NVS blob without holding either
   mutex or a spinlock. Publish only after successful commit, wiping the old key.
   The security mutex protects short state admission/copies/publication, not crypto
   or flash. Initialization/migration retains its existing startup locking.
5. Keep the identity reservation through canonical reserved stop/start. Start can
   copy the committed TLS pair under the ordinary security mutex without deadlock.
   Running rotation/reset restarts HTTPS; stopped rotation remains stopped and
   stopped reset starts HTTPS. No unreserved stop-to-start gap is introduced.
6. Release the nonreused identity token on return. Only its reserving owner may
   replace once and release it; old tokens cannot replace/release a new reservation.

Identity generation advances only on committed replacement, retaining the existing
persisted UINT32_MAX fail-closed rule. Service generation advances on admitted
replacement and on each admitted stop/start, including failed attempts, saturating
at UINT32_MAX without wrapping. Failed identity admission does not advance it;
failed crypto/storage after combined admission can advance service generation
without changing identity or stopping HTTPD. Reservation IDs never reuse; exhaustion
denies identity mutations until reboot, while existing canonical service recovery
remains available. Reboot invalidates all old browser sessions.

Public snapshot copies use separate zero-wait service/security observations, not
one cross-owner atomic view. They authorize nothing; any intervening service or
identity change is rejected by the combined execution-time reservations. Unavailable
identity metadata yields generation0/empty fingerprint/rotatable false without
removing ordinary lifecycle controls.

### Failures, revocation and console safety

- Generation, RNG, NVS open/set/commit errors before publication leave the live
  identity unchanged and do not stop HTTPD or revoke its logins. Host NVS faults
  also verify unchanged stored bytes; doubles do not prove power-loss behavior.
- `committed=true` is irreversible publication even when stop/start returns an
  error. Stop failure skips start and retains canonical cleanup ownership. Failed
  start follows the existing retained-handle cleanup rules. No rollback to the old
  identity occurs. CLI reports the precommit/postcommit distinction; typed results
  deliberately retain conservative failure/unknown-outcome language.
- Stop invokes canonical `web_cookie_auth_stop()` and transport detach/stopped
  paths, invalidating all web sessions/tickets and closing browser routes. No
  unrelated account or SSH revocation is added. Later logout/revocation is not
  cancellation of admitted crypto/commit/restart.
- `web_console.c` rotate/reset wrappers and `web_admin_transport.c` deferred
  browser-shell rotation call the same combined owner, not separate commit/stop/start.
  Browser shell retains its existing parsed force policy and drain/dispatcher mask;
  its acknowledgement is still the bounded application-drain heuristic, not HTTP ACK.
- Inspected `web_admin_transport_detach()`: it waits only for queue submitters, not
  console execution. `admin_ssh_console_close()` marks an executing slot inactive
  without waiting/wiping it; dispatcher return retires that slot. No self-console
  cleanup wait, command-string replay, HTTPD self-stop or lock-held HTTPD wait.

## Bounded HTTP and UI contracts

Reuse the three existing optional routes and single lifecycle operation slot:

- `GET /api/settings/lifecycle`: seven fields, **320-byte response buffer**:
  `generation`, `running`, `transitioning`, `controllable`, `identity_generation`,
  `fingerprint` (64 lowercase hex digits, SHA-256), `rotatable`.
- `POST /api/settings/lifecycle-operation`: `stop|restart|reboot` still require
  exactly action + service generation. `rotate` requires exactly
  `{"action":"rotate","generation":7,"identity_generation":11}`.
  No reset, extra fields, duplicate fields, escaped/coerced values, zero/saturated
  generations or trailing bytes. Existing **256-byte/four-receive** bound retained.
- `GET /api/settings/lifecycle-operation`: existing **96-byte** login-isolated
  `id/action/state` result. `failed` can mean rejected admission OR partial effects;
  no falsely precise success, cancellation, rollback or durable-history guarantee.

Current cookie/principal/admin checks protect all routes; mutations additionally
require existing JSON/Origin/CSRF policy. The operation keeps the original login,
not merely the account. Reuse8D.20's send-return → nonreused-ID HTTPD callback →
existing dispatcher handoff: no crypto, persistence or lifecycle on HTTPD and no
request/fd/operation pointer captured. Two-second ACK and post-validation30-second
dequeue deadlines remain admission bounds, not peer receipt or execution deadlines.
One accepted-but-lost callback reservation survives until callback arrival or
successful HTTPD destruction; failed destruction cannot accumulate new callbacks.

`web_ui.c` extends the existing lifecycle view/controller, not a parallel Security
controller. It validates the complete bounded public schema, captures both versions
before confirmation/session validation, and shares single-flight/pending gates with
all lifecycle actions. Identity unavailable/exhausted disables rotation. Explicit
manual Check Result/Refresh,15-second whole-request deadline, late-result/context
fencing and original-login rules remain. Unknown/lost ACK cannot clear the duplicate
gate by adopting another result; navigation/pagehide does not replay or restore a
mutation. No generated secret, secret input or retained retry material is introduced.
No generated assets or loader/CSP changes; existing HTML/CSP tests pass.

## Validation performed

All commands below passed in this continuation; no hardware/runtime pass inferred.

- `python3 tests/web_security/run.py`: **17 production-security groups** with real
  host mbedTLS, including exact NVS open/set/commit transactional failures, real
  fingerprint/key changes, generation exhaustion, public-only projection, zero-wait
  contention, one-use/nonreused tokens and competing canonical writers during real
  crypto outside locks. Existing migration/invalid storage/recovery tests retained.
- `python3 tests/web_admin_transport/server_lifecycle.py`: **44 orchestration/ACK
  groups plus2 integrated owner/security groups**. The second executable links the
  extracted production server owner to the complete production security implementation
  and real mbedTLS, with NVS/HTTPD/scheduler doubles. Tests real generation-time RNG
  and NVS open/set/commit failures, exact unchanged identity/storage before commit,
  committed identity surviving stop/start failures, stale service/identity rejection,
  and canonical/direct-security exclusion during crypto AND stop/start. The44 groups
  additionally cover CLI stopped rotation/reset, saturated/stale/busy/currentness
  admission, every detach failure, original ACK sequencing and lifecycle recovery.
- `python3 tests/web_cookie_auth/run.py --lifecycle`: **8 extended lifecycle groups
  plus shared auth tests**, actual handlers/parser/auth/session store; rotate's exact
  third generation, malformed/truncated inputs, bounded fingerprint-only projection,
  all four actions off-handler/off-callback, login invalidation after admission,
  queue/send/lost/late callback/request storage reuse and deadline/wrap regressions.
- `python3 tests/web_ui_session/run.py`: **156 browser behavior groups plus C-renderer,
  seven-view HTML and exact CSP checks**. Three new groups cover fingerprint/version
  confirmation, missing/invalid generation/fingerprint/secret-field rejection,
  unavailable identity, shared pending gate, partial-effect warning, role loss,
  lost ACK, expired login and no replay/restore. Existing lifecycle timing and
  navigation tests remain. Not real-browser geometry or TLS target validation.
- `python3 tests/admin_console_boundary/run.py` and `lifecycle.py`: PASS, including
  deferred browser certificate dispatcher/executing-slot/self-detach safety and
  typed dispatcher queue isolation. `python3 tests/web_admin_transport/run.py`:
  **25 groups**; its owner dependency double now represents the combined API.
- `python3 tests/web_cookie_auth/run.py` with no option and each of `--settings`,
  `--serial-settings`, `--accounts`, `--network`, `--display`, `--broker`, `--ssh`,
  `--lifecycle`, `--admin`: PASS.
- `python3 tests/<suite>/run.py` for `admin_ssh_policy`, `web_auth_parse`,
  `web_session_store`, `web_httpd_idle`, `web_serial_performance`, `web_diagnostics`,
  `web_login_ui`, `ssh_management`, `web_network_settings`,
  `session_broker_diagnostics`, `web_security`, `web_admin_tickets`: PASS.
  `python3 tests/web_session_store/run.py --serial`: PASS.
- Final combined owner/integration suite rerun after adding generation-time RNG and
  stop/start reservation interleavings: PASS. The initial integration-fixture compile
  found an unused copied test counter; fixture extraction was corrected, not compiler
  warnings disabled. No production test failure was hidden.
- `git --no-pager diff --check`: PASS. Final worktree inspection shows only the
  intended source/tests/docs; no assets, configuration or device-operation changes.

## Build, resources and preserved work

- Prior audit baseline: `pio run` PASS7.17s, **100,508 B RAM / 1,821,505 B flash**.
  It was not redundantly rerun before implementation.
- Intermediate implementation build: PASS21.92s,100,532 /1,824,889 B.
- Final production `pio run`: **PASS22.78s,100,532 B RAM /1,825,073 B flash**.
  Final delta against audited baseline: **+24 B RAM /+3,568 B flash**.
- Still **39 handlers/six sockets**, same existing lifecycle slot/dispatcher item
  and queue depth, no new task/timer/stack size/session/ticket/heap allocation/schema.
  Lifecycle snapshot buffer grows128→320 bytes; identity-operation slot adds one
  uint32 field; security adds bounded token/one-use state. The existing local TLS
  candidate is retained, not another persistent private-material allocation.
- Defaults, active sdkconfig and generated sdkconfig.h independently checked:
  **CPU160MHz**. No configuration changes. Combined binary WebSocket send,
  signed-off throughput, serial broker isolation and native USB path unchanged.
- No assets regeneration, SDK/dependency changes, device actions, upload, erase,
  commit or new branch. User/prior-agent documentation work preserved and updated.
  Runtime HTTPD/dispatcher stack margins and heap minima are not inferred from
  linked RAM or host sizeof; the larger snapshot handler frame needs target checks.

## Pending parent/target gates and remaining phase

- [ ] Independent parent review of combined ownership, direct canonical exclusions,
      snapshots/ACK/auth/parser/UI and failure/uncertainty contracts.
- [ ] On target, verify old/new stored and served fingerprints, trusted UART0
      verification, browser trust renewal, fresh login and persisted identity after
      a normal reboot (no automatic reboot in this feature).
- [ ] Verify canonical CLI reset with unavailable TLS material and stopped-service
      rotation/reset semantics, including low-memory/cleanup failure recovery.
- [ ] Real-client lost response, timeout, two-admin concurrency, stale confirmations,
      changed role/login and cleanup behavior; no mutation replay.
- [ ] Full mix at160MHz (two web serial, browser admin, SSH serial/admin and USB),
      unchanged binary transparency/writer isolation, UART0 recovery/native USB
      availability through HTTPS rotation, HTTPD/dispatcher stack and heap minima.
- [ ] User target sign-off. No reserve approval, full M3 or full8D.21 claim.

**HTTPS slice has no remaining implementation item identified here.** SSH identity
metadata/rotation/reset remains outside this slice and requires a separate request;
no speculative SSH owner design or implementation is included.