Add admin firmware upload support

Implement authenticated HTTPS OTA uploads with bounded streaming, image
validation, reboot coordination, and lifecycle exclusion. Add the admin
UI,
regression tests, and Phase 10 acceptance documentation.
This commit is contained in:
2026-09-18 22:22:11 +02:00
parent 4f628a4098
commit 31a22eba06
31 changed files with 1442 additions and 88 deletions
+7 -1
View File
@@ -104,7 +104,7 @@ TinyUSB callbacks enqueue/copy data and state; the transport task owns broker li
### HTTPS, WebSocket, and web serial
`web_server` owns HTTPS on port 443 with a persisted self-signed P-256 identity. `web_serial_transport` mediates two fixed WebSocket slots through the broker; HTTPD owns socket sends/close, the transport task owns broker IO. Four outstanding serial tickets, four cookie sessions, one optional admin WebSocket and six total HTTPD sockets are distinct limits; LRU is disabled. Current handler capacity is 39. Base HTTPS can serve authenticated non-WebSocket routes if optional serial/admin transport initialization fails.
`web_server` owns HTTPS on port 443 with a persisted self-signed P-256 identity. `web_serial_transport` mediates two fixed WebSocket slots through the broker; HTTPD owns socket sends/close, the transport task owns broker IO. Four outstanding serial tickets, four cookie sessions, one optional admin WebSocket and six total HTTPD sockets are distinct limits; LRU is disabled. Current handler capacity is 40. Base HTTPS can serve authenticated non-WebSocket routes if optional serial/admin transport initialization fails.
Cookie login/logout replaces Basic/cache. Digest-only records carry copied principals, CSRF state, absolute expiry and nonreused originating-session IDs. Strict same-origin/CSRF mutations and session/principal checks gate admission; logout invalidates its session before transport cleanup, account mutations invalidate only the affected account, and ongoing currentness is authoritative. Authentication initialization failure gates HTTPS; failed start/accepted stop wipes records. RNG/SHA/database calls run outside short spinlocks with post-call epoch/identity revalidation. [Authentication contract](../web_administration.md#authentication-and-admission).
@@ -122,6 +122,12 @@ Cookie login/logout replaces Basic/cache. Digest-only records carry copied princ
`web_server_replace_identity` reserves service before identity and retains both through commit → reserved stop/start. Direct security and canonical CLI/browser-shell paths share task-bound nonreused identity reservations. Crypto/NVS run outside short security/service locks; commit precedes publication/wipe. Precommit failure leaves identity/HTTPD/logins unchanged; postcommit lifecycle failure never rolls back identity and can leave served/stored fingerprints different. Failed stop skips start and retains canonical recovery. Public service/security projections are separate observations, not authorization. [HTTPS ownership, generation and recovery contract](../web_administration.md#https-and-reboot).
### Admin firmware upload
`web_firmware_update` streams a raw admin-only application image through HTTPD into the inactive OTA slot using standard ESP-IDF APIs. Existing Origin/CSRF/session checks precede body IO; raw length is bounded before HTTPD's narrowed length is trusted, SDK image validation and final principal currentness precede boot selection. One 4KiB buffer and a transient reboot task bound application storage; NVS and partition layout are untouched. HTTPD is occupied during upload, so normal web traffic can stall; deadlines bound receive progress, not flash-operation latency.
The server transition and identity reservations exclude competing lifecycle work. A separate atomic gate excludes ordinary reboot paths without depending on HTTPS initialization, preserving UART0 recovery. Successful response schedules delayed reset while retaining reservations; failed response after selection retains the selected-image latch but releases resources for manual reboot. A subsequent upload is refused until reset. No automatic retry, rollback, signing infrastructure or dependency patching. [Contract and validation limits](../roadmap.md#phase-10--simple-admin-web-firmware-upload).
### SSH
Typed SSH settings use the existing ID dispatcher and original-login result slot, never HTTPD wolfSSH calls or owner waits. Conditional lifecycle/session controls compare a saturated service generation and exact nonreused session ID under canonical locks. `ssh_transport_replace_identity` reserves service then identity before stop, retaining the command mutex across stop → commit → conditional restart. Failed stop skips mutation/start; failed persistence may follow disconnection; committed identity is never rolled back after restart failure. Only the SSH owner frees context after all slots retire, and start rejects orphan handles. Direct security/CLI/deferred SSH callers share task-bound identity reservations; crypto/NVS run outside security locks. HTTPS remains available, so no self-cutting HTTP ACK gate is needed. [SSH contracts](../web_administration.md#ssh).
+7 -1
View File
@@ -88,10 +88,16 @@ This is a semantic map, not a complete file inventory. Start here, then read the
### Browser admin backend
- Files: `src/web_admin_transport.{c,h}`, `src/web_admin_tickets.{c,h}`, protected registration/lifecycle in `web_server.c`, revocation through `web_serial_transport_revoke_*`, diagnostics in `web_console.c`.
- Routes: CSRF-protected admin-only `POST /api/admin/ws-ticket`; ordinary `GET /ws/admin` with cookie/Origin/ticket/shared-console admission before explicit 101. Admin UI entry is explicit; no admin broker client. One socket, two tickets, existing two shared console slots; six total HTTPD sockets, LRU disabled; current overall capacity is 39 URI handlers.
- Routes: CSRF-protected admin-only `POST /api/admin/ws-ticket`; ordinary `GET /ws/admin` with cookie/Origin/ticket/shared-console admission before explicit 101. Admin UI entry is explicit; no admin broker client. One socket, two tickets, existing two shared console slots; six total HTTPD sockets, LRU disabled; current overall capacity is 40 URI handlers.
- Currentness/policy: `admin_ssh_console_open_available()` shares two slots with runtime SSH; transport-qualified tokens and owner adapters revalidate outside console locks before commands/prompts. Parsed browser policy remains narrower than typed Settings; [shell contract](../web_administration.md#browser-shell-policy). Tests: `tests/admin_console_boundary/{run,accounts,lifecycle}.py`, `tests/admin_ssh_policy/run.py`, `tests/web_admin_transport/run.py --tickets`, `tests/web_cookie_auth/run.py --admin`.
- Ownership: 20 ms ESP timer queues at most one HTTPD poll, no new task; HTTPD owns 1,552 B PSRAM-only payload and IO. Closure uses HTTPD-owned `shutdown`, not IDF's reusable-pointer queued close. Detach fences submitters; only successful HTTPD stop retires queued state before restart. Session/principal currentness and generation checks protect all sensitive boundaries.
## Firmware upload
- `src/web_firmware_update.{c,h}`: admin raw `POST /api/firmware`, cookie/Origin/CSRF admission via `web_cookie_auth`, registration in `web_server`, UI in `web_ui`. Standard SDK OTA into inactive app only; bounded4KiB buffer, validated raw length/header/final image and principal before boot selection. No NVS/layout writes or vendor patches.
- HTTPD handles streaming synchronously; upload is deliberately disruptive to web traffic. Server/identity reservations fence lifecycle changes; atomic ordinary-reboot gate also covers console/SSH/browser/button reset paths. Success-response schedules delayed reset; response failure after selection latches uploads until deliberate manual reboot. No blind retries.
- Tests: `tests/web_firmware_update/run.py` (active-build SDK5.5.0 headers and actual begin/abort contract), `tests/web_ui_session/run.py`, existing auth/lifecycle tests. [Usage, acceptance limits and reusable regression checks](../roadmap.md#phase-10--simple-admin-web-firmware-upload). Phase 10 complete by explicit user acceptance on 2026-09-18: upload works and normal operation verified; no specific fault/NVS-comparison/power-loss/recovery pass implied. Initial install by wire; subsequent upload is application `firmware.bin`, not full-flash image.
## Typed settings source and regression map
HTTPD reads zero-wait projections and queues only IDs to the existing dispatcher. One original-login slot per domain; canonical owners compare/reserve at execution. [API/lifetime and failure contracts](../web_administration.md#typed-settings-api-and-operation-lifetime).
+11 -6
View File
@@ -2,17 +2,22 @@
Working memory, not an implementation timeline. Source is authoritative; begin with [code map](code-map.md), [architecture](architecture.md) and [decisions](design-decisions.md).
## Phase 10 plan — simplified by user, 2026-09-18
## Phase 10 COMPLETE — explicit user acceptance, 2026-09-18
- Plan only: admin HTTPS file picker/upload for a locally built ESP32-S3 application firmware.bin, existing auth/same-origin/CSRF, standard ESP-IDF OTA APIs and bounded streaming into inactive4MiBslot, SDK image/target/size validation before boot selection and controlled reboot.
- Preserve NVS and all other data partitions; only inactiveapp/otadata writes. No wholeflash/bootloader/partitiontable uploads or erase. Storage bytes preserved, but user-selected firmware must remain schema-compatible. Wired USB-to-UART recovery if a valid image is nonfunctional.
- Explicitly no signature/key infrastructure, antirollback/version rules, automatic rollback/healthconfirmation, remote downloadservice or dependency patches. This replaces the old broader OTA plan, not an implementation authorization. [Roadmap](../roadmap.md#phase-10--simple-admin-web-firmware-upload) is authoritative.
- User confirmed after firmware upload implementation and the concise-UI fix: “That works perfectly. And the usual operation is also verified.” Acceptance establishes that upload works and normal operation is verified. Do not infer specific fault-injection, NVS before/after comparisons, power-loss or recovery passes. The roadmap's compact regression guidance is reusable, not an acceptance blocker.
- Latest concise-UI change was copy-only; reported UI regression **169 groups PASS**. No rebuild after that text change. The integration build below is historical, not validation of a newly rebuilt UI or this documentation update.
- User authorized simple OTA implementation; initial Git clean. New web_firmware_update module with standard SDK APIs, raw POST /api/firmware, admin cookie/Origin/CSRF and final principal currentness. Settings → HTTPS / Reboot has File/XHR upload progress, confirmation, session fencing and no automatic retries. Only inactive app/otadata writes; NVS/layout untouched. No signatures/antirollback/automaticrollback, dependencies or generated assets changed. Consolidated procedure, contract and regression guidance: [Phase 10](../roadmap.md#phase-10--simple-admin-web-firmware-upload); standalone guide removed.
- 4KiB internal buffer + transient2048B-stack reboot owner allocated before erase; 10s stall/120s receive-loop budget, not totalflashdeadline. HTTPD synchronously blocks other web work during upload; networkserial maystall/drop, reboot disruptsall. No task/request/socket lifetime capture after handler. Service/identity reservation and atomic ordinary-reboot gate cover UI/UART0/SSH/browser/localbutton paths. Failed response after bootselect schedules noautomaticreset; selected latch rejects further uploads409, manual reboot available. Successful response schedules500ms reboot retaining reservations.
- Review fixed two actualSDK5.5.0 edge cases: failed esp_ota_begin maypublishlivehandle beforeeraseerror (abortthat handle); rawContentLength64 canwrapHTTPDsize_t32 (overflow-safe actualslotbound/strictdecimal/equality check beforebody/erase). End consumes handle evenerror. SDKvalidation followed by exactparsedimage length includingSHA; basic header requiresS3appdescriptor/hash. Unrelated old/privateSDK code unpatched.
- Parent final pio PASS **94,220 B RAM / 1,847,645 B flash**, +24RAM/+19,080flash vsPhase9, not runtimeheadroom. Parent newbackend88cases+actualSDKbeginfailurecontract, UI169groups+CSP, serverlifecycle44, admin25, consolelifecycle, SSHruntime, cookielifecyclePASS. Additionalbase/admin/display/lifecyclecookie, idle18, SSHmanagement/runtime/security agentPASS after adding missing rebootfake to adminfixture (no productionchange). Independent review final noactionablefindings; realbuilt firmware parsed with SDKmetadata bothOTAoffsets (notdeviceflashproof).
- **Next:** Phase 11 BLE remains planned, not authorized by this acceptance update. Preserve small scope and existing uncommitted implementation/UI work; do not resurrect Phase 9 patches. This handoff changed documentation only; no build, test, upload, erase, device operation or commit was performed.
## Accepted state — 2026-09-18
- **Reduced Phase 9 complete by explicit user sign-off.** User waived a new whole-phase device check based on prior Phase 8 validation. Application code was unchanged, but no-core-dump/silent-panic defaults changed. Do not record the waived check as executed or claim a new panic/hardware pass.
- Small scope: standard `sdkconfig.defaults` options, source-reviewed [operational checklist](../security_operations.md), README/roadmap guidance. No dependency patches, crypto policy replacement, allocator hooks, SDK migration, encryption, eFuse or partition changes. User abandoned the extensive earlier Phase 9 and restored baseline `f40c09c`; do not resurrect it.
- Last actual build: normal `pio run` PASS on PlatformIO6.12.0 / IDF5.5.0 / original20241119 toolchains, **94,196 B linked RAM / 1,828,565 B flash**. Generated configuration confirmed no dumps/silent panic; compilation inputs had no abandoned overlays/crypto guard. Existing generated config already selected these options. Defaults do not override saved sdkconfig; README explains verification. No new build is implied by this documentation consolidation.
- Historical Phase 9 build: normal `pio run` PASS on PlatformIO6.12.0 / IDF5.5.0 / original20241119 toolchains, **94,196 B linked RAM / 1,828,565 B flash**. Generated configuration confirmed no dumps/silent panic; compilation inputs had no abandoned overlays/crypto guard. Existing generated config already selected these options. Defaults do not override saved sdkconfig; README explains verification. No new build is implied by this documentation consolidation.
- **Phase 8 complete:** 8AC target validated; explicit 8D.22 user acceptance on 2026-09-13. Completion/telemetry and legacy compatibility now live in [roadmap acceptance](../roadmap.md#phase8-acceptance-evidence) and [storage compatibility](../roadmap.md#phase8-legacy-credential-compatibility). Separate acceptance/legacy history documents removed; active [web contracts](../web_administration.md) and [regression procedures](../user_administration_tests.md) retained.
- Post-acceptance baseline includes PSRAM-only ping payload/user snapshot allocations and refined web quick panels. Prior focused host/geometry/build checks passed; those are not additional hardware claims. Preserve lazy-allocation failure isolation and unchanged serial hot path.
@@ -21,7 +26,7 @@ Working memory, not an implementation timeline. Source is authoritative; begin w
- Previously accepted combined binary WS send: CPU160MHz / 230400 baud full mix including browser admin. Latest recorded telemetry has very low internal/DMA lifetime minima (2,052/460 B); these are nonblocking headroom follow-ups, not approved reserves or proof of simultaneous allocation failure. Full table, capture workload and counter limits are preserved in the roadmap.
- TLS `-0x004C` means generic NET_RECV_FAILED, not OOM. Historical authentication/admission symptoms do not establish a cause. Do not invent fault, soak, timing or power-loss passes.
- Credentials remain unencrypted; old flash contents are not erased. Intermittent trusted-network operation reduces exposure, not physical-extraction risk. Upstream upgrades are separate deliberate tasks, not an endless local backport programme.
- Phase 10 is planned, not automatically authorized by acceptance. No device operations, branch/reset, commits or dependency upgrades are part of this documentation task.
- Phase 10 is complete by the explicit acceptance above; detailed unreported regression scenarios remain unevidenced, not completion blockers. Device operations, branch/reset, commits and dependency upgrades remain outside this documentation task.
## Contracts to preserve