Files
ESP32_Serial_Swiss_Army_Knife/docs/user_administration_tests.md
T

33 KiB
Raw Blame History

User administration and authentication tests

This document retains phase-specific regression procedures. The Phase 8A and 8B sections describe the behavior of those historical implementation baselines; they are not the current end-to-end acceptance behavior. In current Phase 8C firmware, HTTPS and SSH authenticate through the user database, role-user SSH sessions receive the broker-backed serial stream, and role-admin SSH sessions receive the administration shell. Use the Phase 8C section for current routing and shared-console validation. Never include generated or entered passwords in test logs.

Current Broker management regression procedure — 8D.16

Implemented and host/build verified; independent parent review, target validation and user sign-off pending. The 8D.16 contract and target checklist defines exact API, generation/wrap safety, resource bounds and evidence limits. This procedure is not an execution claim.

  • Verify admin-only detailed client rows against non-consuming UART0 snapshots/counters; direct normal-user management GET/POST denied. Observe zero/one/full client mix without reading UART payloads as a probe.
  • Opening, selecting, cancelling, refreshing and navigating must leave writer and both terminals unchanged. Refresh clears selection. Native confirmation must identify exact target and snapshot writer; no8D.17/18 popover behavior is included.
  • Assign among USB/SSH/two browser serial clients with both admins connected. Disconnect/reuse target while confirmation is open; race writer release/reacquire, competing requests, shell force and local release. Stale confirmation must fail without changing the intervening lease; Refresh/reselect/confirm explicitly.
  • Test separate-login result isolation, same-login tab replacement, pending capacity, lost ACK/result, bounded polling/manual recovery, logout/expiry/revocation and HTTPS stop/restart. Never infer cancellation from connection loss or replay automatically.
  • At signed-off160MHz/230400 baud, check full-mix binary traffic/drop isolation, UART0/USB recovery and optional-route failure isolation. Capture exact revision/mix and internal/DMA/PSRAM plus HTTPD/dispatcher margins; host tests/build do not approve reserves or target throughput for this phase.

Current Network settings regression procedure — 8D.12/8D.13

Both phases were authorized and implemented together. The implementation record defines the complete API, limits, result states, exclusions and pending target checklist. Backend/cookie Network PASS, UI agent97+renderer/CSP/review PASS and lifecycle agent21 PASS are reported evidence; final parent build/tests and target sign-off remain pending. Do not treat this procedure as executed or infer M3 completion/8D.14 authorization.

  • Verify admin-only Settings/Network and direct-route normal-user denial, current cookie/principal, body/query/framing/Origin/CSRF checks, unavailable/contended snapshots and generation races against CLI/local controls.
  • Round-trip UTF-8 and arbitrary SSID bytes through text/hex, including NUL/BOM/non-UTF-8 and 32-byte boundaries. No saved PSK or length may appear in responses, status/logs/completion/local display. Keep omits credentials; Replace never accepts blank; disabled-STA Clear (including disable+clear) works; enabled-STA/AP clear is denied even when AP policy is off. Check transient-input expiry and context/session clearing.
  • Distinguish RAM Apply, explicit Save, stored-only Wi-Fi Load and reboot persistence. Missing/invalid/failing Wi-Fi storage must not install generated defaults or new AP secrets. Exercise stale generations, queue failure/drop accounting and NVS failures without secret logging. No Wi-Fi reset/default/export action exists.
  • Prepare UART0 and USB before confirming disruptive actions. Test Start/Stop (including RAM boot policy), Reconnect/Next, AP policies and stopped no-ops. The selected profile is an edit target, not explicit connection selection; Next uses canonical priority/wrap. Cancel confirmations and exercise lost ACK/401/disconnect, manual Check Result/Refresh, another-tab result replacement and no automatic replay. accepted is not online; delivery before disconnection is not guaranteed. Reconnect via STA/AP; UART0 administers recovery, USB preserves independent UART1 access.
  • Test mDNS generation/Set/Save/Load/Defaults, live/offline reannouncement, next STA IP, init/live failure isolation and applied_not_queued. Verify actual client DNS and changed-hostname browser trust/login, not merely announced. Confirm no unintentional Wi-Fi secret reset.
  • With USB/two web serial/SSH serial and both admin routes, verify hidden output draining and writer/observer preservation through Settings navigation. Separate actual network-disruption losses from serial/broker regressions. Exercise optional Network route/timer failure and stop/restart without taking down unrelated routes. Browser-shell restrictions remain unchanged.
  • Capture boot/full-mix internal/DMA/PSRAM free/minimum/largest blocks and memory floors during TLS/Network operations; timer heap/slot costs, repeated-operation cleanup/soak and HTTPD/dispatcher stack margins remain required. Record exact revision/client mix and nonsecret counters, including broker and manager queue drops. Host tests do not establish target reserve or hard scheduling/cancellation guarantees.

Current legacy-removal regression procedure

The legacy bootstrap/credential/reconciliation instructions in the phase baselines below are historical only, superseded by legacy credential removal and the current command reference. Basic authentication is also historical; current HTTPS uses cookie login. The current overrides apply to later Phase 8C/browser procedures too: no user bootstrap or web credentials command remains, first-admin creation uses normal UART0 user add, and recovery rebuilds empty. Never treat the checklist below as evidence of execution.

  1. On a disposable controlled NVS image with only user_db/database missing, boot and confirm an empty database is persisted with zero accounts/admins and no imported credential. Reboot and confirm it remains empty. Keep physical UART0 attached; do not factory-erase the device for this test.
  2. Run user add maint admin on UART0, check hidden password confirmation and cancellation, then confirm account/password persistence after reboot. Separately exercise user add operator user --generate and secure one-time display. Final-administrator delete/demotion must still fail. Existing SSH own-password generation restrictions and typed browser generated-password support remain unchanged.
  3. Load a valid existing v1 user image and verify accounts, roles, IDs, auth generations, verifiers and keys are unchanged, including a formerly migrated role-user account. No bootstrap status should appear and no account should be silently promoted. No web material change may synchronize a verifier.
  4. On a disposable malformed user image, confirm authentication fails closed without automatically overwriting storage. On UART0 run user recover --force, confirm empty storage, then user add maint admin. Recovery must refuse healthy databases, including healthy empty storage, and be unavailable through SSH/browser. Verify serial/Wi-Fi configuration and TLS/SSH identities remain intact. Inject read/write/commit failures where available and check failure isolation and complete committed records, not partial live mutations.
  5. Upgrade valid 1,392-byte v1 web_sec/material; verify persisted 1,340-byte TLS-only v2 and exact certificate/key DER, fingerprint and generation retention in a controlled fixture without logging private data. Reboot and confirm identity continuity. Malformed/unknown/cryptographically invalid records and migration read/write/commit failures must fail closed without fallback replacement. Do not interpret a failed commit as proof that no flash write occurred.
  6. Confirm help/completion/status expose no legacy credential/bootstrap operation or secret. Removed commands must reject without mutation. Exercise retained user generation and web certificate rotate --force through supported frontends. web reset --force must change TLS only, require new certificate trust/login after HTTPS restart, leave user credentials/generations unchanged, and not revoke unrelated SSH sessions.
  7. Recheck UART0/native USB availability and broker one-writer/isolated-observer behavior with network authentication unavailable. Record only nonsecret counters and telemetry. Older v1-only firmware cannot read HTTPS v2; logical NVS replacement is not secure flash erasure and no factory erase is required.

Evidence: implementing agents report security 15 groups with real installed mbedTLS and account tests PASS; final integration/build and this target checklist are pending. The user's report that idle cleanup worked is separate, with no additional soak or broad sign-off inferred.

Historical Phase 8A baseline — role-based database and UART0 administration

Keep UART0 physically attached throughout these tests. At the Phase 8A baseline, HTTPS and SSH still used the legacy shared credential while the new database and public keys were prepared for the Phase 8B cutover.

1. First-boot migration and persistence

Upgrade a device that already has working HTTPS/SSH credentials and no user_db blob. After boot:

  1. Run user status and user show admin.
  2. Confirm exactly one admin-named account exists with role user, bootstrapped=no, no displayed password/salt/verifier, and no public keys.
  3. Confirm the pre-upgrade credential still authenticates to HTTPS and SSH and that serial terminal behavior is unchanged.
  4. Reboot and confirm user status reports the same account ID and authentication generation.
  5. Confirm routine boot logs, web status, and ssh status contain no user password, verifier, salt, or raw public-key blob.

A transient failure reading the legacy web credential must not create or commit an empty user database. If a valid user_db blob already exists, it must still load without the legacy credential being readable.

2. Administrator bootstrap

Run user bootstrap, verify that typed password characters are not echoed, enter a 1264 character printable password twice, and confirm user show admin now reports role admin and a higher authentication generation. Reboot and confirm bootstrapped=yes and one administrator persist. On a separate resettable test image, exercise user bootstrap --generate; record the generated password securely for the test, verify it is displayed once, then clear the terminal scrollback after use.

During Phase 8A, verify the newly bootstrapped password does not replace the legacy HTTPS/SSH login yet and the existing network credential continues to work. This separation is intentional until Phase 8B. A second user bootstrap must be rejected without changing the account.

3. Account CRUD, capacity, and invariants

Use generated test passwords where practical:

user add operator user --generate
user add maint admin --generate
user list
user show operator
user role operator admin --force
user role operator user --force
user password operator --generate

Confirm unique valid usernames are accepted; uppercase, empty, oversized, or punctuation-invalid usernames are rejected. Confirm duplicate users are rejected. Fill all eight slots and verify a ninth account is rejected without altering existing records; delete a non-final account and confirm its slot can be reused with a new account ID.

With two administrators, demote or delete one and confirm the operation persists. Then attempt both deletion and demotion of the final administrator and confirm both are rejected. Before bootstrap on a separate migration run, verify user delete admin --force is rejected so legacy synchronization remains recoverable. Ctrl-C during either hidden password prompt must cancel without changing the account or leaving stale input in the next console command.

4. SSH public-key import and fingerprints

Generate disposable Ed25519 and ECDSA P-256 OpenSSH keys on a trusted host. For each supported type, run user key add <username>, paste the single .pub line, and confirm user show <username> reports the type, stable slot index, and expected SHA256: fingerprint without displaying the Base64 blob or comment.

Verify all of the following fail without changing the account generation or key list:

  • Unsupported RSA, certificate, or unknown key types.
  • Truncated/invalid Base64, mismatched outer and embedded types, trailing blob data, malformed Ed25519 lengths, and an invalid/off-curve ECDSA point.
  • Adding the same key again to the same account.
  • Adding a fourth key to an account that already has three.
  • Deleting an empty/out-of-range slot.

Assign the same public key to a second account and confirm it is accepted, receives an independent slot, and authenticates as the username selected by the SSH client.

Then exercise user key delete <username> <index> --force and user key clear <username> --force, reboot, and confirm the exact remaining fingerprints persist. Phase 8A stores these keys but does not yet accept SSH key login.

5. Legacy credential rotation boundary and reboot reconciliation

Before administrator bootstrap on a resettable test image:

  1. Run web credentials rotate --force.
  2. Confirm the console reports that the pre-bootstrap migrated verifier was synchronized.
  3. Verify the new legacy credential works for HTTPS and SSH, reboot, and confirm user show admin retains the synchronized authentication generation.
  4. Where NVS fault injection or controlled power interruption is available, interrupt after the web credential commits but before the user blob commits. On the next boot, confirm reconciliation updates the migrated verifier rather than committing an empty database or losing network recovery.

After bootstrap, rotate the legacy credential again. Confirm the console explicitly says it remains separate from user passwords, existing network sessions are revoked, the new legacy credential works, and the bootstrapped administrator generation/password are not silently replaced. Apply the same checks to web reset --force, additionally confirming HTTPS certificate recovery/restart behavior remains intact.

6. Explicit database recovery and storage failures

Using a disposable device or test NVS image, inject a wrong-size/version or malformed user_db blob and reboot. Confirm normal database initialization fails without erasing shared NVS, but the user command is still registered. user status must report the database unavailable. Run:

user recover --force
user status
user bootstrap --generate

Confirm recovery explicitly replaces only user_db, imports the current legacy credential as role user, and leaves serial, Wi-Fi, HTTPS identity, SSH host key, and legacy network login intact. Confirm recovery is rejected while a valid database is initialized. Simulate NVS write/commit exhaustion or failure during create, password, role, key, and recovery operations; after reboot, each operation must expose either the complete old blob or complete new blob, never a partially live mutation.

Because NVS is not yet encrypted, this test validates logical verifier storage rather than resistance to physical extraction. PBKDF2 verifiers remain susceptible to offline guessing, and append-oriented NVS may retain historical plaintext legacy credentials until a full partition/flash erase.

7. Phase 8A regression

With the maximum supported mix of USB CDC, WebSocket, and SSH serial clients active, repeat user list/show and several UART0 mutations. Confirm password prompts remain physical-UART-only, no user operation acquires a broker client or writer lease, existing Phase 8A network authentication and revocation behavior remains functional, and serial/broker counters show no unexplained loss. Record memory, web counters, and ssh counters before and after the run.

Historical Phase 8B baseline — role-aware HTTPS and SSH authentication

These procedures preserve the Phase 8B regression baseline. Complete the Phase 8A bootstrap first and retain two disposable accounts, one user and one admin. At that baseline both roles received the serial/status interface; current Phase 8C firmware instead routes SSH administrators to the administration shell.

1. HTTPS passwords and fail-closed behavior

For both roles, authenticate to /, /api/status, and the browser terminal with the account password. Confirm the full 16-character username and a 64-character password work, while absent, malformed, oversized, wrong-user, and wrong-password Basic headers fail without exposing which component was wrong. After bootstrap, confirm the value from web credentials show no longer authenticates unless it independently equals a current user password.

On a disposable malformed-database image, confirm HTTPS returns authentication-service unavailable and never falls back to the legacy credential. Recover through physical UART0 only. Verify web status reports user-database authentication and active WebSocket sessions show account, user role, and password method without secrets.

2. SSH password and public-key login

For both user and admin, verify password SSH login reaches the same broker-backed serial stream as before. Confirm ssh sessions shows the account, user role, password method, and writer/observer role. An administrator must not receive UART0 commands yet.

Import one Ed25519 key and one ECDSA P-256 key through user key add, then verify each matching private key authenticates without a password and ssh sessions reports public-key. Confirm an unregistered key, wrong username, unsupported RSA key, certificate key, and three failed completed attempts are rejected. Normal client probe-then-sign behavior must consume one completed attempt, not two; unsigned probes alone must never create an authenticated session or broker client.

3. WebSocket ticket binding

Authenticate separately as two users and request tickets. Confirm each ticket is accepted exactly once, expires after 30 seconds, and cannot be reused or altered. Change one account after ticket issuance but before WebSocket connection; its ticket must fail closed without creating a broker client, while the other account's ticket still works. Repeat with delete followed by recreation of the same username and confirm the old ticket remains invalid.

4. Targeted session revocation

Connect simultaneous WebSocket and SSH sessions for two different users and place one affected session in writer state. One mutation at a time, exercise password change, role change, SSH-key add/delete/clear, and account deletion. Required results:

  1. Outstanding tickets and active sessions for the changed username are revoked promptly and any writer lease is released.
  2. The stale session cannot inject additional serial bytes or reacquire writer ownership.
  3. The unrelated user's WebSocket/SSH sessions remain connected and retain their expected broker roles.
  4. New authentication succeeds only with the updated password/key/role state.
  5. Deleting and recreating the same username does not revive old tickets or sessions.

Run web status, ssh sessions, web counters, ssh counters, and broker clients around each mutation. Also rotate web credentials rotate --force after administrator bootstrap and confirm it changes only recovery material: role-based HTTPS/SSH sessions remain connected and user passwords are unchanged. web reset --force restarts HTTPS because the TLS identity changes, but must not revoke unrelated SSH sessions.

5. Concurrency regression

With USB CDC, two role-based network users, one WebSocket terminal, one SSH terminal, and UART1 traffic active, alternate writer ownership and mutate one account. Confirm binary transparency, observer isolation, bounded authentication/handshake behavior, UART0 responsiveness, and no unexpected disconnect of the unaffected user. Record memory, broker, web, SSH, and serial counters before and after. Repeat after reboot to verify passwords, roles, keys, and authentication methods persist.

Phase 8C SSH administrative shell

Validation status: passed on target hardware. The full matrix below passed, including concurrent USB/WebSocket/user-SSH/admin-SSH operation. At 460800 baud with SSH and WebSocket clients in parallel, substantial packet drops and slower display controls were observed under load, but memory exhaustion could not be provoked and no lower baud-rate cap is planned.

Use one disposable admin and one disposable user. Keep UART0 attached throughout. The SSH server still accepts only shell sessions: exec, subsystem/SFTP/SCP, forwarding, and unauthenticated connections must remain rejected.

1. Route separation and normal shells

  1. Connect as the normal user and confirm the existing broker-backed UART1 serial stream, broker client, and writer/observer behavior are unchanged.
  2. Connect as the administrator and confirm the admin@serial-tool> prompt appears. Run help, memory, serial status, wifi status, web status, broker status, and broker clients. Compare representative output with UART0 and confirm both routes execute the same registered command implementations.
  3. From UART0 run ssh sessions and broker clients. The user session must show route=broker; the admin session must show route=admin-console, broker=0, broker-role=n/a, and no writer lease. The admin session must not start UART1 or alter broker client/writer counts merely by connecting.

2. Bounded command processing

Exercise printable input, backspace, Ctrl-C, CR/LF, an empty line, and a line longer than the documented limit. Confirm the command line is bounded, overflow is discarded through a clear diagnostic, and a new prompt remains usable. Run help, user list, and broker clients in a normal ANSI terminal and confirm every line starts in column zero: canonical LF output must be normalized to CRLF without doubling handlers that already emit CRLF.

Run at least five distinct commands, then use Up/Down to navigate the four-entry per-session history, return to a saved draft with Down, and confirm older entries are bounded out. Verify history does not survive reconnect and is not shared with a second administrator. Exercise Tab on root and nested prefixes such as us, user l, wifi ap sh, wifi next, and ssh host-key i; confirm the same hints are offered by UART0 and SSH. For an ambiguous prefix, confirm both routes print the same matching candidates and redraw the unchanged input line instead of cycling candidates; unique/common prefixes must redraw cleanly without inserting escape-sequence bytes into the command. Type a command, use Left/Right to insert and delete characters in its middle, then use Home/End and the terminal's Pos1/Ende keys; verify cursor placement, Backspace, and Delete change exactly the intended byte before submission.

Run an unsupported command and confirm it is rejected without affecting UART0 or the serial broker. Run the full root help output to exercise output-ring draining. With the SSH client temporarily unable to read output, confirm the worker applies input backpressure rather than accepting an unbounded command/output backlog; inspect ssh counters for admin-console admission and input-backpressure values.

3. Remote account administration

Run user list, user show <name>, user add <name> user --generate, user password <name> --generate, user role <name> admin --force, and the key delete/clear operations from the administrative shell. Confirm generated passwords appear once only on that authenticated channel, affected account sessions are revoked, and unrelated sessions remain connected.

Import both supported key types through the remote form:

user key add <username> ssh-ed25519 <base64-blob>
user key add <username> ecdsa-sha2-nistp256 <base64-blob>

Confirm the full ECDSA P-256 command is accepted, fingerprints appear in user show, a duplicate on the same account is rejected, and the same key can be imported for a second account. Verify subsequent private-key SSH login uses the selected SSH username.

4. Interactive administration, lifecycle actions, and revocation

Confirm user bootstrap and user recover --force remain unavailable from SSH and continue to work through physical UART0. Also confirm that an administrator cannot run user password <own-username> --generate over its own SSH session; entered replacement passwords and generated passwords for other accounts remain available. From admin SSH, exercise manually entered user passwords and public keys, Wi-Fi station/AP secret entry, AP secret display, legacy web recovery credential display/rotation, HTTPS certificate rotation/reset, and both ping and wifi ping. Hidden characters must not echo or enter command history; visible key input must support Backspace and Ctrl-C; ping lines must remain ordered and correctly attributed to the invoking SSH session.

Exercise ssh disconnect for another session and the current session. For the other-session case, confirm the acknowledgement drains, the target closes, and the source shell returns. Separately test reboot, ssh stop, ssh host-key rotate --force, and ssh reset --force: each must acknowledge scheduling, stop accepting another command, drain output, and then close/reboot as appropriate. Reconnect after key replacement and verify the new fingerprint. Simulate an unread SSH output window and confirm the destructive action cancels after its bounded drain timeout rather than remaining pending forever.

While an administrative command is queued or running, use UART0 to change that admin's role/password/key or delete it. Confirm no second remote administrative command runs after the mutation, the SSH session is revoked promptly, and queued output is not delivered to a reused SSH slot. Repeat with a different account mutation and confirm the administrator remains connected.

Finally, issue commands concurrently from UART0 and admin SSH, including user list, long help output, and one UART0 interactive password or key prompt while an SSH command waits. Confirm the single dispatcher serializes all esp_console_run() calls, UART0 retains its line editing/history/completion, prompt input is consumed only from UART0, outputs are not mixed between transports, and there is no stack overflow, corrupted argument parsing, database damage, or broker disruption.

Planned Phase 8D integrated web administration

These are acceptance requirements; listing an item does not claim its execution. The 8D.3/M1 checkpoint is host-tested/build-verified and validated by explicit user sign-off on 2026-09-06 after both-role login, mixed-client operation and post-soak evidence. Unrecorded individual checks remain coverage/evidence limitations, not blockers to that sign-off. Browser administration remains planned. Execute them incrementally using the Phase 8D work-unit plan, not only at the end of the phase.

Validation checkpoints:

  • 8D.0: Establish working browser login/serial, UART0/USB/SSH, and measured build/runtime memory baselines before changing authentication. Record the revision/configuration, workload, numeric resource budgets, and repetition counts.
  • 8D.18D.3 / M1: Exercise session primitives and session-specific cleanup as they land, then run section 1's authentication checks against the real login page and existing serial UI. Administration-denial checks apply as each admin route is added. Test new and previously Basic-authenticated browser profiles and direct-IP/mDNS access where available. Stop for target/browser validation before adding the admin shell.
  • 8D.48D.7 / M2: Regress UART0/admin SSH after the console-boundary change; then run section 2 and the applicable section 5 concurrency/failure checks with both browser WebSockets active. Stop for target/browser and memory validation before settings.
  • 8D.88D.21: Run sections 34 one settings/control domain at a time, including direct server-side role/CSRF/origin denial, malformed/oversized input, persistence, and applicable lifecycle/revocation checks. Reuse previously verified endpoints for popovers rather than postponing API validation until popovers exist.
  • 8D.22: Run all sections together plus the bounded concurrent soak. This is cumulative acceptance, not a substitute for the earlier checkpoints.

Every runtime-changing chunk requires a bounded pio run, the small transport/login smoke check, and its focused acceptance checks. Record implemented, build-verified, and target-verified separately. Measure UART0 memory internal/PSRAM free, minimum-free, and largest-block values at settled boot, connected load, and after repeated logout/disconnect/reconnect; include stack margins where available and compare incremental and cumulative resource costs. Record pending hardware checks or timed-out commands honestly; neither is a pass. The incremental plan defines the stop/split policy if a resource budget or validation gate fails.

1. Integrated login and authorization

Authenticate as both roles through the same-origin login page, explicitly log out, and switch accounts without clearing browser state or relying on an HTTP Basic credential cache. Verify bounded session capacity, expiry, login throttling, malformed cookies, CSRF rejection, Origin rejection, password/role/key invalidation, deletion/recreation, and unrelated-account isolation. A normal user must be unable to discover through navigation or invoke directly any admin ticket, admin WebSocket, typed mutation API, quick setting, detailed client list, or writer-transfer operation.

2. Serial/admin terminal switching and lease preservation

As an administrator, connect the browser serial terminal, acquire the writer lease, send and observe serial data, then switch repeatedly between Serial terminal and Admin shell. The visible terminal contents and input route must change, but broker clients, the displayed browser client ID, and active writer ID must remain unchanged. Request control/Release control and writer/observer state must stay visible in both modes. While Admin shell is selected, have a normal user request the writer lease and confirm the retained browser lease prevents unintended takeover. Only explicit Release control, confirmed writer transfer, Disconnect, logout, revocation, expiry, or connection failure may release it.

Verify the browser admin shell executes the canonical registry through the single dispatcher, preserves bounded history/completion/prompts and backpressure, and does not itself become a second broker client. Physical-only bootstrap/recovery commands remain rejected. Closing only the admin-console route must leave the browser serial client and its lease intact.

3. Quick settings and client popovers

For an administrator, open the Serial and Wi-Fi card popovers by hover, keyboard focus, and click/tap. Confirm each path exposes the same controls, opening alone performs no mutation, Escape/outside click closes cleanly, current values refresh safely, common edits have clear apply-versus-save behavior, and links reach the full Settings section. Saved Wi-Fi secrets must not appear by default or enter routine DOM text, logs, URLs, or status responses.

Open Broker clients and confirm its secret-free list matches authoritative broker snapshots: client ID, transport/name, writer/observer role, and bounded queue/drop state. Open Active writer, choose another currently connected eligible client, confirm the transfer explicitly, and verify exactly one writer remains. Disconnect or recycle the target before confirmation and confirm its stale generation/ID is rejected without releasing or reassigning the current writer. Hover/focus alone must never transfer ownership.

4. Typed settings and destructive operations

Exercise user/password/role/key management, serial settings and persistence, Wi-Fi profiles/AP policy/secrets, service/session controls, display settings, and network diagnostics through typed bounded APIs. Compare resulting subsystem state with the equivalent canonical CLI behavior without routing API requests through command strings. Generated passwords appear once in no-store responses; destructive or self-terminating HTTPS/SSH/reboot/security actions require explicit confirmation and explain the expected connection loss.

5. Concurrency and failure isolation

Run USB, browser serial, browser admin shell, user SSH, admin SSH, UART0, and active UART1 traffic concurrently. Alternate explicit writer transfers while issuing administrative commands and opening/closing popovers. Verify one writer, isolated observers, bounded memory/queues, principal revocation, no mixed admin output, no hidden lease loss during terminal switching, and continued UART0/native-USB recovery if web-session or admin-console initialization fails.