ESP32_Serial_Swiss_Army_Knife/docs/security_operations.md

Security operations — Phase 9D

Operator runbook for the current firmware; procedures below are not execution records. Use the command reference for complete syntax and the browser contracts for frontend permissions and result handling. No device operation is authorized merely by this document. Schedule disruptive work, retain trusted UART0 access, and obtain explicit approval for destructive recovery/erase. The user will validate whole Phase 9, not provide per-segment sign-off. 9D is in progress; Phase 9 is not complete or production-ready. The partial wolfSSL/wolfSSH advisory review leaves current-path security findings open; the bounded license inventory leaves distribution/corresponding-source/notice questions unresolved. This runbook neither resolves those findings nor claims device validation or license clearance.

Safety and access

• Restrict physical access: UART0 is the trusted administrative recovery authority. Use the USB-to-UART bridge at 115200 baud, not native USB CDC.
• Native USB is network-independent, binary-transparent UART1 access through the one-writer/multiple-observer broker; it is not an admin console. Reboot interrupts every transport.
• Use a trusted workstation and restricted provisioning network. Disable terminal recording, screen sharing and clipboard history before secret entry/reveal; do not put secrets in command arguments, tickets, screenshots or routine logs.
• UART0 and authenticated role-admin SSH provide the operational commands below, except user recover --force is UART0-only. Role-user SSH is serial, not administration.
• Browser shell is narrower: other-account interactive add/password and forced role/delete only; no self/generated/key/recovery account commands. Wi-Fi/mDNS shell access is status-only. Typed Settings has separate account/network/identity workflows; do not infer shell parity.
• NVS is unencrypted: Wi-Fi PSKs and HTTPS/SSH private keys are plaintext at rest; password verifiers are also sensitive. Secure boot and NVS encryption are not provided by this phase. Physical flash extraction and firmware replacement remain outside the protection boundary.

First provisioning — UART0 first

1. Start in a controlled RF environment with UART0 attached. Defaults enable Wi-Fi with a random-password fallback AP and no station profiles; networking can start before the console is ready. A new AP name is not an authentication secret.
2. Once the console is ready, use wifi stop, inspect wifi status, then wifi save if Wi-Fi must remain disabled after reboot. Stop is asynchronous: check actual state, not only the queue acknowledgement. Do not rely on this to prevent the initial boot's radio exposure.
3. Inspect user status. Missing user storage initializes empty; create the first administrator with user add <username> admin on UART0 and the hidden confirmation prompts. There is no bootstrap/shared default web credential. A healthy empty database needs add, not recovery.
4. Store the chosen password in the operator's protected vault. Usernames and password bounds are in the command reference (passwords: 12–64 printable ASCII characters). Optionally use --generate only with private, reliable one-time capture. Create/test a separately controlled recovery administrator if policy requires one; the final administrator cannot be deleted or demoted.
5. Choose AP policy explicitly with wifi ap policy off or wifi ap policy fallback (use always only intentionally). For retained AP access, set wifi ap secret through the hidden 8–63-character prompt, or privately retrieve the generated password with wifi ap show-secret. Never include its output in routine captures.
6. Configure an unused/disabled station slot using wifi profile set <slot> <priority> <mixed|wpa3> <ssid>, then wifi profile secret <slot>, then wifi profile enable <slot>. Check peer compatibility before choosing WPA3. Profile set preserves the existing secret/enabled state; disable an existing slot before staging its replacement. Enabling or editing an enabled profile/AP can reconnect the radio.
7. Use wifi start when ready, inspect wifi status / wifi profiles, and wifi save to persist the intended working policy, profiles, secrets and boot-enabled state. Restrict upstream network access to intended operators; do not expose ports 443/22 to untrusted networks.
8. Read web certificate info and ssh host-key info through trusted UART0. Compare the certificate SHA-256 fingerprint in the HTTPS client and the OpenSSH SHA256: host-key fingerprint before supplying account credentials or accepting changed trust. A self-signed certificate or an unsolicited network fingerprint alone is not proof of device identity.
9. Establish narrowly scoped browser certificate trust and SSH known-host trust only after comparison. Investigate an unexpected mismatch rather than disabling verification or deleting all known-host entries. Test fresh HTTPS login and both intended SSH roles; an admin SSH login does not acquire a serial writer lease.
10. Set required serial framing, mDNS suffix and display policy with their documented commands and explicit saves. Record intended state privately; later rehearse a reboot and compare actual loaded settings/trust before production use.

Fallback-default caution: true first boot attempts to persist generated Wi-Fi defaults, but that save can fail. Incompatible stored configuration selects generated RAM defaults without overwriting the rejected blob. wifi defaults generates fresh RAM defaults; wifi reset generates and saves them. Both restore enabled/fallback policy and replace the AP password, not merely repair connectivity. Re-protect the AP and verify/save intended state rather than assuming an old password or disabled radio survived.

Working configuration versus service lifecycle

Operation/domain	Persistence and operational effect
Serial, Wi-Fi, mDNS, display edits	Working RAM until serial save, wifi save, mdns save or display save, respectively; load reads that domain's storage, not an external backup. defaults is RAM; reset applies/persists domain defaults. Check errors and actual state; these domains do not share one atomic transaction.
wifi start / wifi stop	Queue runtime action and set RAM enabled-at-boot true/false; only wifi save persists it. Local start/stop has the same semantics.
web start / web stop, ssh start / ssh stop	Runtime service controls, not persistent enable/disable configuration. No corresponding service save or durable boot-disable command. Stops disconnect that service's clients.
Boot network services	main.c attempts HTTPS/SSH startup independently when Wi-Fi initialization/start has no error and each service's security/runtime initialization succeeds. Wi-Fi configured off can still initialize successfully: listeners may be started without radio reachability. Wi-Fi-off is not a persistent HTTPS/SSH-disable flag.
Accounts and identities	Mutations commit their own NVS blob as part of the operation; no follow-up save. Commit acknowledgement is not a general power-loss/durability guarantee.
reboot	Loses unsaved RAM and all live sessions. A runtime service stop does not survive normal boot startup.

Routine account, password and authorized-key rotation

1. Keep UART0 or a separate current administrator available. Inspect user show <username> privately to confirm role and current key slots; do not operate from stale indices. The database allows eight accounts and three authorized keys per account.
2. Set a known replacement with user password <username> using hidden prompts. user password <username> --generate commits then displays a value once: protect it immediately. Lost generated output has no retrieval command; inspect account state, then deliberately set another known password via UART0/another admin rather than blindly replaying generation.
3. Admin SSH rejects its own generated-password replacement; its ordinary hidden self-password change is allowed but revokes the invoking principal. Browser shell denies all self account mutations. Typed Accounts supports self changes; its Generate action only returns a value, does not change the account, and has no retained retrieval. Securely save/acknowledge it before separately submitting the mutation.
4. For an authorized-key rotation, generate/protect the private key on the operator's host; import only the supported Ed25519 or ECDSA-P256 public key with user key add <username>. With capacity available, add first, test a fresh login using the new key, then remove the old key using its freshly inspected slot: user key delete <username> <0..2> --force. If full, use independent recovery access before removing a slot. user key clear <username> --force removes all that account's keys, not its password.
5. Password, role and key changes stale the affected principal; deletion removes the account. Committed changes trigger targeted web-ticket/session and SSH revocation, supplemented by authoritative currentness checks. Self changes can lose their result before it arrives; unrelated accounts should remain usable. Disconnect/timeout is not proof of rollback or cancellation of admitted work.
6. Reauthenticate to verify the new credential and rejection of the retired one; do not confuse admission throttling with a bad password. For account retirement use user delete <username> --force; for reduced privilege use user role <username> user --force. Preserve a usable administrator. A password change does not remove authorized keys, and key removal does not change the password: revoke both paths for a compromised account.
7. Update the protected operator record and revoke any reused credentials/keys on other systems separately. A key can authorize multiple device accounts; remove every affected authorization. Retire superseded host-side private-key copies according to storage policy.

HTTPS and SSH server identity rotation

Plan a maintenance window and distribute newly verified public trust through a trusted channel. Server identity changes are separate from account passwords/authorized keys and do not replace them.

• HTTPS: web certificate rotate --force replaces certificate and private key. Owner ordering is reserve → generate/commit/publish → stop/restart if running. Precommit failure leaves the old identity/logins in place; after commit, lifecycle failure never rolls the identity back. Failed stop can leave the old certificate served while UART0 reports new stored material. Rotation while stopped stays stopped.
• SSH: ssh host-key rotate --force reserves ownership → stops sessions → generates/commits/publishes → restarts if previously running. Failed stop skips mutation/start. Persistence failure may follow client disconnection and attempts to restart with unchanged material; that restart can also fail. A committed new key is never rolled back after restart failure. Rotation while stopped stays stopped.
• UART0/admin SSH expose both rotations. Browser shell allows only exact web certificate rotate --force, not SSH host-key mutation; typed HTTPS/SSH Settings provides its own bounded rotation flow. Neither browser route provides identity reset/recovery/export.
• Remote scheduling/drain/HTTP acknowledgement is not peer receipt or operation completion. On timeout/lost result, use Check Result/Refresh where available, then inspect UART0 service status and fingerprints. Do not automatically repeat rotation.
• Resolve lifecycle failures with deliberate UART0 web stop / web start or ssh stop / ssh start, checking each outcome; do not start over failed cleanup or rotate again to repair it. If recovery needs reboot, account for unsaved state and all-transport interruption.
• After recovery, compare the actually served identity against trusted UART0, replace only the relevant client trust entry, and test a fresh connection. HTTPS restart requires fresh login; SSH rotation closes SSH sessions but does not inherently require HTTPS relogin. USB/UART0 remain independent of these network-service rotations, not of reboot.

Configuration backup and reconstruction

There is no implemented configuration export/import or full backup/restore workflow, nor a supported private-identity export/restore command. save writes working state to the same device; it is not an off-device backup. Do not use raw NVS/flash dumps as routine backups or duplicate them onto replacement devices.

Maintain an access-controlled operator inventory outside the repository: reviewed firmware/build provenance, intended serial settings, station/AP policy and profile ordering, boot-enabled intent, mDNS/display settings, account roles, authorized-public-key provenance, verified public server fingerprints, and last successful save/rehearsal outcomes. Keep device/account/network identifiers in that protected record, not public issue logs. This runbook intentionally contains no populated inventory or identifier template.

Keep passwords, Wi-Fi PSKs and client private keys in a separately protected, backed-up vault with controlled recovery access and retention. Inventory entries should reference vault records rather than duplicate secrets. Ordinary status cannot reconstruct PSKs/passwords/private keys; public fingerprints cannot recreate an identity. Do not collect verifier material, session cookies or tickets for backup.

Reconstruction means first-admin provisioning, manually re-entering reviewed settings and secrets, re-importing authorized public keys, explicitly saving each configuration domain, and verifying new server trust. Replacement hardware/newly generated identities require client trust changes, not promises to restore the old identity. Test the reconstruction on an isolated spare with synthetic credentials. Exceptional forensic flash capture requires separate approval, restricted handling and retention; it remains secret-bearing, not a supported restore format.

Recovery — smallest affected domain first

Use UART0 status/error observations to distinguish connectivity, authentication, storage and lifecycle faults. Avoid reset for ordinary login throttling or protocol mismatch; restrict hostile traffic and allow natural budget refill. Native USB preserves serial access during network failure, not administrative recovery.

Fault	Deliberate recovery and scope
Wrong/lost password, healthy database	user password <username> through UART0 or another current admin; inspect/revoke authorized keys separately if compromised. Do not recover/erase the database.
User database unavailable after failed initialization	Only with explicit approval, UART0 user recover --force replaces only the user blob with an empty database, discarding all accounts/verifiers/authorized keys. Then user add <username> admin. Healthy databases, including healthy empty ones, are refused. Underlying NVS/RNG/allocation failure can still prevent recovery. Empty storage commits before dummy-verifier initialization; a later initialization error does not prove the old accounts survived.
HTTPS material unavailable/damaged	With approval, UART0 web reset --force replaces only HTTPS certificate/private key, not users or SSH/Wi-Fi configuration; it can also replace healthy material. Unlike stopped rotation, reset attempts to start a stopped service. Verify new trust and lifecycle state.
SSH material unavailable/damaged	With approval, UART0 ssh reset --force replaces only SSH host identity, not account keys/passwords or HTTPS material; it can also replace healthy material and attempts startup. Verify new trust and lifecycle state.
Ordinary configuration wrong/incompatible	Review the domain's status, then choose load, manual correction plus save, or explicitly approved domain reset. serial reset, wifi reset, mdns reset, display reset are not factory resets; Wi-Fi reset can reopen fallback access with a new password.
Service/network fault with healthy identity	Recover radio/profile or stop/start the affected service; preserve identities. If initialization remains unavailable, investigate the reported dependency/storage fault before an approved reboot or broader recovery.

Identity resets are also available to authenticated admin SSH, but UART0 is preferred for recovery; browser reset is unavailable. None of these domain commands promises secure deletion of superseded flash bytes. A failed write or lost result calls for inspection, not assumptions about persisted state.

Destructive factory-style reset — explicit approval only

There is no on-device factory-reset command. Full reset is the host-side erase/reflash procedure, not a collection of identity resets. It is not a routine upgrade, backup, password repair or prerequisite for this runbook.

1. Obtain explicit approval for this particular target and all flash/NVS data loss; confirm the physical device/port, maintenance window, protected reconstruction inventory and reviewed firmware. Disconnect sensitive serial equipment and isolate RF before erase/reflash.
2. Only in that separately authorized hardware session, from the repository root, run pio run --target erase; check success, then reflash with pio run --target upload. Use pio device monitor -b 115200 on the USB-to-UART bridge for provisioning. Do not automate or run these operations as documentation validation.
3. Erase removes firmware and all NVS domains, including users, Wi-Fi secrets/configuration, TLS/SSH identities, serial, mDNS and display settings, plus other flash contents. Ordinary upload alone is not a reset. Reflash and follow UART0-first provisioning; expect newly generated identities/AP secret and no old accounts. Re-establish client trust only after verification.
4. Erase success is not a forensic secure-erase guarantee, proof that external copies disappeared, or a general persistence guarantee. Preserve custody/disposal controls below. See the documented partition migration before changing layouts.

Decommissioning

• Isolate/disconnect the device and attached serial equipment; revoke access outside the device before custody changes. Rotate upstream/shared Wi-Fi PSKs or revoke network admission as applicable; remove saved device-AP credentials on peers.
• Remove retired device certificate exceptions/pins and SSH known-host trust on operator systems. Revoke account/key authorizations and reused credentials on other devices/services separately; a local reset cannot revoke remote trust or a copied private key.
• If approved, perform the destructive erase above; do not reflash merely to retire hardware. Logical rotation/reset can leave historical plaintext NVS pages, older credentials and private keys in flash. Retired flash/RAM captures, backups, terminal records, vault versions and workstation copies require separate retention/disposal handling.
• Flash erase, file deletion and software RAM wiping do not prove all copies are unrecoverable. Maintain physical custody; use approved media destruction/device disposal when confidentiality requires it. Do not claim secure boot, encrypted NVS, extraction resistance or guaranteed erasure.

Deferred whole-Phase-9 hardware rehearsal

Use a separately authorized, isolated expendable target and synthetic secrets. Record sanitized outcomes, exact tested build and limitations, not passwords, raw dumps or live identifiers. These checks are planned, not passed, and join the combined Phase 9 gates and administration regressions:

• First boot/healthy-empty provisioning; no remote first-admin path; fallback AP protection, saved radio-off/on behavior, and fresh trusted HTTPS/SSH login for both roles.
• Save/load/default/reset/reboot per configuration domain; failed first-default save and incompatible Wi-Fi storage; compare runtime radio/listener state with saved boot intent.
• Password/key/role/delete rotation: old credential rejection, self-revocation, unrelated-account isolation, final-admin protection, remote self-generated denial, typed generation-before-submit, lost/partial one-time output and stale results.
• Both identity rotations running/stopped; failed stop/commit/restart and lost ACK/result using reviewed fault injection; no blind replay, stored-versus-served trust comparison, reconnect/relogin and independent UART0/USB access.
• Each domain recovery preserves unrelated domains; unavailable-user recovery is UART0-only, healthy recovery refused. Reconstruct a spare from protected records without raw NVS restore.
• Separately approved destructive erase/reflash rehearsal: all domains reset, fresh identities/AP credential/accounts provisioning; rehearse external trust revocation and disposal checklist without claiming secure erasure.
• Complete existing whole-phase crash/no-dump, authentication/admission/refill, hidden-input, protocol/rekey, allocation-failure and full transport-mix/resource checks; verify one writer, isolated observers and recovery. Do not substitute this runbook review for those measurements.

Source basis: startup, console policy, users / storage, Wi-Fi commands / manager / storage, HTTPS owner / material, and SSH owner / material. Commit ordering is an implementation contract, not power-loss or hardware-validation evidence.