ESP32_Serial_Swiss_Army_Knife/docs/security_hardening.md

Security hardening — Phase 9

Status: in progress. Phase 8 is complete at the accepted 8D.22 scope. 9A crash/debug policy and 9B SSH admission/credential handling are implemented with host/build validation. At the user's request, hardware validation is deferred to Phase 9 as a whole, not an approval gate between implementation slices. This document records policy and procedures, not unrun passes or production certification.

Scope and threat model

Reduce network abuse, accidental diagnostic disclosure and unnecessary secret retention while preserving one UART1 broker writer, isolated observers and binary transparency. UART0 remains trusted physical administration/recovery; native USB remains network-independent UART1 access, not an admin console. Whole-device reboot interrupts every transport.

Secure boot and encrypted NVS are explicitly excluded by user preference. Physical flash/RAM extraction and firmware replacement remain outside the threat model even after Phase 9. There is no commitment to flash/PSRAM encryption, eFuse provisioning or physical JTAG restrictions. Software debugger-aware configuration checks do not disable physical debug access by fuse.

9A does not change partitions, at-rest encryption, dependencies or generated assets, and requires no upload or erase as part of host/build validation. The unused nvs_key and coredump partitions remain for layout compatibility. Disabling new dumps does not clear old coredump contents. Logical NVS replacement, reset and credential rotation are not secure erasure; historical plaintext copies can remain.

9A supported build baseline

src/security_build_policy.c enforces the following at compile time, with explicit settings in sdkconfig.defaults:

• Require CONFIG_ESP_COREDUMP_ENABLE_TO_NONE=y and CONFIG_ESP_SYSTEM_PANIC_SILENT_REBOOT=y.
• Reject enabled core-dump support, flash or UART dump destinations.
• Reject panic print/reboot, print/halt and GDBstub modes, plus runtime GDBstub.
• Reject CONFIG_ESP_DEBUG_OCDAWARE and CONFIG_FREERTOS_DEBUG_OCDAWARE when enabled.

The host matrix in tests/security_build_policy/run.py compiles the actual guard against synthetic configurations; it is not merely a text check of defaults. Existing generated SDK configuration must also satisfy the guard: defaults alone are not evidence of the effective build configuration.

Silent panic reboot deliberately sacrifices panic text, register dumps and backtraces for reduced crash disclosure. Reset-reason/boot information and ordinary status/logging can remain; neither silence across the full boot sequence nor general log redaction is guaranteed. A monitor exception decoder cannot reconstruct a backtrace that was never emitted.

9B SSH admission and credential handling

Boot-lifetime admission budgets

src/ssh_auth_policy.{c,h} owns three independent, fixed-size token buckets. Only the SSH owner task accesses the shared 72-byte policy; no allocation, per-peer/account map, timer task or sleep is added.

Admission class	Initial/maximum burst	Refill
New SSH handshake	6	One token per 10 seconds
Password or signed-key authentication request	6	One token per 10 seconds
Unsigned public-key probe	12	One token per 5 seconds

These are burst-plus-refill limits, not six/twelve requests in every rolling minute. All peers/accounts and both slots share each class. Idle refill stops at capacity; denials do not extend the refill deadline. Reconnects, service stop/start, identity rotation and ssh clear-counters do not replenish the pools. Reboot starts a new policy lifetime. Timestamp regression fails closed. No persistent account lockout or NVS write is introduced.

• A handshake token is taken after finding capacity but before wolfSSH_new()/handshake work. Full-capacity rejection takes no token; later allocation/IO failure does not refund it.
• Password/signed-key admission precedes database verification/authorization and ordinary key signature work. Success, invalid credentials, backend errors and rejected password-change requests do not refund admission. Unsigned probes use their own pool and cannot authenticate.
• Exhaustion shuts down/rejects the new or authenticating connection without waiting inside the owner task. Already-authenticated streams do not pass through this admission gate. The existing three-counted-attempt failure closure, two-slot bound and 15-second handshake deadline remain.
• Availability tradeoff: a client can consume the handshake burst by opening/abandoning connections and race legitimate clients for each refill. Global verification/probe pools can also starve other users. This bounds admitted work, not fair access or immunity to denial of service. TCP accept/rejection work and library parsing still occur; target latency under abuse is not yet measured. Restrict network access, stop the offending traffic and allow natural refill rather than repeatedly reconnecting/restarting. UART0/USB remain independent of these pools; HTTPS keeps its separate policy.

Callback and library contract

src/ssh_transport.c requires wolfSSH 1.4.20, certificates disabled and none authentication disabled at compile time. The reviewed parser calls ordinary-key authorization before signature verification; rejected authorizations and unsigned probes have no result callback. Password results are completed within the password callback. An explicit pending-result marker fences signed-key completion; duplicate/unexpected/closing-session results cannot promote a principal or count another completed attempt. Principal currentness is still checked at successful signature completion and route admission.

Advertising only password/publickey is not a dispatch filter in this wolfSSH version. An explicit rejecting keyboard-interactive prompt callback and per-slot context prevent its unregistered-callback path; it creates/sends no prompts and closes the connection. The advertised methods remain password/publickey. This does not certify every malformed-packet path in the library.

tests/wolfssh_auth_contract/run.py checks the reviewed internal.c SHA-256 and version, preprocesses the actual build's feature profile, and executes extracted vendor parser/send functions with narrow crypto/IO doubles. A same-version source change requires re-audit, not blindly replacing the hash. It does not replace real-client/cryptographic integration testing. The positive SendChannelData() return contract means the caller's accepted prefix has been copied, including its consumed-data WANT_WRITE case; it is not peer acknowledgement.

Counters and secret lifetime

ssh counters adds aggregate-only diagnostics:

• handshakes / handshake-throttled: admitted handshake work / rate-denied connections, separate from capacity failures.
• verifications / verification-throttled: admitted password/signed-key requests / rate-denied requests. Admission does not imply the verifier ran or completed.
• probes / probe-throttled: admitted/denied unsigned-key lookups, not completed credential attempts.
• attempt-limit-closes, backend-errors, method-rejects: three-attempt closures, database auth/authorization/currentness errors, and rejected callback-level methods (including keyboard). These are not counts of every malformed SSH packet.

Existing auth-attempts/auth-failures remain completed counted outcomes; rejected password changes count, unsigned probes and rate-denied requests do not. Signed-key results finalize once after authorized work. These admission/auth counters saturate at UINT64_MAX, contain no submitted credentials/identities, and may be cleared independently of enforcement state.

The transport now wipes consumed admin RX bytes, positively accepted admin TX bytes, and the full retired slot while retaining its generation. Partial/retry paths preserve pending bytes. Serial-route hot-path behavior is unchanged. This shortens application plaintext lifetime; it is not a claim that wolfSSH/wolfSSL/mbedTLS, stack or PSRAM copies are all erased.

Hidden UART0 and shared remote-console prompts now reject overflow or unsupported bytes on submission with a wiped output buffer and ESP_ERR_INVALID_SIZE, rather than accepting a truncated/normalized prefix. The failure remains sticky after Backspace/Delete. Printable ASCII, CR/LF submission, Backspace/Delete and Ctrl-C retain their defined roles; visible command-line editing is unchanged. Existing callers prevent a rejected password or confirmation from reaching persistence. For pasted passwords, exceeding 64 characters or including unsupported bytes requires a fresh attempt; the password policy itself is unchanged.

Operational profiles

These are handling and validation profiles of the same supported build baseline, not separate PlatformIO environments or selectable security overrides.

Profile	Operational rules
Development	Keep the guard enabled; use synthetic credentials and controlled serial payloads for fault investigation. Keep UART0 recovery available. Review captures before sharing.
Test	Use an isolated, expendable target and synthetic secrets; record exact source/configuration, host/build results and device observations. Exercise crashes and recovery without exporting raw memory.
Production	Use the same guard, restrict physical/network access, verify device identity through trusted UART0, and apply reviewed provisioning/rotation/recovery procedures. Readiness remains pending Phase 9 review and target evidence.

If richer crash debugging is essential, use an isolated synthetic-secret build outside this supported baseline. It requires explicit reviewed changes to the source policy and applicable configuration; no bypass flag is provided. Do not use real credentials or deploy that build as production firmware. Restore and revalidate the supported policy before release.

Raw flash, RAM and dumps can contain Wi-Fi passwords, private keys, password verifiers, session material and serial payloads. Treat them as secret-bearing and do not export them as routine diagnostics. Prefer bounded status/counter observations and reviewed synthetic-secret reproductions. Restrict any exceptional artifacts and define retention/deletion before collecting them; deletion is not a secure-erase guarantee.

Validation gates

Host and build — passed 2026-09-15 (9A and 9B)

From the repository root:

python3 tests/security_build_policy/run.py
python3 tests/ssh_auth_policy/run.py
python3 tests/ssh_auth_transport/run.py
python3 tests/hidden_input/run.py
pio run
python3 tests/security_build_policy/run.py --sdkconfig-header .pio/build/esp32-s3-devkitc-1-n16r8/config/sdkconfig.h
python3 tests/wolfssh_auth_contract/run.py

Latest 9B pio run passed with 94,340 B linked RAM / 1,829,925 B flash, +144 B RAM / +1,360 B flash against 9A. This is linked size, not measured runtime headroom. All four new focused host suites passed, including 35 pinned-vendor parser/control-flow cases and actual compiler-feature preprocessing. The crash-policy generated-header matrix passed all 18 cases. Related SSH management/security/runtime, console boundary/accounts/lifecycle/policy, cookie-auth SSH/accounts and browser-admin transport/tickets regressions passed (11 commands). Independent review and git diff --check passed. Use CCACHE_DISABLE=1 on host commands if the compiler wrapper's cache is read-only in a sandbox. No upload, erase, eFuse operation or target test was performed.

Record the revision, compiler/build outcome and effective configuration. Confirm that the matrix accepts the supported configuration, rejects each prohibited option independently, and rejects absent/disabled required settings. Confirm the normal firmware build compiles the guard. A rejected unsafe configuration is an expected negative-test result, not a firmware build pass. Neither these commands nor a successful build proves target panic behavior.

Combined Phase 9 target validation — deferred, not run

Retain these checks for the final phase test session; do not stop implementation for a separate 9A/9B sign-off.

Crash and recovery

1. On an isolated synthetic-secret target, record the tested image/configuration and capture UART0 at 115200 baud. Verify normal boot, UART0 administration, native USB UART1 access, HTTPS and SSH before fault testing.
2. Through separately reviewed test-only fault injection, trigger a controlled panic with the supported build policy intact. Verify reboot rather than halt/debugger wait, no panic register/backtrace output and no UART core dump. Record any remaining boot/reset information; do not promise complete UART silence.
3. Verify no new flash core dump is written using a reviewed target-side pass/fail check that does not export partition contents. Distinguish old partition contents from a new write; do not erase the partition merely to claim this test passed.
4. After reboot, verify UART0 recovery and USB serial access, then authenticated HTTPS/SSH and broker writer/observer behavior. With network services unavailable, verify UART0 and native USB still work. Review routine status/log output using synthetic secrets; this is bounded evidence, not universal redaction proof.
5. Record outcomes and limitations in the combined Phase 9 acceptance. Device flashing/fault injection requires a separately authorized hardware session; no eFuse changes, partition migration or erase is required by this policy.

Authentication, input and loaded isolation

1. On a restricted test network using synthetic credentials, exercise password and Ed25519/P-256 key login for both roles, including a client offering multiple keys. Verify unsigned probes, wrong passwords/signatures, stale-principal rejection and normal shell admission. Explicit keyboard-interactive requests must close/reject without a crash or prompt.
2. Exhaust each admission class separately, respecting the independent budgets. For verification testing reuse admitted connections (up to the existing three-failure limit) so handshake exhaustion does not mask the verification gate. Verify counter deltas, reconnect resistance, natural refill and that successful logins also consume capacity. Unsigned probes must not increase completed auth-attempts.
3. From UART0, clear counters and stop/start SSH while exhausted; observe that neither grants fresh tokens. Account for time elapsed during these operations. Do not assume that a reconnect failure indicates bad credentials. A quiet 60-second period replenishes all pools; ongoing hostile traffic can keep them depleted.
4. Keep an established SSH serial stream and USB/browser clients active while generating bounded invalid-login/reconnect traffic. Record serial/broker drops, UART0 command latency, SSH stream responsiveness, internal/DMA minima and recovery. Do not use this admission policy to claim zero CPU impact; TCP/kernel work, KDF/signature work within budget and two-slot occupancy still matter.
5. Test hidden credentials at maximum length and one byte over, different suffixes past the limit, unsupported input bytes, overflow followed by editing, Ctrl-C, disconnect and confirmation failure on UART0 and remote administration. No rejected prefix may be persisted or echoed. Check both CR/LF behavior, including delayed UART0 LF delivery: the current UART0 reader relies on next-prompt input flushing, unlike the remote reader's explicit paired-LF handling; host fakes do not prove device timing.
6. Exercise generated-password delivery with slow/partial remote output and short subsequent commands, then disconnect/reconnect. Application-buffer wipe assertions are host evidence; do not export live RAM to establish a device pass.

Staged next work

• Next: complete the secret-lifetime and protocol review. 9B fixes confirmed admin-buffer retention and hidden-input truncation, but does not complete the library-allocation audit. Inspect pinned private-key import/destruction, password-packet storage and buffer growth/free paths in mbedTLS/wolfSSL/wolfSSH before claiming full zeroization. Review negotiated TLS/SSH algorithms, certificate trust/validity, browser security headers and password/KDF policy without casually changing shared crypto or persisted identities.
• Web admission review remains separate. src/web_cookie_auth.c still limits credential verifications to five per 60-second fixed global window, shared across clients. This is not a sliding window or persistent account lockout; auth lifecycle restart resets it. Challenge starvation/global-budget starvation and the receive-before-throttle path remain review items, not changes delivered by 9B.
• Later: maintenance and lifecycle. Review ESP-IDF/wolfSSL/wolfSSH advisories and dependency licenses, then plan any upgrades separately. Complete provisioning, key rotation, backup, factory reset, recovery and decommissioning runbooks without claiming physical-extraction resistance or secure erasure.
• Phase 10: OTA trust. Define independent image-signature verification, trust-anchor provisioning, rotation/revocation, rollback/downgrade and recovery policy without secure boot. Authenticated transport alone is not image-signing policy, and OTA signature checks cannot prevent physical firmware replacement.

See the roadmap, electrical procedures and administration regressions for wider gates. Production readiness remains pending; Phase 8 acceptance is not reopened by these follow-ups.