ESP32_Serial_Swiss_Army_Knife/docs/security_hardening.md

Security hardening — Phase 9

Status: in progress. Phase 8 is complete at the accepted 8D.22 scope. 9A crash/debug policy, 9B SSH admission/credential handling and 9C library cleanup/protocol policy are implemented with host/build validation. At the user's request, hardware validation is deferred to Phase 9 as a whole, not an approval gate between implementation slices. This document records policy and procedures, not unrun passes or production certification.

Scope and threat model

Reduce network abuse, accidental diagnostic disclosure and unnecessary secret retention while preserving one UART1 broker writer, isolated observers and binary transparency. UART0 remains trusted physical administration/recovery; native USB remains network-independent UART1 access, not an admin console. Whole-device reboot interrupts every transport.

Secure boot and encrypted NVS are explicitly excluded by user preference. Physical flash/RAM extraction and firmware replacement remain outside the threat model even after Phase 9. There is no commitment to flash/PSRAM encryption, eFuse provisioning or physical JTAG restrictions. Software debugger-aware configuration checks do not disable physical debug access by fuse.

9A does not change partitions, at-rest encryption, dependencies or generated assets, and requires no upload or erase as part of host/build validation. The unused nvs_key and coredump partitions remain for layout compatibility. Disabling new dumps does not clear old coredump contents. Logical NVS replacement, reset and credential rotation are not secure erasure; historical plaintext copies can remain.

9A supported build baseline

src/security_build_policy.c enforces the following at compile time, with explicit settings in sdkconfig.defaults:

• Require CONFIG_ESP_COREDUMP_ENABLE_TO_NONE=y and CONFIG_ESP_SYSTEM_PANIC_SILENT_REBOOT=y.
• Reject enabled core-dump support, flash or UART dump destinations.
• Reject panic print/reboot, print/halt and GDBstub modes, plus runtime GDBstub.
• Reject CONFIG_ESP_DEBUG_OCDAWARE and CONFIG_FREERTOS_DEBUG_OCDAWARE when enabled.

The host matrix in tests/security_build_policy/run.py compiles the actual guard against synthetic configurations; it is not merely a text check of defaults. Existing generated SDK configuration must also satisfy the guard: defaults alone are not evidence of the effective build configuration.

Silent panic reboot deliberately sacrifices panic text, register dumps and backtraces for reduced crash disclosure. Reset-reason/boot information and ordinary status/logging can remain; neither silence across the full boot sequence nor general log redaction is guaranteed. A monitor exception decoder cannot reconstruct a backtrace that was never emitted.

9B SSH admission and credential handling

Boot-lifetime admission budgets

src/ssh_auth_policy.{c,h} owns three independent, fixed-size token buckets. Only the SSH owner task accesses the shared 72-byte policy; no allocation, per-peer/account map, timer task or sleep is added.

Admission class	Initial/maximum burst	Refill
New SSH handshake	6	One token per 10 seconds
Password or signed-key authentication request	6	One token per 10 seconds
Unsigned public-key probe	12	One token per 5 seconds

These are burst-plus-refill limits, not six/twelve requests in every rolling minute. All peers/accounts and both slots share each class. Idle refill stops at capacity; denials do not extend the refill deadline. Reconnects, service stop/start, identity rotation and ssh clear-counters do not replenish the pools. Reboot starts a new policy lifetime. Timestamp regression fails closed. No persistent account lockout or NVS write is introduced.

• A handshake token is taken after finding capacity but before wolfSSH_new()/handshake work. Full-capacity rejection takes no token; later allocation/IO failure does not refund it.
• Password/signed-key admission precedes database verification/authorization and ordinary key signature work. Success, invalid credentials, backend errors and rejected password-change requests do not refund admission. Unsigned probes use their own pool and cannot authenticate.
• Exhaustion shuts down/rejects the new or authenticating connection without waiting inside the owner task. Already-authenticated streams do not pass through this admission gate. The existing three-counted-attempt failure closure, two-slot bound and 15-second handshake deadline remain.
• Availability tradeoff: a client can consume the handshake burst by opening/abandoning connections and race legitimate clients for each refill. Global verification/probe pools can also starve other users. This bounds admitted work, not fair access or immunity to denial of service. TCP accept/rejection work and library parsing still occur; target latency under abuse is not yet measured. Restrict network access, stop the offending traffic and allow natural refill rather than repeatedly reconnecting/restarting. UART0/USB remain independent of these pools; HTTPS keeps its separate policy.

Callback and library contract

src/ssh_transport.c requires wolfSSH 1.4.20, certificates disabled and none authentication disabled at compile time. The reviewed parser calls ordinary-key authorization before signature verification; rejected authorizations and unsigned probes have no result callback. Password results are completed within the password callback. An explicit pending-result marker fences signed-key completion; duplicate/unexpected/closing-session results cannot promote a principal or count another completed attempt. Principal currentness is still checked at successful signature completion and route admission.

Advertising only password/publickey is not a dispatch filter in this wolfSSH version. An explicit rejecting keyboard-interactive prompt callback and per-slot context prevent its unregistered-callback path; it creates/sends no prompts and closes the connection. The advertised methods remain password/publickey. This does not certify every malformed-packet path in the library.

tests/wolfssh_auth_contract/run.py checks the reviewed internal.c SHA-256 and version, preprocesses the actual build's feature profile, and executes extracted vendor parser/send functions with narrow crypto/IO doubles. A same-version source change requires re-audit, not blindly replacing the hash. It does not replace real-client/cryptographic integration testing. The positive SendChannelData() return contract means the caller's accepted prefix has been copied, including its consumed-data WANT_WRITE case; it is not peer acknowledgement.

Counters and secret lifetime

ssh counters adds aggregate-only diagnostics:

• handshakes / handshake-throttled: admitted handshake work / rate-denied connections, separate from capacity failures.
• verifications / verification-throttled: admitted password/signed-key requests / rate-denied requests. Admission does not imply the verifier ran or completed.
• probes / probe-throttled: admitted/denied unsigned-key lookups, not completed credential attempts.
• attempt-limit-closes, backend-errors, method-rejects: three-attempt closures, database auth/authorization/currentness errors, and rejected callback-level methods (including keyboard). These are not counts of every malformed SSH packet.

Existing auth-attempts/auth-failures remain completed counted outcomes; rejected password changes count, unsigned probes and rate-denied requests do not. Signed-key results finalize once after authorized work. These admission/auth counters saturate at UINT64_MAX, contain no submitted credentials/identities, and may be cleared independently of enforcement state.

The transport now wipes consumed admin RX bytes, positively accepted admin TX bytes, and the full retired slot while retaining its generation. Partial/retry paths preserve pending bytes. Serial-route hot-path behavior is unchanged. This shortens application plaintext lifetime; it is not a claim that wolfSSH/wolfSSL/mbedTLS, stack or PSRAM copies are all erased.

Hidden UART0 and shared remote-console prompts now reject overflow or unsupported bytes on submission with a wiped output buffer and ESP_ERR_INVALID_SIZE, rather than accepting a truncated/normalized prefix. The failure remains sticky after Backspace/Delete. Printable ASCII, CR/LF submission, Backspace/Delete and Ctrl-C retain their defined roles; visible command-line editing is unchanged. Existing callers prevent a rejected password or confirmation from reaching persistence. For pasted passwords, exceeding 64 characters or including unsupported bytes requires a fresh attempt; the password policy itself is unchanged.

9C library cleanup and protocol policy

Library review and maintenance contract records the scoped audit, corrected paths, existing cleanup and limits. This is not exhaustive library certification or a dependency security-release review.

Reproducible source corrections

tools/security_overrides.py verifies full original-file SHA-256 values and ESP-IDF 5.5.0, applies exact-once edits, and generates four corrected sources under the build directory. cmake/security_overrides.cmake, included after project(), replaces exactly the corresponding sources in existing IDF/component targets, retaining compilation properties. Installed SDK/managed sources and their notices remain unchanged. Missing, changed or ambiguous sources fail configuration; there is no unpatched fallback. Do not edit derived files or repin a hash merely to make an upgrade build.

• HTTPS: delete TLS on post-handshake transport-allocation failure; fully destroy retained TLS configuration on failed HTTPD start; wipe the copied raw private key before free. Failed stop still retains live ownership.
• HTTPD parser: allocate/copy/wipe/free scratch on resize, preserve the old pointer on allocation failure, wipe final scratch, and handle the null initial parser pointer without undefined subtraction. Pending/unread bytes retain their existing behavior.
• wolfSSH password parser: bound both password lengths against the actual packet before application callbacks, reject malformed change-password fields without calling authentication, and wipe the bounded method-specific payload suffix before failure responses. Username/service/method prefixes remain intact. The current project callbacks are synchronous; library WS_AUTH_PENDING retains the payload for retry and is not claimed wiped.
• ESP-TLS server configuration: enforce the static-lifetime TLS list below before handshake setup; client defaults and global cryptographic primitives remain unchanged. IDF dynamic TLS buffers are rejected because their cleanup bypasses the reviewed upstream record-buffer wipe.

The new src/ssh_memory.{c,h} wolfSSL/wolfCrypt allocation hooks wipe the full owned usable allocation before release, including library import-failure and dynamic packet-buffer copies. They require the reviewed unpoisoned IDF 5.5.0 heap configuration; poisoning modes fail compilation rather than risking canary writes. No allocation header is added. Shrink retains capacity and wipes the tail; growth allocates/copies before wiping/freeing the old block, preserving it on allocation failure. PSRAM preference/internal fallback is unchanged. Growth and HTTPD scratch resizing temporarily need both blocks; lower linked size is not evidence of safe runtime headroom. Live inline buffers, in-place compaction tails, stack spills and every crypto intermediate are not comprehensively covered.

Explicit network protocol policy

Setting	Allowed values, in preference order
HTTPS version	TLS 1.2 only; server renegotiation disabled
HTTPS suites	TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
SSH KEX	curve25519-sha256, ecdh-sha2-nistp256
SSH host key	ecdsa-sha2-nistp256
SSH ciphers, both directions	aes128-gcm@openssh.com, aes256-gcm@openssh.com
SSH MAC advertisement	hmac-sha2-256 (GCM supplies packet authentication)
SSH user-key advertisement	ssh-ed25519, ecdsa-sha2-nistp256; database authorization remains authoritative

src/ssh_protocol_policy.c uses permanent strings and checks every setter; any failure destroys the unpublished context without fallback. Tests verify the actual compiler's available algorithms and generated KEXINIT lists, not merely setter success. The server-only TLS correction avoids breaking future outbound HTTPS clients by globally removing RSA-certificate suites.

Compatibility: TLS CBC/CCM/ARIA-only and SSH CBC/CTR-only clients no longer connect; removed KEX-only clients also fail. Mainstream-client interoperability and rekey remain target gates, not host-test claims. Existing TLS/SSH identity and user-key storage need no rotation or migration. Password/KDF, certificate validity/trust and browser-header policy were reviewed and retained with documented limits; no blind KDF-cost increase or HSTS policy was introduced.

Web admission and shorter plaintext lifetimes

A non-consuming quota/epoch check now runs after valid challenge consumption but before body receive. An already-exhausted verification budget returns 429/Retry-After without receiving/parsing credentials or calling the verifier; unread bodies still cause connection closure, not draining. The authoritative reservation remains after parsing. Raw JSON is wiped before verification, parsed credentials afterward, and both before error-response sending. Header strings remain live through synchronous serialization.

The existing global five-verifications/60-second fixed window is unchanged: malformed requests do not charge it, counter clear does not replenish it, and web service restart does. This differs deliberately from SSH's boot-lifetime buckets. The first boot-minute window remains anchored at uptime zero. Challenge monopolization, global-budget starvation and malformed-body work while budget is available are not solved by this early check.

Operational profiles

These are handling and validation profiles of the same supported build baseline, not separate PlatformIO environments or selectable security overrides.

Profile	Operational rules
Development	Keep the guard enabled; use synthetic credentials and controlled serial payloads for fault investigation. Keep UART0 recovery available. Review captures before sharing.
Test	Use an isolated, expendable target and synthetic secrets; record exact source/configuration, host/build results and device observations. Exercise crashes and recovery without exporting raw memory.
Production	Use the same guard, restrict physical/network access, verify device identity through trusted UART0, and apply reviewed provisioning/rotation/recovery procedures. Readiness remains pending Phase 9 review and target evidence.

If richer crash debugging is essential, use an isolated synthetic-secret build outside this supported baseline. It requires explicit reviewed changes to the source policy and applicable configuration; no bypass flag is provided. Do not use real credentials or deploy that build as production firmware. Restore and revalidate the supported policy before release.

Raw flash, RAM and dumps can contain Wi-Fi passwords, private keys, password verifiers, session material and serial payloads. Treat them as secret-bearing and do not export them as routine diagnostics. Prefer bounded status/counter observations and reviewed synthetic-secret reproductions. Restrict any exceptional artifacts and define retention/deletion before collecting them; deletion is not a secure-erase guarantee.

Validation gates

Host and build — passed 2026-09-15 (9A–9C)

From the repository root:

python3 tests/security_build_policy/run.py
python3 tests/ssh_auth_policy/run.py
python3 tests/ssh_auth_transport/run.py
python3 tests/hidden_input/run.py
pio run
python3 tests/security_build_policy/run.py --sdkconfig-header .pio/build/esp32-s3-devkitc-1-n16r8/config/sdkconfig.h
python3 tests/wolfssh_auth_contract/run.py
python3 tests/sdk_security_overrides/run.py --build-dir .pio/build/esp32-s3-devkitc-1-n16r8
python3 tests/ssh_memory/run.py
python3 tests/ssh_protocol_policy/run.py
python3 tests/web_cookie_auth/run.py --admission

Latest 9C pio run passed with 94,340 B linked RAM / 1,831,309 B flash, unchanged linked RAM / +1,384 B flash against 9B. This is linked size, not measured runtime headroom. Five focused suites passed after final HTTPD first-read correction: pinned SDK cleanup/TLS/source registration, 135 generated wolfSSH parser/control-flow cases, secure allocator, SSH policy (including 15 actual context-integration cases), and web early admission/wiping. The allocator's optional installed-SDK extent contract was also run with the installed IDF path and passed; plain invocation reports that optional check skipped. Seventeen integrated regression commands passed before the final first-read addition, including SSH auth/management, HTTPD idle, HTTPS lifecycle, five cookie-auth modes and the 18-case crash-policy matrix. All ten existing cookie-auth domain modes also passed during implementation. Independent reviews found no blocking issues; the identified inherited first-read pointer issue was corrected and tested. Use CCACHE_DISABLE=1 on host commands if the compiler wrapper's cache is read-only in a sandbox. No upload, erase, eFuse operation or target test was performed.

Record the revision, compiler/build outcome and effective configuration. Confirm that the matrix accepts the supported configuration, rejects each prohibited option independently, and rejects absent/disabled required settings. Confirm the normal firmware build compiles the guard. A rejected unsafe configuration is an expected negative-test result, not a firmware build pass. Neither these commands nor a successful build proves target panic behavior.

Combined Phase 9 target validation — deferred, not run

Retain these checks for the final phase test session; do not stop implementation for a separate 9A/9B/9C sign-off.

Crash and recovery

1. On an isolated synthetic-secret target, record the tested image/configuration and capture UART0 at 115200 baud. Verify normal boot, UART0 administration, native USB UART1 access, HTTPS and SSH before fault testing.
2. Through separately reviewed test-only fault injection, trigger a controlled panic with the supported build policy intact. Verify reboot rather than halt/debugger wait, no panic register/backtrace output and no UART core dump. Record any remaining boot/reset information; do not promise complete UART silence.
3. Verify no new flash core dump is written using a reviewed target-side pass/fail check that does not export partition contents. Distinguish old partition contents from a new write; do not erase the partition merely to claim this test passed.
4. After reboot, verify UART0 recovery and USB serial access, then authenticated HTTPS/SSH and broker writer/observer behavior. With network services unavailable, verify UART0 and native USB still work. Review routine status/log output using synthetic secrets; this is bounded evidence, not universal redaction proof.
5. Record outcomes and limitations in the combined Phase 9 acceptance. Device flashing/fault injection requires a separately authorized hardware session; no eFuse changes, partition migration or erase is required by this policy.

Authentication, input and loaded isolation

1. On a restricted test network using synthetic credentials, exercise password and Ed25519/P-256 key login for both roles, including a client offering multiple keys. Verify unsigned probes, wrong passwords/signatures, stale-principal rejection and normal shell admission. Explicit keyboard-interactive requests must close/reject without a crash or prompt.
2. Exhaust each admission class separately, respecting the independent budgets. For verification testing reuse admitted connections (up to the existing three-failure limit) so handshake exhaustion does not mask the verification gate. Verify counter deltas, reconnect resistance, natural refill and that successful logins also consume capacity. Unsigned probes must not increase completed auth-attempts.
3. From UART0, clear counters and stop/start SSH while exhausted; observe that neither grants fresh tokens. Account for time elapsed during these operations. Do not assume that a reconnect failure indicates bad credentials. A quiet 60-second period replenishes all pools; ongoing hostile traffic can keep them depleted.
4. Keep an established SSH serial stream and USB/browser clients active while generating bounded invalid-login/reconnect traffic. Record serial/broker drops, UART0 command latency, SSH stream responsiveness, internal/DMA minima and recovery. Do not use this admission policy to claim zero CPU impact; TCP/kernel work, KDF/signature work within budget and two-slot occupancy still matter.
5. Test hidden credentials at maximum length and one byte over, different suffixes past the limit, unsupported input bytes, overflow followed by editing, Ctrl-C, disconnect and confirmation failure on UART0 and remote administration. No rejected prefix may be persisted or echoed. Check both CR/LF behavior, including delayed UART0 LF delivery: the current UART0 reader relies on next-prompt input flushing, unlike the remote reader's explicit paired-LF handling; host fakes do not prove device timing.
6. Exercise generated-password delivery with slow/partial remote output and short subsequent commands, then disconnect/reconnect. Application-buffer wipe assertions are host evidence; do not export live RAM to establish a device pass.

Protocol compatibility and allocation-failure recovery

1. Verify both allowed TLS suites and both SSH GCM ciphers using compatible clients; force excluded CBC/CTR/other-only offers and confirm rejection. Exercise both SSH KEX choices and both user-key types, initial handshake and rekey, plus TLS renegotiation rejection. Retain UART0 access; do not rotate identities to work around an algorithm mismatch.
2. With synthetic credentials, test truncated/oversized SSH password and change-password packets: no authentication callback for malformed fields, no crash, bounded disconnect/recovery. Host canary assertions are not real encrypted-packet coverage.
3. Exercise HTTPS failed-start, post-handshake allocation failure, normal/failed-stop retry and split-header scratch allocation failure on a separately reviewed fault-injection image. Observe recovery/no accumulating allocation loss without exporting keys or RAM. Failed stop must not prematurely free live TLS state.
4. Repeatedly start/stop HTTPS and SSH and stress header parsing/authentication under the full transport mix. Capture internal/DMA/PSRAM free/minimum/largest-block and stack margins alongside serial/broker loss counters. Specifically measure old-plus-new allocation peaks and secure-free CPU cost; previous very low internal minima remain important.
5. Verify exhausted web login returns early without stalled-body work, clears the used pre-login challenge, and recovers after the documented window. Check malformed requests below quota and correct credentials for normal behavior; do not infer fairness from a rate-limit pass.

Staged next work

• Next — 9D maintenance and lifecycle. Review current ESP-IDF/wolfSSL/wolfSSH advisories and third-party licenses, then plan any upgrades and source re-audits separately. Complete provisioning, key rotation, backup, factory reset, recovery and decommissioning runbooks without claiming physical-extraction resistance or secure erasure. No external advisory or complete license review has been performed by 9C.
• Retained evidence limits: 9C completes a bounded cleanup/protocol review, not every-library-copy zeroization. Live inline residue, compaction tails, hardware/stack intermediates, global admission starvation and resource/interop measurements remain documented limitations or combined target gates. Any additional hardening must preserve owner lifetimes and bounded recovery.
• Phase 10: OTA trust. Define independent image-signature verification, trust-anchor provisioning, rotation/revocation, rollback/downgrade and recovery policy without secure boot. Authenticated transport alone is not image-signing policy, and OTA signature checks cannot prevent physical firmware replacement.

See the roadmap, electrical procedures and administration regressions for wider gates. Production readiness remains pending; Phase 8 acceptance is not reopened by these follow-ups.