ESP32_Serial_Swiss_Army_Knife/docs/phase9_validation.md

Phase 9 bounded host validation

tools/validate_phase9.py is a thin, sequential orchestrator for existing test runners, not a new security test implementation or a Phase 9 acceptance gate. See security hardening for scope, historical evidence, and the deferred whole-phase target checklist.

Current candidate execution — 2026-09-18

Supplied final parent command, run from the repository root:

CCACHE_DISABLE=1 python3 -B tools/validate_phase9.py --build-dir .pio/idf-candidate-5.5.3/app-validated/.pio/build/esp32-s3-devkitc-1-n16r8 --idf-path .pio/idf-candidate-5.5.3/core/packages/framework-espidf --platformio-core-dir .pio/idf-candidate-5.5.3/core --interop --web-performance

PASS 24/24. This selects the actual fresh isolated build, SDK and toolchain core; it is not a default-root-build execution. The separate fresh .pio/idf-candidate-5.5.3/app-validated build PASS reports 95,552 B linked RAM / 1,749,493 B flash, versus historical 94,340 / 1,768,901 B (+1,212 B RAM / −19,408 B flash). The default parent pio run timed out after 200 seconds during installation, before compilation; no normal root build PASS is claimed.

Pre/post source equality: 3,237 files, SHA-256 3a1af78c15cfdd02da1055a8957b4086f9018862b7aa1c7c52fd2401a1a0a031. Actual generated-input registration covers nine C sources plus one forced header. WS receive tests exercise generated code: 982 cases / 10 mutation checks, including the five signed sizeof corrections. The stale web-cookie fixture asserting IDF 5.5.0 was corrected, not bypassed. Historical stale-WS compilation/coverage failures in the rebase review are resolved by this final snapshot, not hidden or retroactively called passes.

The fix-bearing Wi-Fi bundle is integrated in the candidate, with unchanged PMF/WPA3; radio-hardware vulnerability closure and full target/resource/recovery gates remain pending. Nine notice catalog entries were semantically rebased, the other 66 unchanged (75 total); supplied notice evidence is 36 fixtures PASS, two actual deterministic bundles each 77 files / 4,433,930 bytes. Archive pins and source equality do not prove complete immutable root/ancillary/Python dependency closure, legal clearance or Phase 9 acceptance. This documentation update records supplied parent evidence; it did not rerun these builds, suites, bundles or hardware tests.

Usage

From the repository root (the script itself also resolves the root independently of the caller's working directory):

python3 -B tools/validate_phase9.py --dry-run
python3 -B tools/validate_phase9.py
python3 -B tools/validate_phase9.py --fail-fast
python3 -B tools/validate_phase9.py --timeout 300

Build and OpenSSH interoperability are independent explicit opt-ins:

python3 -B tools/validate_phase9.py --build --build-timeout 600
python3 -B tools/validate_phase9.py --interop --timeout 300
python3 -B tools/validate_phase9.py --build --interop --dry-run

These examples are usage, not execution claims. --build runs exactly pio run first; it does not upload, erase, monitor, or select another PlatformIO environment. PlatformIO itself may download dependencies or write its normal external caches. The caller/parent must choose whether to build and arrange any required permission separately. No installer, permission grant, network retry, or sandbox escalation is implemented. A failed/timed-out build skips every dependent host command to avoid testing stale artifacts as if that build succeeded.

--interop adds only --interop to tests/wolfssh_order_contract/run.py (its default single matrix). That runner uses OpenSSH and local AF_UNIX sockets, not an IP host or device. Socket permissions and installed OpenSSH tooling must already be available; denial is a failure, not a reason to grant permissions. Neither option is enabled by default. --web-performance independently adds the generated-WS performance runner (24 commands total). --build-dir, --idf-path and --platformio-core-dir select existing candidate artifacts/SDK/core; --build rejects a non-default build directory, which must be built separately. No target address, credentials, arbitrary command, or device-operation argument is accepted.

Coverage and prerequisites

The fixed default plan contains 23 commands; --dry-run prints their exact argv vectors and timeouts without launching anything or checking prerequisites. There is no discovery-time execution of runners (some run at import/top level). Paths and options were inspected in their actual run.py implementations.

Scope	Existing runners / modes
9A effective crash policy	security_build_policy --sdkconfig-header .pio/build/esp32-s3-devkitc-1-n16r8/config/sdkconfig.h
9B admission / credentials	ssh_auth_policy, ssh_auth_transport, hidden_input
9C allocator	ssh_memory
Current seven-suite baseline	sdk_security_overrides --build-dir .pio/build/esp32-s3-devkitc-1-n16r8, wolfssh_auth_contract, ssh_protocol_policy, wolf_crypto_policy, wolfssh_parser_contract, wolfssh_order_contract, release_notices
Focused SSH/admin and HTTPS lifecycle regressions	ssh_management, admin_console_boundary, admin_ssh_policy, web_admin_transport, web_admin_tickets, web_httpd_idle
Web early admission, credentials and administrative lifecycle	web_cookie_auth default plus separate --admin, --accounts, --ssh, --lifecycle commands

Every runner path is tests/<name>/run.py. Auth contract, protocol policy, and crypto policy receive explicit --compile-commands .pio/build/esp32-s3-devkitc-1-n16r8/compile_commands.json arguments. No --host-only, candidate injection, or silent fallback is used. Default operation requires an existing configured build, installed pinned SDK/managed sources, the production toolchain and host compiler/libraries needed by these runners (including OpenSSL development support for relevant tests). It does not run PlatformIO to satisfy those requirements. Missing explicit files/executables are PREREQ failures; deeper source/toolchain prerequisites are checked by the existing runners and surface as nonzero FAIL results. Existing artifacts are not proof that the current checkout has been freshly built.

The current cookie-auth runner has no --admission branch: the default exercises early admission/wiping, and each selected domain mode repeats that baseline. Unrelated display/network/UI suites are intentionally excluded; web performance is an explicit opt-in. ssh_memory retains its documented optional installed-IDF extent-source check as skipped when no --idf-path is supplied; explicit candidate selection enables it. Its normal allocator contracts still run. The ordering runner's candidate --target-contracts, separate --pio-adapter regression, and repeated interop matrices are not added implicitly. Run those separately when their particular evidence is needed. This is a focused full Phase 9 host regression set, not every test in the repository.

Bounds, output, and result semantics

• Each host command has a 180-second wall-clock timeout by default; the optional build gets 600 seconds. Both timeout flags accept finite values in (0, 3600]. The fixed sequential plan bounds total child execution by the sum of its timeouts (default 4,140 seconds), plus launch/reporting/cleanup overhead.
• POSIX process groups are required. Timeout or Ctrl-C kills the active process group and reaps the runner; successful/failed exit also retires any remaining group descendants. This covers normal compiler/test children, not deliberately detached processes. Group reaping has a five-second wait bound.
• Commands use argv lists, no shell evaluation or interpolation, with stdin closed. The child environment is copied with CCACHE_DISABLE=1; explicit --idf-path and --platformio-core-dir also set resolved IDF_PATH and PLATFORMIO_CORE_DIR for children; arbitrary compiler, SDK, proxy and other caller environment settings are not overwritten. The parent's environment is not mutated.
• stdout/stderr are inherited live. The orchestrator creates no persistent log, JSON capture, environment dump, or secret artifact. Existing runners retain their own temporary fixture/build behavior; forced termination can bypass their temporary-directory cleanup. Console output is not a redaction guarantee: use synthetic fixtures, review before sharing, and do not redirect output into routine secret-bearing persistent captures. Build outputs remain normal PlatformIO artifacts only when explicitly requested.
• PASS means the entire runner exited zero, not that an earlier printed PASS line appeared. The final summary records each command's exit, timeout or missing prerequisite. Defaults collect failures and continue independent commands; --fail-fast reports remaining commands as SKIP. Build failure always stops dependent commands. No selected suite is silently dropped.
• Exit 0 means all selected commands passed, or a dry-run plan was printed. Dry-run uses PLAN, never PASS, and is not validation. Failures, timeouts, prerequisites and dependent skips return 1; invalid CLI arguments return 2; Ctrl-C returns 130. Optional build/interop omissions are announced separately and do not turn a successful host run into failure.

Orchestrator fixture tests

python3 -B tests/phase9_validation/run.py

Offline unittest groups check the exact plan/paths/strict arguments, independent opt-ins, timeout configuration and invalid values, dry-run nonexecution, default build/interop omission, failure collection, fail-fast and failed-build skips, missing file/executable handling, literal argv/environment preservation, timeout/descendant termination, and inherited streams/closed stdin. They execute only small Python fixtures, never PlatformIO, interop, devices, or network.

Execution record — 2026-09-16

Initial orchestrator-agent execution, under the existing sandbox without elevation (historical first run):

• python3 -B tests/phase9_validation/run.py: 10 tests PASS.
• python3 -B tools/validate_phase9.py --dry-run: exit 0; plan only, no prerequisite or suite validation.
• python3 -B tools/validate_phase9.py: exit 1, 22 PASS / 1 FAIL, no timeouts; about 57 seconds of summed command time. Build and interop explicitly skipped.
• Direct rerun CCACHE_DISABLE=1 python3 -B tests/admin_console_boundary/run.py: reproduced exit 1 at line 40, ValueError: substring not found for ssh.index("static void *ssh_malloc("). Earlier subtest PASS lines do not make the runner pass. The existing runner is outside this change's ownership and was not edited.

Passed commands: security_build_policy, ssh_auth_policy, ssh_auth_transport, hidden_input, ssh_memory, all seven current baseline suites listed above, ssh_management, admin_ssh_policy, web_admin_transport, web_admin_tickets, web_httpd_idle, and all five selected web_cookie_auth modes. Only admin_console_boundary failed.

No firmware build, interop, device, runtime reserve, hardware panic/recovery, release delivery, or whole-Phase-9 acceptance result is claimed for this task.

Follow-up supplied evidence

The owning agent corrected the stale admin_console_boundary source anchor; its subsequent default orchestrator run reports 23/23 PASS. This supersedes the initial 22/23 host result above, without turning its early PASS lines into a successful initial run. Final supplied parent execution now confirms:

• CCACHE_DISABLE=1 python3 -B tools/validate_phase9.py: PASS 23/23.
• tests/phase9_validation/run.py: PASS 10 tests.

No build or interop ran this turn; both remain separate opt-ins, not default execution. Firmware source is unchanged. These are actual parent results supplied for this record, not another execution by this documentation integration.

The supplied notice agent also reports 36 fixture tests PASS and two actual 75-input bundles, each 77 files / 4,433,930 bytes, verified deterministic; see packaging evidence. These are agent measurements, not a new build or bundle measurement by this documentation integration.

The finite advisory search is complete for its declared snapshot, but the PMF-capable SoftAP blocker was not fixed at that historical snapshot. The current fix-bearing bundle is now integrated in the validated candidate; host success does not validate radio behavior, clear distribution obligations or complete Phase 9.