# Phase 9 bounded host validation

`tools/validate_phase9.py` is a thin, sequential orchestrator for existing test
runners, not a new security test implementation or a Phase 9 acceptance gate.
See [security hardening](security_hardening.md) for scope, historical evidence,
and the deferred whole-phase target checklist.

## Usage

From the repository root (the script itself also resolves the root independently
of the caller's working directory):

```sh
python3 -B tools/validate_phase9.py --dry-run
python3 -B tools/validate_phase9.py
python3 -B tools/validate_phase9.py --fail-fast
python3 -B tools/validate_phase9.py --timeout 300
```

Build and OpenSSH interoperability are **independent explicit opt-ins**:

```sh
python3 -B tools/validate_phase9.py --build --build-timeout 600
python3 -B tools/validate_phase9.py --interop --timeout 300
python3 -B tools/validate_phase9.py --build --interop --dry-run
```

These examples are usage, not execution claims. `--build` runs exactly `pio run`
first; it does not upload, erase, monitor, or select another PlatformIO environment.
PlatformIO itself may download dependencies or write its normal external caches.
The caller/parent must choose whether to build and arrange any required permission
separately. No installer, permission grant, network retry, or sandbox escalation
is implemented. A failed/timed-out build skips every dependent host command to
avoid testing stale artifacts as if that build succeeded.

`--interop` adds only `--interop` to `tests/wolfssh_order_contract/run.py` (its
default single matrix). That runner uses OpenSSH and local AF_UNIX sockets, not
an IP host or device. Socket permissions and installed OpenSSH tooling must
already be available; denial is a failure, not a reason to grant permissions.
Neither option is enabled by default. No target address, credentials, arbitrary
command, or device-operation argument is accepted.

## Coverage and prerequisites

The fixed default plan contains **23 commands**; `--dry-run` prints their exact
argv vectors and timeouts without launching anything or checking prerequisites.
There is no discovery-time execution of runners (some run at import/top level).
Paths and options were inspected in their actual `run.py` implementations.

| Scope | Existing runners / modes |
|---|---|
| 9A effective crash policy | `security_build_policy --sdkconfig-header .pio/build/esp32-s3-devkitc-1-n16r8/config/sdkconfig.h` |
| 9B admission / credentials | `ssh_auth_policy`, `ssh_auth_transport`, `hidden_input` |
| 9C allocator | `ssh_memory` |
| Current seven-suite baseline | `sdk_security_overrides --build-dir .pio/build/esp32-s3-devkitc-1-n16r8`, `wolfssh_auth_contract`, `ssh_protocol_policy`, `wolf_crypto_policy`, `wolfssh_parser_contract`, `wolfssh_order_contract`, `release_notices` |
| Focused SSH/admin and HTTPS lifecycle regressions | `ssh_management`, `admin_console_boundary`, `admin_ssh_policy`, `web_admin_transport`, `web_admin_tickets`, `web_httpd_idle` |
| Web early admission, credentials and administrative lifecycle | `web_cookie_auth` default plus separate `--admin`, `--accounts`, `--ssh`, `--lifecycle` commands |

Every runner path is `tests/<name>/run.py`. Auth contract, protocol policy, and
crypto policy receive explicit `--compile-commands
.pio/build/esp32-s3-devkitc-1-n16r8/compile_commands.json` arguments. No
`--host-only`, candidate injection, or silent fallback is used. Default operation
requires an existing configured build, installed pinned SDK/managed sources,
the production toolchain and host compiler/libraries needed by these runners
(including OpenSSL development support for relevant tests). It **does not run
PlatformIO** to satisfy those requirements. Missing explicit files/executables
are `PREREQ` failures; deeper source/toolchain prerequisites are checked by the
existing runners and surface as nonzero `FAIL` results. Existing artifacts are
not proof that the current checkout has been freshly built.

The current cookie-auth runner has no `--admission` branch: the default exercises
early admission/wiping, and each selected domain mode repeats that baseline.
Unrelated display/network/UI/performance suites are intentionally excluded.
`ssh_memory` retains its documented optional installed-IDF extent-source check
as skipped (no `--idf-path` supplied); its normal allocator contracts still run.
The ordering runner's candidate `--target-contracts`, separate `--pio-adapter`
regression, and repeated interop matrices are not added implicitly. Run those
separately when their particular evidence is needed. This is a focused full
Phase 9 host regression set, not every test in the repository.

## Bounds, output, and result semantics

- Each host command has a 180-second wall-clock timeout by default; the optional
  build gets 600 seconds. Both timeout flags accept finite values in `(0, 3600]`.
  The fixed sequential plan bounds total child execution by the sum of its
  timeouts (default 4,140 seconds), plus launch/reporting/cleanup overhead.
- POSIX process groups are required. Timeout or Ctrl-C kills the active process
  group and reaps the runner; successful/failed exit also retires any remaining
  group descendants. This covers normal compiler/test children, not deliberately
  detached processes. Group reaping has a five-second wait bound.
- Commands use argv lists, no shell evaluation or interpolation, with stdin
  closed. The child environment is copied intact except `CCACHE_DISABLE=1`;
  arbitrary compiler, SDK, proxy and other caller environment settings are not
  overwritten. The parent's environment is not mutated.
- stdout/stderr are inherited live. The orchestrator creates no persistent log,
  JSON capture, environment dump, or secret artifact. Existing runners retain
  their own temporary fixture/build behavior; forced termination can bypass
  their temporary-directory cleanup. Console output is **not a redaction
  guarantee**: use synthetic fixtures, review before sharing, and do not redirect
  output into routine secret-bearing persistent captures. Build outputs remain
  normal PlatformIO artifacts only when explicitly requested.
- `PASS` means the entire runner exited zero, not that an earlier printed PASS
  line appeared. The final summary records each command's exit, timeout or
  missing prerequisite. Defaults collect failures and continue independent
  commands; `--fail-fast` reports remaining commands as `SKIP`. Build failure
  always stops dependent commands. No selected suite is silently dropped.
- Exit 0 means all selected commands passed, or a dry-run plan was printed.
  Dry-run uses `PLAN`, never `PASS`, and is not validation. Failures, timeouts,
  prerequisites and dependent skips return 1; invalid CLI arguments return 2;
  Ctrl-C returns 130. Optional build/interop omissions are announced separately
  and do not turn a successful host run into failure.

## Orchestrator fixture tests

```sh
python3 -B tests/phase9_validation/run.py
```

Ten offline unittest groups check the exact plan/paths/strict arguments, independent
opt-ins, timeout configuration and invalid values, dry-run nonexecution, default
build/interop omission, failure collection, fail-fast and failed-build skips,
missing file/executable handling, literal argv/environment preservation,
timeout/descendant termination, and inherited streams/closed stdin. They execute
only small Python fixtures, never PlatformIO, interop, devices, or network.

## Execution record — 2026-09-16

Initial orchestrator-agent execution, under the existing sandbox without elevation (historical first run):

- `python3 -B tests/phase9_validation/run.py`: **10 tests PASS**.
- `python3 -B tools/validate_phase9.py --dry-run`: exit 0; **plan only**, no
  prerequisite or suite validation.
- `python3 -B tools/validate_phase9.py`: **exit 1, 22 PASS / 1 FAIL**, no timeouts;
  about 57 seconds of summed command time. Build and interop explicitly skipped.
- Direct rerun `CCACHE_DISABLE=1 python3 -B tests/admin_console_boundary/run.py`:
  reproduced exit 1 at line 40, `ValueError: substring not found` for
  `ssh.index("static void *ssh_malloc(")`. Earlier subtest PASS lines do not make
  the runner pass. The existing runner is outside this change's ownership and
  was not edited.

Passed commands: `security_build_policy`, `ssh_auth_policy`, `ssh_auth_transport`,
`hidden_input`, `ssh_memory`, **all seven current baseline suites listed above**,
`ssh_management`, `admin_ssh_policy`, `web_admin_transport`, `web_admin_tickets`,
`web_httpd_idle`, and all five selected `web_cookie_auth` modes.
Only `admin_console_boundary` failed.

No firmware build, interop, device, runtime reserve, hardware panic/recovery,
release delivery, or whole-Phase-9 acceptance result is claimed for this task.

### Follow-up supplied evidence

The owning agent corrected the stale `admin_console_boundary` source anchor;
its subsequent default orchestrator run reports **23/23 PASS**. This supersedes
the initial 22/23 host result above, without turning its early PASS lines into
a successful initial run. Final supplied parent execution now confirms:

- `CCACHE_DISABLE=1 python3 -B tools/validate_phase9.py`: **PASS 23/23**.
- `tests/phase9_validation/run.py`: **PASS 10 tests**.

No build or interop ran this turn; both remain separate opt-ins, not default
execution. Firmware source is unchanged. These are actual parent results supplied
for this record, not another execution by this documentation integration.

The supplied notice agent also reports **36 fixture tests PASS** and two actual
75-input bundles, each **77 files / 4,433,930 bytes**, verified deterministic; see
[packaging evidence](release_packaging.md). These are agent measurements, not a
new build or bundle measurement by this documentation integration.

The [finite advisory search](dependency_advisory_coverage.md) is complete for its
declared snapshot, but the [PMF-capable SoftAP blocker](wifi_security_update_plan.md)
is **not fixed**. Host success does not repair that vendor bundle, validate radio
behavior, clear distribution obligations or complete Phase 9.