Commander1024
797d2681ac
Pin PlatformIO packages and toolchains, rebase protected SDK overrides, and add WebSocket receive regression coverage. Document isolated candidate validation, archive provenance, and remaining gates.
186 lines
12 KiB
Markdown
186 lines
12 KiB
Markdown
# Phase 9 bounded host validation
|
||
|
||
`tools/validate_phase9.py` is a thin, sequential orchestrator for existing test
|
||
runners, not a new security test implementation or a Phase 9 acceptance gate.
|
||
See [security hardening](security_hardening.md) for scope, historical evidence,
|
||
and the deferred whole-phase target checklist.
|
||
|
||
## Current candidate execution — 2026-09-18
|
||
|
||
Supplied final parent command, run from the repository root:
|
||
|
||
```sh
|
||
CCACHE_DISABLE=1 python3 -B tools/validate_phase9.py --build-dir .pio/idf-candidate-5.5.3/app-validated/.pio/build/esp32-s3-devkitc-1-n16r8 --idf-path .pio/idf-candidate-5.5.3/core/packages/framework-espidf --platformio-core-dir .pio/idf-candidate-5.5.3/core --interop --web-performance
|
||
```
|
||
|
||
**PASS 24/24.** This selects the actual fresh isolated build, SDK and toolchain core; it is not a default-root-build execution. The separate fresh `.pio/idf-candidate-5.5.3/app-validated` build **PASS** reports **95,552 B linked RAM / 1,749,493 B flash**, versus historical **94,340 / 1,768,901 B** (**+1,212 B RAM / −19,408 B flash**). The default parent `pio run` timed out after **200 seconds during installation, before compilation**; no normal root build PASS is claimed.
|
||
|
||
Pre/post source equality: **3,237 files**, SHA-256 **`3a1af78c15cfdd02da1055a8957b4086f9018862b7aa1c7c52fd2401a1a0a031`**. Actual generated-input registration covers **nine C sources plus one forced header**. WS receive tests exercise generated code: **982 cases / 10 mutation checks**, including the five signed `sizeof` corrections. The stale web-cookie fixture asserting IDF 5.5.0 was corrected, not bypassed. Historical stale-WS compilation/coverage failures in the [rebase review](idf_553_rebase_review.md) are resolved by this final snapshot, not hidden or retroactively called passes.
|
||
|
||
The [fix-bearing Wi-Fi bundle](wifi_security_update_plan.md) is integrated in the candidate, with unchanged PMF/WPA3; radio-hardware vulnerability closure and full target/resource/recovery gates remain pending. Nine notice catalog entries were semantically rebased, the other 66 unchanged (75 total); supplied notice evidence is **36 fixtures PASS**, two actual deterministic bundles each **77 files / 4,433,930 bytes**. Archive pins and source equality do not prove complete immutable root/ancillary/Python dependency closure, legal clearance or Phase 9 acceptance. This documentation update records supplied parent evidence; it did not rerun these builds, suites, bundles or hardware tests.
|
||
|
||
## Usage
|
||
|
||
From the repository root (the script itself also resolves the root independently
|
||
of the caller's working directory):
|
||
|
||
```sh
|
||
python3 -B tools/validate_phase9.py --dry-run
|
||
python3 -B tools/validate_phase9.py
|
||
python3 -B tools/validate_phase9.py --fail-fast
|
||
python3 -B tools/validate_phase9.py --timeout 300
|
||
```
|
||
|
||
Build and OpenSSH interoperability are **independent explicit opt-ins**:
|
||
|
||
```sh
|
||
python3 -B tools/validate_phase9.py --build --build-timeout 600
|
||
python3 -B tools/validate_phase9.py --interop --timeout 300
|
||
python3 -B tools/validate_phase9.py --build --interop --dry-run
|
||
```
|
||
|
||
These examples are usage, not execution claims. `--build` runs exactly `pio run`
|
||
first; it does not upload, erase, monitor, or select another PlatformIO environment.
|
||
PlatformIO itself may download dependencies or write its normal external caches.
|
||
The caller/parent must choose whether to build and arrange any required permission
|
||
separately. No installer, permission grant, network retry, or sandbox escalation
|
||
is implemented. A failed/timed-out build skips every dependent host command to
|
||
avoid testing stale artifacts as if that build succeeded.
|
||
|
||
`--interop` adds only `--interop` to `tests/wolfssh_order_contract/run.py` (its
|
||
default single matrix). That runner uses OpenSSH and local AF_UNIX sockets, not
|
||
an IP host or device. Socket permissions and installed OpenSSH tooling must
|
||
already be available; denial is a failure, not a reason to grant permissions.
|
||
Neither option is enabled by default. `--web-performance` independently adds the generated-WS performance runner (24 commands total). `--build-dir`, `--idf-path` and `--platformio-core-dir` select existing candidate artifacts/SDK/core; `--build` rejects a non-default build directory, which must be built separately. No target address, credentials, arbitrary
|
||
command, or device-operation argument is accepted.
|
||
|
||
## Coverage and prerequisites
|
||
|
||
The fixed default plan contains **23 commands**; `--dry-run` prints their exact
|
||
argv vectors and timeouts without launching anything or checking prerequisites.
|
||
There is no discovery-time execution of runners (some run at import/top level).
|
||
Paths and options were inspected in their actual `run.py` implementations.
|
||
|
||
| Scope | Existing runners / modes |
|
||
|---|---|
|
||
| 9A effective crash policy | `security_build_policy --sdkconfig-header .pio/build/esp32-s3-devkitc-1-n16r8/config/sdkconfig.h` |
|
||
| 9B admission / credentials | `ssh_auth_policy`, `ssh_auth_transport`, `hidden_input` |
|
||
| 9C allocator | `ssh_memory` |
|
||
| Current seven-suite baseline | `sdk_security_overrides --build-dir .pio/build/esp32-s3-devkitc-1-n16r8`, `wolfssh_auth_contract`, `ssh_protocol_policy`, `wolf_crypto_policy`, `wolfssh_parser_contract`, `wolfssh_order_contract`, `release_notices` |
|
||
| Focused SSH/admin and HTTPS lifecycle regressions | `ssh_management`, `admin_console_boundary`, `admin_ssh_policy`, `web_admin_transport`, `web_admin_tickets`, `web_httpd_idle` |
|
||
| Web early admission, credentials and administrative lifecycle | `web_cookie_auth` default plus separate `--admin`, `--accounts`, `--ssh`, `--lifecycle` commands |
|
||
|
||
Every runner path is `tests/<name>/run.py`. Auth contract, protocol policy, and
|
||
crypto policy receive explicit `--compile-commands
|
||
.pio/build/esp32-s3-devkitc-1-n16r8/compile_commands.json` arguments. No
|
||
`--host-only`, candidate injection, or silent fallback is used. Default operation
|
||
requires an existing configured build, installed pinned SDK/managed sources,
|
||
the production toolchain and host compiler/libraries needed by these runners
|
||
(including OpenSSL development support for relevant tests). It **does not run
|
||
PlatformIO** to satisfy those requirements. Missing explicit files/executables
|
||
are `PREREQ` failures; deeper source/toolchain prerequisites are checked by the
|
||
existing runners and surface as nonzero `FAIL` results. Existing artifacts are
|
||
not proof that the current checkout has been freshly built.
|
||
|
||
The current cookie-auth runner has no `--admission` branch: the default exercises
|
||
early admission/wiping, and each selected domain mode repeats that baseline.
|
||
Unrelated display/network/UI suites are intentionally excluded; web performance is an explicit opt-in.
|
||
`ssh_memory` retains its documented optional installed-IDF extent-source check
|
||
as skipped when no `--idf-path` is supplied; explicit candidate selection enables it. Its normal allocator contracts still run.
|
||
The ordering runner's candidate `--target-contracts`, separate `--pio-adapter`
|
||
regression, and repeated interop matrices are not added implicitly. Run those
|
||
separately when their particular evidence is needed. This is a focused full
|
||
Phase 9 host regression set, not every test in the repository.
|
||
|
||
## Bounds, output, and result semantics
|
||
|
||
- Each host command has a 180-second wall-clock timeout by default; the optional
|
||
build gets 600 seconds. Both timeout flags accept finite values in `(0, 3600]`.
|
||
The fixed sequential plan bounds total child execution by the sum of its
|
||
timeouts (default 4,140 seconds), plus launch/reporting/cleanup overhead.
|
||
- POSIX process groups are required. Timeout or Ctrl-C kills the active process
|
||
group and reaps the runner; successful/failed exit also retires any remaining
|
||
group descendants. This covers normal compiler/test children, not deliberately
|
||
detached processes. Group reaping has a five-second wait bound.
|
||
- Commands use argv lists, no shell evaluation or interpolation, with stdin
|
||
closed. The child environment is copied with `CCACHE_DISABLE=1`; explicit `--idf-path` and `--platformio-core-dir` also set resolved `IDF_PATH` and `PLATFORMIO_CORE_DIR` for children;
|
||
arbitrary compiler, SDK, proxy and other caller environment settings are not
|
||
overwritten. The parent's environment is not mutated.
|
||
- stdout/stderr are inherited live. The orchestrator creates no persistent log,
|
||
JSON capture, environment dump, or secret artifact. Existing runners retain
|
||
their own temporary fixture/build behavior; forced termination can bypass
|
||
their temporary-directory cleanup. Console output is **not a redaction
|
||
guarantee**: use synthetic fixtures, review before sharing, and do not redirect
|
||
output into routine secret-bearing persistent captures. Build outputs remain
|
||
normal PlatformIO artifacts only when explicitly requested.
|
||
- `PASS` means the entire runner exited zero, not that an earlier printed PASS
|
||
line appeared. The final summary records each command's exit, timeout or
|
||
missing prerequisite. Defaults collect failures and continue independent
|
||
commands; `--fail-fast` reports remaining commands as `SKIP`. Build failure
|
||
always stops dependent commands. No selected suite is silently dropped.
|
||
- Exit 0 means all selected commands passed, or a dry-run plan was printed.
|
||
Dry-run uses `PLAN`, never `PASS`, and is not validation. Failures, timeouts,
|
||
prerequisites and dependent skips return 1; invalid CLI arguments return 2;
|
||
Ctrl-C returns 130. Optional build/interop omissions are announced separately
|
||
and do not turn a successful host run into failure.
|
||
|
||
## Orchestrator fixture tests
|
||
|
||
```sh
|
||
python3 -B tests/phase9_validation/run.py
|
||
```
|
||
|
||
Offline unittest groups check the exact plan/paths/strict arguments, independent
|
||
opt-ins, timeout configuration and invalid values, dry-run nonexecution, default
|
||
build/interop omission, failure collection, fail-fast and failed-build skips,
|
||
missing file/executable handling, literal argv/environment preservation,
|
||
timeout/descendant termination, and inherited streams/closed stdin. They execute
|
||
only small Python fixtures, never PlatformIO, interop, devices, or network.
|
||
|
||
## Execution record — 2026-09-16
|
||
|
||
Initial orchestrator-agent execution, under the existing sandbox without elevation (historical first run):
|
||
|
||
- `python3 -B tests/phase9_validation/run.py`: **10 tests PASS**.
|
||
- `python3 -B tools/validate_phase9.py --dry-run`: exit 0; **plan only**, no
|
||
prerequisite or suite validation.
|
||
- `python3 -B tools/validate_phase9.py`: **exit 1, 22 PASS / 1 FAIL**, no timeouts;
|
||
about 57 seconds of summed command time. Build and interop explicitly skipped.
|
||
- Direct rerun `CCACHE_DISABLE=1 python3 -B tests/admin_console_boundary/run.py`:
|
||
reproduced exit 1 at line 40, `ValueError: substring not found` for
|
||
`ssh.index("static void *ssh_malloc(")`. Earlier subtest PASS lines do not make
|
||
the runner pass. The existing runner is outside this change's ownership and
|
||
was not edited.
|
||
|
||
Passed commands: `security_build_policy`, `ssh_auth_policy`, `ssh_auth_transport`,
|
||
`hidden_input`, `ssh_memory`, **all seven current baseline suites listed above**,
|
||
`ssh_management`, `admin_ssh_policy`, `web_admin_transport`, `web_admin_tickets`,
|
||
`web_httpd_idle`, and all five selected `web_cookie_auth` modes.
|
||
Only `admin_console_boundary` failed.
|
||
|
||
No firmware build, interop, device, runtime reserve, hardware panic/recovery,
|
||
release delivery, or whole-Phase-9 acceptance result is claimed for this task.
|
||
|
||
### Follow-up supplied evidence
|
||
|
||
The owning agent corrected the stale `admin_console_boundary` source anchor;
|
||
its subsequent default orchestrator run reports **23/23 PASS**. This supersedes
|
||
the initial 22/23 host result above, without turning its early PASS lines into
|
||
a successful initial run. Final supplied parent execution now confirms:
|
||
|
||
- `CCACHE_DISABLE=1 python3 -B tools/validate_phase9.py`: **PASS 23/23**.
|
||
- `tests/phase9_validation/run.py`: **PASS 10 tests**.
|
||
|
||
No build or interop ran this turn; both remain separate opt-ins, not default
|
||
execution. Firmware source is unchanged. These are actual parent results supplied
|
||
for this record, not another execution by this documentation integration.
|
||
|
||
The supplied notice agent also reports **36 fixture tests PASS** and two actual
|
||
75-input bundles, each **77 files / 4,433,930 bytes**, verified deterministic; see
|
||
[packaging evidence](release_packaging.md). These are agent measurements, not a
|
||
new build or bundle measurement by this documentation integration.
|
||
|
||
The [finite advisory search](dependency_advisory_coverage.md) is complete for its
|
||
declared snapshot, but the [PMF-capable SoftAP blocker](wifi_security_update_plan.md)
|
||
was **not fixed at that historical snapshot**. The current fix-bearing bundle is now integrated in the validated candidate; host success does not validate radio behavior, clear distribution obligations or complete Phase 9.
|