Add Phase 9 validation and advisory review
Record the finite dependency search, Wi-Fi maintenance blocker, and pinned icon provenance. Add bounded host orchestration and fixture coverage, and update release documentation with current evidence.
This commit is contained in:
@@ -0,0 +1,172 @@
|
||||
# Phase 9 bounded host validation
|
||||
|
||||
`tools/validate_phase9.py` is a thin, sequential orchestrator for existing test
|
||||
runners, not a new security test implementation or a Phase 9 acceptance gate.
|
||||
See [security hardening](security_hardening.md) for scope, historical evidence,
|
||||
and the deferred whole-phase target checklist.
|
||||
|
||||
## Usage
|
||||
|
||||
From the repository root (the script itself also resolves the root independently
|
||||
of the caller's working directory):
|
||||
|
||||
```sh
|
||||
python3 -B tools/validate_phase9.py --dry-run
|
||||
python3 -B tools/validate_phase9.py
|
||||
python3 -B tools/validate_phase9.py --fail-fast
|
||||
python3 -B tools/validate_phase9.py --timeout 300
|
||||
```
|
||||
|
||||
Build and OpenSSH interoperability are **independent explicit opt-ins**:
|
||||
|
||||
```sh
|
||||
python3 -B tools/validate_phase9.py --build --build-timeout 600
|
||||
python3 -B tools/validate_phase9.py --interop --timeout 300
|
||||
python3 -B tools/validate_phase9.py --build --interop --dry-run
|
||||
```
|
||||
|
||||
These examples are usage, not execution claims. `--build` runs exactly `pio run`
|
||||
first; it does not upload, erase, monitor, or select another PlatformIO environment.
|
||||
PlatformIO itself may download dependencies or write its normal external caches.
|
||||
The caller/parent must choose whether to build and arrange any required permission
|
||||
separately. No installer, permission grant, network retry, or sandbox escalation
|
||||
is implemented. A failed/timed-out build skips every dependent host command to
|
||||
avoid testing stale artifacts as if that build succeeded.
|
||||
|
||||
`--interop` adds only `--interop` to `tests/wolfssh_order_contract/run.py` (its
|
||||
default single matrix). That runner uses OpenSSH and local AF_UNIX sockets, not
|
||||
an IP host or device. Socket permissions and installed OpenSSH tooling must
|
||||
already be available; denial is a failure, not a reason to grant permissions.
|
||||
Neither option is enabled by default. No target address, credentials, arbitrary
|
||||
command, or device-operation argument is accepted.
|
||||
|
||||
## Coverage and prerequisites
|
||||
|
||||
The fixed default plan contains **23 commands**; `--dry-run` prints their exact
|
||||
argv vectors and timeouts without launching anything or checking prerequisites.
|
||||
There is no discovery-time execution of runners (some run at import/top level).
|
||||
Paths and options were inspected in their actual `run.py` implementations.
|
||||
|
||||
| Scope | Existing runners / modes |
|
||||
|---|---|
|
||||
| 9A effective crash policy | `security_build_policy --sdkconfig-header .pio/build/esp32-s3-devkitc-1-n16r8/config/sdkconfig.h` |
|
||||
| 9B admission / credentials | `ssh_auth_policy`, `ssh_auth_transport`, `hidden_input` |
|
||||
| 9C allocator | `ssh_memory` |
|
||||
| Current seven-suite baseline | `sdk_security_overrides --build-dir .pio/build/esp32-s3-devkitc-1-n16r8`, `wolfssh_auth_contract`, `ssh_protocol_policy`, `wolf_crypto_policy`, `wolfssh_parser_contract`, `wolfssh_order_contract`, `release_notices` |
|
||||
| Focused SSH/admin and HTTPS lifecycle regressions | `ssh_management`, `admin_console_boundary`, `admin_ssh_policy`, `web_admin_transport`, `web_admin_tickets`, `web_httpd_idle` |
|
||||
| Web early admission, credentials and administrative lifecycle | `web_cookie_auth` default plus separate `--admin`, `--accounts`, `--ssh`, `--lifecycle` commands |
|
||||
|
||||
Every runner path is `tests/<name>/run.py`. Auth contract, protocol policy, and
|
||||
crypto policy receive explicit `--compile-commands
|
||||
.pio/build/esp32-s3-devkitc-1-n16r8/compile_commands.json` arguments. No
|
||||
`--host-only`, candidate injection, or silent fallback is used. Default operation
|
||||
requires an existing configured build, installed pinned SDK/managed sources,
|
||||
the production toolchain and host compiler/libraries needed by these runners
|
||||
(including OpenSSL development support for relevant tests). It **does not run
|
||||
PlatformIO** to satisfy those requirements. Missing explicit files/executables
|
||||
are `PREREQ` failures; deeper source/toolchain prerequisites are checked by the
|
||||
existing runners and surface as nonzero `FAIL` results. Existing artifacts are
|
||||
not proof that the current checkout has been freshly built.
|
||||
|
||||
The current cookie-auth runner has no `--admission` branch: the default exercises
|
||||
early admission/wiping, and each selected domain mode repeats that baseline.
|
||||
Unrelated display/network/UI/performance suites are intentionally excluded.
|
||||
`ssh_memory` retains its documented optional installed-IDF extent-source check
|
||||
as skipped (no `--idf-path` supplied); its normal allocator contracts still run.
|
||||
The ordering runner's candidate `--target-contracts`, separate `--pio-adapter`
|
||||
regression, and repeated interop matrices are not added implicitly. Run those
|
||||
separately when their particular evidence is needed. This is a focused full
|
||||
Phase 9 host regression set, not every test in the repository.
|
||||
|
||||
## Bounds, output, and result semantics
|
||||
|
||||
- Each host command has a 180-second wall-clock timeout by default; the optional
|
||||
build gets 600 seconds. Both timeout flags accept finite values in `(0, 3600]`.
|
||||
The fixed sequential plan bounds total child execution by the sum of its
|
||||
timeouts (default 4,140 seconds), plus launch/reporting/cleanup overhead.
|
||||
- POSIX process groups are required. Timeout or Ctrl-C kills the active process
|
||||
group and reaps the runner; successful/failed exit also retires any remaining
|
||||
group descendants. This covers normal compiler/test children, not deliberately
|
||||
detached processes. Group reaping has a five-second wait bound.
|
||||
- Commands use argv lists, no shell evaluation or interpolation, with stdin
|
||||
closed. The child environment is copied intact except `CCACHE_DISABLE=1`;
|
||||
arbitrary compiler, SDK, proxy and other caller environment settings are not
|
||||
overwritten. The parent's environment is not mutated.
|
||||
- stdout/stderr are inherited live. The orchestrator creates no persistent log,
|
||||
JSON capture, environment dump, or secret artifact. Existing runners retain
|
||||
their own temporary fixture/build behavior; forced termination can bypass
|
||||
their temporary-directory cleanup. Console output is **not a redaction
|
||||
guarantee**: use synthetic fixtures, review before sharing, and do not redirect
|
||||
output into routine secret-bearing persistent captures. Build outputs remain
|
||||
normal PlatformIO artifacts only when explicitly requested.
|
||||
- `PASS` means the entire runner exited zero, not that an earlier printed PASS
|
||||
line appeared. The final summary records each command's exit, timeout or
|
||||
missing prerequisite. Defaults collect failures and continue independent
|
||||
commands; `--fail-fast` reports remaining commands as `SKIP`. Build failure
|
||||
always stops dependent commands. No selected suite is silently dropped.
|
||||
- Exit 0 means all selected commands passed, or a dry-run plan was printed.
|
||||
Dry-run uses `PLAN`, never `PASS`, and is not validation. Failures, timeouts,
|
||||
prerequisites and dependent skips return 1; invalid CLI arguments return 2;
|
||||
Ctrl-C returns 130. Optional build/interop omissions are announced separately
|
||||
and do not turn a successful host run into failure.
|
||||
|
||||
## Orchestrator fixture tests
|
||||
|
||||
```sh
|
||||
python3 -B tests/phase9_validation/run.py
|
||||
```
|
||||
|
||||
Ten offline unittest groups check the exact plan/paths/strict arguments, independent
|
||||
opt-ins, timeout configuration and invalid values, dry-run nonexecution, default
|
||||
build/interop omission, failure collection, fail-fast and failed-build skips,
|
||||
missing file/executable handling, literal argv/environment preservation,
|
||||
timeout/descendant termination, and inherited streams/closed stdin. They execute
|
||||
only small Python fixtures, never PlatformIO, interop, devices, or network.
|
||||
|
||||
## Execution record — 2026-09-16
|
||||
|
||||
Initial orchestrator-agent execution, under the existing sandbox without elevation (historical first run):
|
||||
|
||||
- `python3 -B tests/phase9_validation/run.py`: **10 tests PASS**.
|
||||
- `python3 -B tools/validate_phase9.py --dry-run`: exit 0; **plan only**, no
|
||||
prerequisite or suite validation.
|
||||
- `python3 -B tools/validate_phase9.py`: **exit 1, 22 PASS / 1 FAIL**, no timeouts;
|
||||
about 57 seconds of summed command time. Build and interop explicitly skipped.
|
||||
- Direct rerun `CCACHE_DISABLE=1 python3 -B tests/admin_console_boundary/run.py`:
|
||||
reproduced exit 1 at line 40, `ValueError: substring not found` for
|
||||
`ssh.index("static void *ssh_malloc(")`. Earlier subtest PASS lines do not make
|
||||
the runner pass. The existing runner is outside this change's ownership and
|
||||
was not edited.
|
||||
|
||||
Passed commands: `security_build_policy`, `ssh_auth_policy`, `ssh_auth_transport`,
|
||||
`hidden_input`, `ssh_memory`, **all seven current baseline suites listed above**,
|
||||
`ssh_management`, `admin_ssh_policy`, `web_admin_transport`, `web_admin_tickets`,
|
||||
`web_httpd_idle`, and all five selected `web_cookie_auth` modes.
|
||||
Only `admin_console_boundary` failed.
|
||||
|
||||
No firmware build, interop, device, runtime reserve, hardware panic/recovery,
|
||||
release delivery, or whole-Phase-9 acceptance result is claimed for this task.
|
||||
|
||||
### Follow-up supplied evidence
|
||||
|
||||
The owning agent corrected the stale `admin_console_boundary` source anchor;
|
||||
its subsequent default orchestrator run reports **23/23 PASS**. This supersedes
|
||||
the initial 22/23 host result above, without turning its early PASS lines into
|
||||
a successful initial run. Final supplied parent execution now confirms:
|
||||
|
||||
- `CCACHE_DISABLE=1 python3 -B tools/validate_phase9.py`: **PASS 23/23**.
|
||||
- `tests/phase9_validation/run.py`: **PASS 10 tests**.
|
||||
|
||||
No build or interop ran this turn; both remain separate opt-ins, not default
|
||||
execution. Firmware source is unchanged. These are actual parent results supplied
|
||||
for this record, not another execution by this documentation integration.
|
||||
|
||||
The supplied notice agent also reports **36 fixture tests PASS** and two actual
|
||||
75-input bundles, each **77 files / 4,433,930 bytes**, verified deterministic; see
|
||||
[packaging evidence](release_packaging.md). These are agent measurements, not a
|
||||
new build or bundle measurement by this documentation integration.
|
||||
|
||||
The [finite advisory search](dependency_advisory_coverage.md) is complete for its
|
||||
declared snapshot, but the [PMF-capable SoftAP blocker](wifi_security_update_plan.md)
|
||||
is **not fixed**. Host success does not repair that vendor bundle, validate radio
|
||||
behavior, clear distribution obligations or complete Phase 9.
|
||||
Reference in New Issue
Block a user