Commander1024
cdc4d4a8df
Record the finite dependency search, Wi-Fi maintenance blocker, and pinned icon provenance. Add bounded host orchestration and fixture coverage, and update release documentation with current evidence.
173 lines
9.9 KiB
Markdown
173 lines
9.9 KiB
Markdown
# Phase 9 bounded host validation
|
|
|
|
`tools/validate_phase9.py` is a thin, sequential orchestrator for existing test
|
|
runners, not a new security test implementation or a Phase 9 acceptance gate.
|
|
See [security hardening](security_hardening.md) for scope, historical evidence,
|
|
and the deferred whole-phase target checklist.
|
|
|
|
## Usage
|
|
|
|
From the repository root (the script itself also resolves the root independently
|
|
of the caller's working directory):
|
|
|
|
```sh
|
|
python3 -B tools/validate_phase9.py --dry-run
|
|
python3 -B tools/validate_phase9.py
|
|
python3 -B tools/validate_phase9.py --fail-fast
|
|
python3 -B tools/validate_phase9.py --timeout 300
|
|
```
|
|
|
|
Build and OpenSSH interoperability are **independent explicit opt-ins**:
|
|
|
|
```sh
|
|
python3 -B tools/validate_phase9.py --build --build-timeout 600
|
|
python3 -B tools/validate_phase9.py --interop --timeout 300
|
|
python3 -B tools/validate_phase9.py --build --interop --dry-run
|
|
```
|
|
|
|
These examples are usage, not execution claims. `--build` runs exactly `pio run`
|
|
first; it does not upload, erase, monitor, or select another PlatformIO environment.
|
|
PlatformIO itself may download dependencies or write its normal external caches.
|
|
The caller/parent must choose whether to build and arrange any required permission
|
|
separately. No installer, permission grant, network retry, or sandbox escalation
|
|
is implemented. A failed/timed-out build skips every dependent host command to
|
|
avoid testing stale artifacts as if that build succeeded.
|
|
|
|
`--interop` adds only `--interop` to `tests/wolfssh_order_contract/run.py` (its
|
|
default single matrix). That runner uses OpenSSH and local AF_UNIX sockets, not
|
|
an IP host or device. Socket permissions and installed OpenSSH tooling must
|
|
already be available; denial is a failure, not a reason to grant permissions.
|
|
Neither option is enabled by default. No target address, credentials, arbitrary
|
|
command, or device-operation argument is accepted.
|
|
|
|
## Coverage and prerequisites
|
|
|
|
The fixed default plan contains **23 commands**; `--dry-run` prints their exact
|
|
argv vectors and timeouts without launching anything or checking prerequisites.
|
|
There is no discovery-time execution of runners (some run at import/top level).
|
|
Paths and options were inspected in their actual `run.py` implementations.
|
|
|
|
| Scope | Existing runners / modes |
|
|
|---|---|
|
|
| 9A effective crash policy | `security_build_policy --sdkconfig-header .pio/build/esp32-s3-devkitc-1-n16r8/config/sdkconfig.h` |
|
|
| 9B admission / credentials | `ssh_auth_policy`, `ssh_auth_transport`, `hidden_input` |
|
|
| 9C allocator | `ssh_memory` |
|
|
| Current seven-suite baseline | `sdk_security_overrides --build-dir .pio/build/esp32-s3-devkitc-1-n16r8`, `wolfssh_auth_contract`, `ssh_protocol_policy`, `wolf_crypto_policy`, `wolfssh_parser_contract`, `wolfssh_order_contract`, `release_notices` |
|
|
| Focused SSH/admin and HTTPS lifecycle regressions | `ssh_management`, `admin_console_boundary`, `admin_ssh_policy`, `web_admin_transport`, `web_admin_tickets`, `web_httpd_idle` |
|
|
| Web early admission, credentials and administrative lifecycle | `web_cookie_auth` default plus separate `--admin`, `--accounts`, `--ssh`, `--lifecycle` commands |
|
|
|
|
Every runner path is `tests/<name>/run.py`. Auth contract, protocol policy, and
|
|
crypto policy receive explicit `--compile-commands
|
|
.pio/build/esp32-s3-devkitc-1-n16r8/compile_commands.json` arguments. No
|
|
`--host-only`, candidate injection, or silent fallback is used. Default operation
|
|
requires an existing configured build, installed pinned SDK/managed sources,
|
|
the production toolchain and host compiler/libraries needed by these runners
|
|
(including OpenSSL development support for relevant tests). It **does not run
|
|
PlatformIO** to satisfy those requirements. Missing explicit files/executables
|
|
are `PREREQ` failures; deeper source/toolchain prerequisites are checked by the
|
|
existing runners and surface as nonzero `FAIL` results. Existing artifacts are
|
|
not proof that the current checkout has been freshly built.
|
|
|
|
The current cookie-auth runner has no `--admission` branch: the default exercises
|
|
early admission/wiping, and each selected domain mode repeats that baseline.
|
|
Unrelated display/network/UI/performance suites are intentionally excluded.
|
|
`ssh_memory` retains its documented optional installed-IDF extent-source check
|
|
as skipped (no `--idf-path` supplied); its normal allocator contracts still run.
|
|
The ordering runner's candidate `--target-contracts`, separate `--pio-adapter`
|
|
regression, and repeated interop matrices are not added implicitly. Run those
|
|
separately when their particular evidence is needed. This is a focused full
|
|
Phase 9 host regression set, not every test in the repository.
|
|
|
|
## Bounds, output, and result semantics
|
|
|
|
- Each host command has a 180-second wall-clock timeout by default; the optional
|
|
build gets 600 seconds. Both timeout flags accept finite values in `(0, 3600]`.
|
|
The fixed sequential plan bounds total child execution by the sum of its
|
|
timeouts (default 4,140 seconds), plus launch/reporting/cleanup overhead.
|
|
- POSIX process groups are required. Timeout or Ctrl-C kills the active process
|
|
group and reaps the runner; successful/failed exit also retires any remaining
|
|
group descendants. This covers normal compiler/test children, not deliberately
|
|
detached processes. Group reaping has a five-second wait bound.
|
|
- Commands use argv lists, no shell evaluation or interpolation, with stdin
|
|
closed. The child environment is copied intact except `CCACHE_DISABLE=1`;
|
|
arbitrary compiler, SDK, proxy and other caller environment settings are not
|
|
overwritten. The parent's environment is not mutated.
|
|
- stdout/stderr are inherited live. The orchestrator creates no persistent log,
|
|
JSON capture, environment dump, or secret artifact. Existing runners retain
|
|
their own temporary fixture/build behavior; forced termination can bypass
|
|
their temporary-directory cleanup. Console output is **not a redaction
|
|
guarantee**: use synthetic fixtures, review before sharing, and do not redirect
|
|
output into routine secret-bearing persistent captures. Build outputs remain
|
|
normal PlatformIO artifacts only when explicitly requested.
|
|
- `PASS` means the entire runner exited zero, not that an earlier printed PASS
|
|
line appeared. The final summary records each command's exit, timeout or
|
|
missing prerequisite. Defaults collect failures and continue independent
|
|
commands; `--fail-fast` reports remaining commands as `SKIP`. Build failure
|
|
always stops dependent commands. No selected suite is silently dropped.
|
|
- Exit 0 means all selected commands passed, or a dry-run plan was printed.
|
|
Dry-run uses `PLAN`, never `PASS`, and is not validation. Failures, timeouts,
|
|
prerequisites and dependent skips return 1; invalid CLI arguments return 2;
|
|
Ctrl-C returns 130. Optional build/interop omissions are announced separately
|
|
and do not turn a successful host run into failure.
|
|
|
|
## Orchestrator fixture tests
|
|
|
|
```sh
|
|
python3 -B tests/phase9_validation/run.py
|
|
```
|
|
|
|
Ten offline unittest groups check the exact plan/paths/strict arguments, independent
|
|
opt-ins, timeout configuration and invalid values, dry-run nonexecution, default
|
|
build/interop omission, failure collection, fail-fast and failed-build skips,
|
|
missing file/executable handling, literal argv/environment preservation,
|
|
timeout/descendant termination, and inherited streams/closed stdin. They execute
|
|
only small Python fixtures, never PlatformIO, interop, devices, or network.
|
|
|
|
## Execution record — 2026-09-16
|
|
|
|
Initial orchestrator-agent execution, under the existing sandbox without elevation (historical first run):
|
|
|
|
- `python3 -B tests/phase9_validation/run.py`: **10 tests PASS**.
|
|
- `python3 -B tools/validate_phase9.py --dry-run`: exit 0; **plan only**, no
|
|
prerequisite or suite validation.
|
|
- `python3 -B tools/validate_phase9.py`: **exit 1, 22 PASS / 1 FAIL**, no timeouts;
|
|
about 57 seconds of summed command time. Build and interop explicitly skipped.
|
|
- Direct rerun `CCACHE_DISABLE=1 python3 -B tests/admin_console_boundary/run.py`:
|
|
reproduced exit 1 at line 40, `ValueError: substring not found` for
|
|
`ssh.index("static void *ssh_malloc(")`. Earlier subtest PASS lines do not make
|
|
the runner pass. The existing runner is outside this change's ownership and
|
|
was not edited.
|
|
|
|
Passed commands: `security_build_policy`, `ssh_auth_policy`, `ssh_auth_transport`,
|
|
`hidden_input`, `ssh_memory`, **all seven current baseline suites listed above**,
|
|
`ssh_management`, `admin_ssh_policy`, `web_admin_transport`, `web_admin_tickets`,
|
|
`web_httpd_idle`, and all five selected `web_cookie_auth` modes.
|
|
Only `admin_console_boundary` failed.
|
|
|
|
No firmware build, interop, device, runtime reserve, hardware panic/recovery,
|
|
release delivery, or whole-Phase-9 acceptance result is claimed for this task.
|
|
|
|
### Follow-up supplied evidence
|
|
|
|
The owning agent corrected the stale `admin_console_boundary` source anchor;
|
|
its subsequent default orchestrator run reports **23/23 PASS**. This supersedes
|
|
the initial 22/23 host result above, without turning its early PASS lines into
|
|
a successful initial run. Final supplied parent execution now confirms:
|
|
|
|
- `CCACHE_DISABLE=1 python3 -B tools/validate_phase9.py`: **PASS 23/23**.
|
|
- `tests/phase9_validation/run.py`: **PASS 10 tests**.
|
|
|
|
No build or interop ran this turn; both remain separate opt-ins, not default
|
|
execution. Firmware source is unchanged. These are actual parent results supplied
|
|
for this record, not another execution by this documentation integration.
|
|
|
|
The supplied notice agent also reports **36 fixture tests PASS** and two actual
|
|
75-input bundles, each **77 files / 4,433,930 bytes**, verified deterministic; see
|
|
[packaging evidence](release_packaging.md). These are agent measurements, not a
|
|
new build or bundle measurement by this documentation integration.
|
|
|
|
The [finite advisory search](dependency_advisory_coverage.md) is complete for its
|
|
declared snapshot, but the [PMF-capable SoftAP blocker](wifi_security_update_plan.md)
|
|
is **not fixed**. Host success does not repair that vendor bundle, validate radio
|
|
behavior, clear distribution obligations or complete Phase 9.
|