Add broker and web throughput diagnostics

docs/agent/architecture.md

@@ -84,6 +84,8 @@ UART RX -> serial-service RX stream -> broker task
 
 The broker drains serial input even with no clients. A full client output stream drops only that client's copy and updates drop counters; it does not block UART reception or other clients.
 
+Active-client counter snapshots expose ID/type/pending/HWM/UART/queued/read/dropped. HWM is maintained independently of web tracing and counter clear seeds current occupancy. Read means transport handoff, not peer receipt. Disconnect removes the client row while global totals retain traffic and unread-output discards. Independent default-disabled web performance capture uses two fixed slot records and nonwrapping epoch/generation fences; toggles fence in-flight samples, disable freezes aggregates, clear preserves enable state. Binary-only timestamps bound reservation-entry to callback-entry before locking, synchronous send calls and completion to broker-read return; idle and intervening work prevent scheduler-only/backlog-at-completion claims. No new instrumentation allocations or scheduling/buffer changes. Exact fields, overhead and capture contracts: `docs/web_throughput_diagnostics.md`.
+
 ### Clients to RS-232
 
 ```text

docs/agent/code-map.md

@@ -43,6 +43,7 @@ This is a semantic map, not a complete file inventory. Start here, then read the
 - Data path: `transport -> broker -> serial service -> UART1`; reverse data is fanned out per client.
 - Ownership: client IDs are slot/generation-safe; events are advisory and can drop, so use snapshots as authority.
 - Lifecycle: one permanent task and eight preallocated client slots; slow output drops only for the affected client.
+- Diagnostics: `broker counters` adds active-client ID/type/pending/HWM/UART/queued/read/dropped rows; clear seeds HWM from pending, disconnect removes rows but retains global discard accounting. `tests/session_broker_diagnostics/run.py`; capture before disconnect, never use consuming `broker read` as a probe. Semantics/recipe: `docs/web_throughput_diagnostics.md`.
 
 ## Native USB CDC
 
@@ -66,6 +67,7 @@ This is a semantic map, not a complete file inventory. Start here, then read the
 
 - Files: `src/web_server.{h,c}`, `src/web_serial_transport.{h,c}`, `src/web_ui.{h,c}`, `src/web_console.{h,c}`
 - Ordinary HTTPS idle cleanup: `src/web_httpd_idle.{c,h}`, owner sweep in `web_httpd_adapter.{c,h}`, lifecycle/TLS composition in `web_server.c`; `tests/web_httpd_idle/run.py`. Independent of diagnostics/optional transports: 15-second observed idle, one-second timer/one queued probe, six rows, actual WS/async/pending exemptions, safe current-owner shutdown and stop/restart fencing. No LRU/socket/timeout/stack increase. SDK queue/owner-delay limits and target checklist: `docs/https_idle_cleanup.md`.
+- Independent throughput diagnostics: `web_serial_transport.{c,h}` owns two fixed per-slot binary-TX aggregates and epoch fences; `web_console.c` exposes default-disabled `web performance enable|disable|show|clear`. Queue-entry/callback-entry, synchronous-send and completion/drain-return estimates, not peer receipt or scheduler-only latency. `tests/web_serial_performance/run.py`; resource/evidence limits and UART0 paired capture: `docs/web_throughput_diagnostics.md`.
 - Opt-in admission diagnostics: `src/web_diagnostics.{c,h}`, `tests/web_diagnostics/run.py`. Public synchronous HTTPS create/close callbacks publish six post-TLS connection records; four ticket/upgrade wrappers feed a 32-entry numeric ring. UART0/admin SSH `web diagnostics enable|disable|show|clear`; no queue/task/cleanup override or capacity change. Full bounds, SDK semantics and preaccept/TLS blind spots: `docs/phase8d11_implementation.md`.
 - Legacy removal **user-signed-off 2026-09-08** (unchanged certificate fingerprint, preexisting users usable, full-mix evidence): `user_database` persists missing storage empty and preserves valid v1 user bytes; private derived `v1_admin_marker`, no public bootstrap/migration/sync APIs. `web_security` privately migrates v1 1392-byte material to TLS-only v2 1340-byte material, exact identity/generation retained, commit before publish, fail closed without fallback overwrite. Credential commands removed; user generated passwords and TLS rotation remain. Contracts, downgrade and evidence limits: `docs/legacy_credential_removal.md`.
 - Security files: `src/web_security.{h,c}`, `src/web_cookie_auth.{h,c}`, `src/web_session_store.{h,c}`, `src/web_auth_parse.{h,c}`. Private IDF boundary: `src/web_httpd_adapter.{h,c}`.

docs/agent/current-state.md

@@ -4,6 +4,8 @@ This file is working memory. Update it during active work and before handoff; do
 
 ## Development state
 
+- **Web throughput diagnostics handoff (2026-09-08), target unmeasured:** Broker active-client ID/type/pending/HWM/UART/queued/read/dropped rows and independent default-disabled `web performance enable|disable|show|clear` now documented against source. Two fixed web slots, epoch/generation fencing, binary-only queue-entry/callback, synchronous-send and completion/drain timing aggregates; nonempty intervals include idle and are not scheduler-only or proof of completion-time backlog. No CPU160MHz/scheduling/priority/4096-byte broker/512-byte web buffer tuning or new instrumentation allocations. User burst71,292 × four=285,168 expected,265,852 queued,19,316 dropped; browser142,584−123,268 matches, without reported UART loss/disconnect/new login. One-browser no-drop report is not a measured paired sample. Parent broker7/performance suite, console boundary/lifecycle/policy (new SSH-allowed/browser-denied command tests), transport25/tickets12, server lifecycle21, admission diagnostics12+guard, cookie Network+shared PASS; review no confirmed actionable bugs. Parent pio run PASS9.07s,100,100 B RAM/1,748,161 B flash (+552/+3,836 vs Settings presentation), diff check PASS. Fixed extracted console fixture missing new handler double; no production behavior fix. Host sanitizer libraries missing, no sanitizer pass. Real scheduling/overhead/stack margins unmeasured. UART0 quiet/settled full-mix and one-browser capture recipe, before-disconnect attribution, enabled/disabled comparison, 177.8ms retention scale and limits: `docs/web_throughput_diagnostics.md`. Documentation-only pass; no source/tests/assets/device/build/commit action.
+
 - **8D.12/8D.13 + Settings presentation functionally signed off (2026-09-08):** User explicitly accepts after 60-second boot/full-mix telemetry. Web writer8, web17/SSH10/USB11 observers, both admins, two cookies at230400 8N1 RTS/CTS DTR active. Web send/queue/protocol/close-failure0, SSH handshake/auth/IO failures0; retain SSH rejected1 byte, web rejected1 frame/1 byte, serial connect3/disconnect1, cookie login3/logout1, authfailures2 boot/3 loaded without causal diagnosis. Loaded free internal/DMA/PSRAM31,512/23,756/8,112,140 B; minima2,276/156/8,070,736 B; largest20,480/20,480/7,995,392 B. SSH stack min18,468 boot/16,276 loaded. Exact samples/counters/limits: `docs/phase8d12_13_implementation.md`. Supersedes target-pending/no-signoff wording below for implemented phases and visual refinements. Very low lifetime internal/DMA minima remain transient-headroom follow-up (conservative non-simultaneous region sums, not proof allocation failed); HTTPD/dispatcher floors/peak correlation and numeric reserves unapproved, not reopening functional acceptance. No invented individual checklist/soak/full-M3 pass, no 8D.14 authorization. Documentation only, no source/config/build/test/device/assets/commit action.
 
 - **Settings presentation unified (2026-09-08), user-requested visual refinement:** Accounts/Network now use Serial-style 600px label/value definition lists, shared form styling, compact muted help, consistent Refresh/result labels and action grouping. Public-key textarea/generated-password fields styled; checkboxes intrinsic-width. Safe DOM text replaces preformatted summaries; readable ASCII SSID/hex fallback retained, pre-wrap preserves significant spaces (review finding fixed with rendered-width regression). IDs/events/auth/mutations/secret cleanup/terminal ownership unchanged; no backend/assets changes. Parent UI100 behavior groups plus HTML/renderer/CSP and headless Chromium geometry/whitespace checks at320/600/1200px PASS; pio run PASS23.69s, 99,548 B RAM / 1,744,325 B flash (+0/+1,744 vs preceding ASCII-summary build). Diff check PASS. Fixtures/browser layout checks are not target visual sign-off or live Wi-Fi validation. No upload/erase/commit.

docs/agent/design-decisions.md

@@ -28,6 +28,8 @@ Only constraints supported by implementation or current project documentation be
 
 **Consequence for future changes:** Do not replace fan-out with a blocking shared queue. Any added transport must tolerate partial/no-progress reads and expose drop/backpressure counters.
 
+**Throughput observation, not tuning:** Preserve CPU160MHz, scheduling/priorities and 4096/512-byte broker/web buffers while gathering per-client HWM/drop attribution and independent opt-in web binary-TX timing. Fixed-slot epoch/generation-fenced aggregates avoid stale attribution; no new runtime allocations. Clear preserves queued data and seeds broker HWM; disconnected rows disappear while global discard counts remain. Callback timestamps precede the transport lock; synchronous send return is not peer receipt. Completion-to-read intervals include broker/mutex/control work and possible idle, even when the first read is nonempty; never label them pure scheduling latency or proof of backlog at completion. Compare enabled/disabled target captures before drawing overhead conclusions. Contracts and reproduction: `docs/web_throughput_diagnostics.md`.
+
 **Relevant files:** `src/session_broker.c`, `src/session_broker.h`, `docs/roadmap.md`
 
 ## Physical UART ownership and logical writer ownership remain separate

docs/command_reference.md

@@ -66,7 +66,7 @@ Defaults are 115200 baud, 8 data bits, no parity, one stop bit, no flow control,
 | Command | Description |
 |---|---|
 | `broker status` / `broker clients` | Show broker state or connected clients. |
-| `broker counters` / `broker clear-counters` | Show or clear broker counters. |
+| `broker counters` / `broker clear-counters` | Show global totals plus active-client ID/type/pending/HWM/UART/queued/read/dropped; clear counters and seed HWM from current pending bytes without draining output. |
 | `broker connect <name>` / `broker disconnect <client-id>` | Create or remove a console test client. |
 | `broker request-writer <client-id>` / `broker release-writer <client-id>` | Request or relinquish the single writer lease. |
 | `broker force-writer <client-id|none>` | Administratively assign or clear the writer lease. |
@@ -74,7 +74,7 @@ Defaults are 115200 baud, 8 data bits, no parity, one stop bit, no flow control,
 | `broker read <client-id> [maximum-bytes]` | Read queued serial output for a client. |
 | `broker events <client-id>` | Show ownership and connection events for a client. |
 
-Each client has a generation-safe ID. There can be one writer and multiple observers; a slow observer loses only its own queued output.
+Each client has a generation-safe ID. There can be one writer and multiple observers; a slow observer loses only its own queued output. HWM is bounded by 4,096 bytes; read counts transport handoff, not peer receipt. Disconnected client rows disappear, while global totals retain traffic and unread-output discards until cleared. Capture counters before disconnect. `broker read` consumes queued data and must not be used as a throughput diagnostic probe. See [Web throughput diagnostics](web_throughput_diagnostics.md).
 
 ## Native USB CDC-ACM
 
@@ -126,11 +126,15 @@ When the Wi-Fi station receives an IPv4 address, the Wi-Fi manager announces `sa
 | `web` / `web help` | Show web-service command usage. |
 | `web status` | Show HTTPS and WebSocket state. |
 | `web start` / `web stop` | Start or stop HTTPS service. |
-| `web counters` / `web clear-counters` | Show or clear web counters. |
+| `web counters` / `web clear-counters` | Show or clear ordinary HTTPS/serial WebSocket counters, independently of performance capture. |
+| `web diagnostics enable\|disable\|show\|clear` | Independent opt-in admission tracing and post-TLS occupancy; not serial throughput timing. |
+| `web performance enable\|disable\|show\|clear` | Default-disabled per-slot binary TX timing/count aggregates. Disable freezes; enable resumes; clear preserves enabled state. Each control operation fences in-flight samples with a new epoch. All actions print a snapshot. |
 | `web certificate info` | Display certificate identity and fingerprint. |
 | `web certificate rotate --force` | Replace the HTTPS certificate and private key. Browser admin defers commit and HTTPS restart; both browser routes close and new certificate trust/relogin is required. UART0/admin SSH behavior is unchanged. |
 | `web reset --force` | Explicitly replace HTTPS certificate/private key only, including missing, incompatible or damaged material; never changes users. |
 
+Use UART0 for quiet throughput captures (authenticated admin SSH also exposes these commands). Performance output includes current-epoch pending age and count/sum/estimated-average/max timings for queue entry to callback entry, synchronous send, and completion to drain return. These are not peer acknowledgements or scheduler-only measurements; nonempty intervals may include idle. No secrets or serial payloads are printed. See [Web throughput diagnostics](web_throughput_diagnostics.md) for exact semantics, saturation limits and the reset/burst/drain/disable/capture-before-disconnect recipe.
+
 HTTPS listens on port 443 only. Authenticate with any current user-database username/password; both `user` and `admin` roles receive the existing status and browser-terminal interface. The device serves vendored xterm.js without Internet access. Browser sessions use one-time account-bound tickets, binary WebSocket frames, and the broker's one-writer rule. The combined **Connect**/**Disconnect** control closes the current WebSocket and pauses automatic reconnect when active; after a user-paused disconnect, it changes to **Connect** to resume connection attempts. Account mutations revoke only that account's tickets and sessions.
 
 ## SSH serial transport

docs/web_throughput_diagnostics.md

@@ -0,0 +1,79 @@
+# Web serial throughput diagnostics
+
+## Scope and evidence (2026-09-08)
+
+This is instrumentation, not throughput tuning or a confirmed scheduler diagnosis. UART0 recovery, network-independent native USB, binary-transparent UART1 transport and the broker's one-writer/isolated-observer model remain unchanged. CPU stays at 160 MHz; scheduling, priorities, 4,096-byte broker output buffers and 512-byte web TX payloads are unchanged.
+
+The motivating user capture reported a 71,292-byte UART burst with four fan-outs: expected 285,168 bytes, queued 265,852, dropped 19,316. The two browsers' expected 142,584 minus actual 123,268 is exactly 19,316. No UART loss, disconnect or new login was reported during this burst. This is consistent with browser-copy loss at broker fan-out, but the old aggregate counters cannot attribute loss to either browser or identify the cause. The user reports no drops with one browser; that is not a measured paired baseline.
+
+## Broker accounting
+
+`broker counters` retains global totals and adds one row per active client: generation-safe ID, type, pending bytes, output high-water mark (HWM), UART bytes considered for that client, queued bytes, read bytes and dropped bytes. `broker clients` helps map the IDs to active transports; `web performance show` supplies each web slot's broker ID.
+
+- HWM is peak output occupancy, bounded by 4,096 bytes. `broker clear-counters` clears totals but seeds each active HWM with current pending occupancy, not zero. It does not drain queues or change ownership.
+- `read` means handed to a transport, not received or rendered by a peer. Queued counts successful copies; UART counts bytes considered while the client was connected.
+- Disconnected per-client rows disappear and slot/generation reuse resets them. Global counters retain disconnected traffic until cleared, including accepted-but-unread output discarded on disconnect. Thus global drops need not mean only overflow, and queued plus dropped need not equal UART fan-out after disconnect discards.
+- Capture before disconnect. **Do not use `broker read` for observation:** it consumes data intended for that transport. Snapshot/counter commands do not consume serial data.
+
+## Independent web performance capture
+
+Use physical UART0 for the least intrusive capture. Commands are also available through the authenticated admin SSH registry. `web performance enable|disable|show|clear` is separate from admission tracing (`web diagnostics enable|disable|show|clear`) and from ordinary `web counters|clear-counters`.
+
+Performance capture defaults disabled. Enable resumes aggregates; disable freezes aggregates; clear resets aggregates while preserving enable state. Each control operation advances a nonwrapping epoch and fences in-flight samples. Exhaustion fails closed. Slot generation and broker identity also fence reuse; this is not a disconnected-session archive.
+
+Each active one of two fixed slots shows fd, generation, broker ID, pending/measured-pending/executing state and current-epoch `pending_age_us`. Age starts at reservation-path entry and includes an executing send. If `measured_pending=0`, zero age is unavailable, not evidence of immediate service. Disabling invalidates current-epoch pending age; capture a live `show` only if investigating a stuck queue, accepting its perturbation.
+
+Only serial **binary TX** is measured. Text/control frames are not samples, though they can delay subsequent binary work. No payloads, passwords, keys, cookies, tickets, verifier material or other authentication secrets are retained or printed.
+
+| Printed timing | Exact interpretation |
+|---|---|
+| `queue->callback-entry` | Reservation-path entry to HTTPD callback entry, timestamped before the callback takes the transport lock. Includes submission-path work, not just HTTPD queue residence; owned callbacks retired without sending can count. |
+| `send-call` | Time around the actual synchronous HTTPD-owner send API call. Return is not peer acknowledgement, browser receipt or rendering. |
+| `completion->first-drain-attempt-return` | Successful send-call completion to return of the first subsequent broker read attempt, including empty/error attempts. |
+| `completion->next-nonempty-drain-return (includes idle)` | Completion to the next successful nonempty broker read return, possibly after empty attempts and source idle time. |
+| `completion->first-attempt-nonempty-return` | Subset where the first subsequent read returns data. Excludes observed empty attempts, but does not prove backlog existed at send completion. |
+
+Each timing reports count, sum in microseconds, integer estimated average (`sum/count`, zero for no samples) and maximum. Completion intervals are not scheduler-only: they include transport work, broker read/mutex time and intervening control-frame work, and may include idle time even in the first-attempt-nonempty subset. They do not isolate a particular task, TLS operation or network cause.
+
+`queued_frames/bytes` counts transport reservations, including reported queue failures; `queue_errors` records those failures. `sent_frames/bytes` counts successful send-call returns. `send_errors` includes owner-context rejection, whereas send-call timings require an actual API call. `retired` records measured callbacks retired without sending. Saturating aggregates set `saturated=1`; totals and averages then cease to be reliable. Epoch fencing intentionally prevents old work contaminating a new capture, so do not demand equality across toggles or in-flight clears.
+
+## Reproducible UART0 capture
+
+1. Record firmware/build identity, baud/framing/flow control, source burst size, browser versions, client IDs/roles and full-mix topology. Establish all serial clients and both admin routes first. Quiet the UART source and allow queues and connection activity to settle. Do not reconnect/login during the trace.
+2. Optionally stop independent admission tracing before the timed capture. Then reset in this order, while quiet:
+
+   ```text
+   web diagnostics disable
+   web performance clear
+   web performance enable
+   broker clear-counters
+   serial clear-counters
+   web clear-counters
+   ```
+
+   The first command is optional; record whether admission tracing was enabled. Resets are sequential, not an atomic cross-service snapshot.
+3. Send the known burst once, stop the source and wait for transport queues to drain. Avoid repeated console/status printing during the burst. A quiet final broker pending count of zero alone does not prove browser receipt; retain peer byte counts too.
+4. Stop performance capture first, then capture all counters **before disconnecting any client**:
+
+   ```text
+   web performance disable
+   web performance show
+   broker counters
+   serial counters
+   web counters
+   broker clients
+   ```
+
+   Disable already prints the performance snapshot; `show` is an explicit repeatable frozen-aggregate record. Keep connection counts and source/peer totals with the sample.
+5. Only after saving evidence, change to one browser, keeping the other full-mix participants and serial settings equivalent. Quiet/settle and repeat the complete reset/burst/drain/stop/capture sequence. Do not compare a fresh capture to lifetime counters.
+6. Separately compare equivalent performance-enabled and performance-disabled bursts with fresh ordinary counter resets. Keep performance disabled for the latter; do not treat retained performance aggregates as that run's measurements. Keep admission tracing state equal and record it.
+
+At 230400 baud, 8N1, 4,096 bytes represents approximately **177.8 ms** of continuous input retention from an empty broker queue, ignoring flow-control pauses. Compare per-client drops/HWM with latency maxima against this scale, not as a hard deadline or proof of cause; occupancy, idle gaps and timings cover different boundaries. Target follow-up is per-client attribution and these controlled comparisons, not buffer, priority, scheduler or CPU tuning.
+
+## Resources and validation handoff
+
+Two fixed web-slot diagnostic records plus epoch/timestamp/state fields and per-client broker HWM fields add bounded storage; there are no new allocations, tasks, queues or payload buffers for this instrumentation. Disabled is not zero overhead: relaxed gate loads/branches remain, and broker occupancy sampling/HWM maintenance is always active. Enabled timestamping and short locked aggregate updates are bounded. Target runtime overhead, stack margins and throughput effects remain unmeasured.
+
+Parent integrated validation PASS: `python3 tests/session_broker_diagnostics/run.py` (7 groups), `python3 tests/web_serial_performance/run.py` (production queue/send/drain paths with host doubles, epochs, idle, retirement and saturation; suite-level PASS), console boundary `run.py`/`lifecycle.py`, `tests/admin_ssh_policy/run.py`, admin transport `--tickets` (25 transport/12 ticket groups), server lifecycle (21), admission diagnostics (12+guard) and cookie `--network`. Parent updated the extracted console fixture for the new command and added explicit SSH-allowed/browser-denied performance command policy regressions. Independent review found no confirmed actionable production bugs. Synthetic reproduction is not target measurement; real scheduling/critical-section overhead and delayed original-pointer reuse still rely on existing owner lifecycle guarantees.
+
+Final parent `pio run` **PASS, 9.07 s**, **100,100 B RAM / 1,748,161 B flash**, **+552 B RAM / +3,836 B flash** against the recorded Settings-presentation build. These are whole-build figures, not isolated record sizes or runtime heap measurements. Parent diff check passed; subsequent policy-test additions change no production source and their reruns passed. Host sanitizer libraries are unavailable; no sanitizer pass is claimed. No upload, erase, commit, generated assets, target acceptance, measured enabled/disabled overhead, lossless paired baseline or tuning result.