Add broker and web throughput diagnostics

docs/web_throughput_diagnostics.md

@@ -0,0 +1,79 @@
+# Web serial throughput diagnostics
+
+## Scope and evidence (2026-09-08)
+
+This is instrumentation, not throughput tuning or a confirmed scheduler diagnosis. UART0 recovery, network-independent native USB, binary-transparent UART1 transport and the broker's one-writer/isolated-observer model remain unchanged. CPU stays at 160 MHz; scheduling, priorities, 4,096-byte broker output buffers and 512-byte web TX payloads are unchanged.
+
+The motivating user capture reported a 71,292-byte UART burst with four fan-outs: expected 285,168 bytes, queued 265,852, dropped 19,316. The two browsers' expected 142,584 minus actual 123,268 is exactly 19,316. No UART loss, disconnect or new login was reported during this burst. This is consistent with browser-copy loss at broker fan-out, but the old aggregate counters cannot attribute loss to either browser or identify the cause. The user reports no drops with one browser; that is not a measured paired baseline.
+
+## Broker accounting
+
+`broker counters` retains global totals and adds one row per active client: generation-safe ID, type, pending bytes, output high-water mark (HWM), UART bytes considered for that client, queued bytes, read bytes and dropped bytes. `broker clients` helps map the IDs to active transports; `web performance show` supplies each web slot's broker ID.
+
+- HWM is peak output occupancy, bounded by 4,096 bytes. `broker clear-counters` clears totals but seeds each active HWM with current pending occupancy, not zero. It does not drain queues or change ownership.
+- `read` means handed to a transport, not received or rendered by a peer. Queued counts successful copies; UART counts bytes considered while the client was connected.
+- Disconnected per-client rows disappear and slot/generation reuse resets them. Global counters retain disconnected traffic until cleared, including accepted-but-unread output discarded on disconnect. Thus global drops need not mean only overflow, and queued plus dropped need not equal UART fan-out after disconnect discards.
+- Capture before disconnect. **Do not use `broker read` for observation:** it consumes data intended for that transport. Snapshot/counter commands do not consume serial data.
+
+## Independent web performance capture
+
+Use physical UART0 for the least intrusive capture. Commands are also available through the authenticated admin SSH registry. `web performance enable|disable|show|clear` is separate from admission tracing (`web diagnostics enable|disable|show|clear`) and from ordinary `web counters|clear-counters`.
+
+Performance capture defaults disabled. Enable resumes aggregates; disable freezes aggregates; clear resets aggregates while preserving enable state. Each control operation advances a nonwrapping epoch and fences in-flight samples. Exhaustion fails closed. Slot generation and broker identity also fence reuse; this is not a disconnected-session archive.
+
+Each active one of two fixed slots shows fd, generation, broker ID, pending/measured-pending/executing state and current-epoch `pending_age_us`. Age starts at reservation-path entry and includes an executing send. If `measured_pending=0`, zero age is unavailable, not evidence of immediate service. Disabling invalidates current-epoch pending age; capture a live `show` only if investigating a stuck queue, accepting its perturbation.
+
+Only serial **binary TX** is measured. Text/control frames are not samples, though they can delay subsequent binary work. No payloads, passwords, keys, cookies, tickets, verifier material or other authentication secrets are retained or printed.
+
+| Printed timing | Exact interpretation |
+|---|---|
+| `queue->callback-entry` | Reservation-path entry to HTTPD callback entry, timestamped before the callback takes the transport lock. Includes submission-path work, not just HTTPD queue residence; owned callbacks retired without sending can count. |
+| `send-call` | Time around the actual synchronous HTTPD-owner send API call. Return is not peer acknowledgement, browser receipt or rendering. |
+| `completion->first-drain-attempt-return` | Successful send-call completion to return of the first subsequent broker read attempt, including empty/error attempts. |
+| `completion->next-nonempty-drain-return (includes idle)` | Completion to the next successful nonempty broker read return, possibly after empty attempts and source idle time. |
+| `completion->first-attempt-nonempty-return` | Subset where the first subsequent read returns data. Excludes observed empty attempts, but does not prove backlog existed at send completion. |
+
+Each timing reports count, sum in microseconds, integer estimated average (`sum/count`, zero for no samples) and maximum. Completion intervals are not scheduler-only: they include transport work, broker read/mutex time and intervening control-frame work, and may include idle time even in the first-attempt-nonempty subset. They do not isolate a particular task, TLS operation or network cause.
+
+`queued_frames/bytes` counts transport reservations, including reported queue failures; `queue_errors` records those failures. `sent_frames/bytes` counts successful send-call returns. `send_errors` includes owner-context rejection, whereas send-call timings require an actual API call. `retired` records measured callbacks retired without sending. Saturating aggregates set `saturated=1`; totals and averages then cease to be reliable. Epoch fencing intentionally prevents old work contaminating a new capture, so do not demand equality across toggles or in-flight clears.
+
+## Reproducible UART0 capture
+
+1. Record firmware/build identity, baud/framing/flow control, source burst size, browser versions, client IDs/roles and full-mix topology. Establish all serial clients and both admin routes first. Quiet the UART source and allow queues and connection activity to settle. Do not reconnect/login during the trace.
+2. Optionally stop independent admission tracing before the timed capture. Then reset in this order, while quiet:
+
+   ```text
+   web diagnostics disable
+   web performance clear
+   web performance enable
+   broker clear-counters
+   serial clear-counters
+   web clear-counters
+   ```
+
+   The first command is optional; record whether admission tracing was enabled. Resets are sequential, not an atomic cross-service snapshot.
+3. Send the known burst once, stop the source and wait for transport queues to drain. Avoid repeated console/status printing during the burst. A quiet final broker pending count of zero alone does not prove browser receipt; retain peer byte counts too.
+4. Stop performance capture first, then capture all counters **before disconnecting any client**:
+
+   ```text
+   web performance disable
+   web performance show
+   broker counters
+   serial counters
+   web counters
+   broker clients
+   ```
+
+   Disable already prints the performance snapshot; `show` is an explicit repeatable frozen-aggregate record. Keep connection counts and source/peer totals with the sample.
+5. Only after saving evidence, change to one browser, keeping the other full-mix participants and serial settings equivalent. Quiet/settle and repeat the complete reset/burst/drain/stop/capture sequence. Do not compare a fresh capture to lifetime counters.
+6. Separately compare equivalent performance-enabled and performance-disabled bursts with fresh ordinary counter resets. Keep performance disabled for the latter; do not treat retained performance aggregates as that run's measurements. Keep admission tracing state equal and record it.
+
+At 230400 baud, 8N1, 4,096 bytes represents approximately **177.8 ms** of continuous input retention from an empty broker queue, ignoring flow-control pauses. Compare per-client drops/HWM with latency maxima against this scale, not as a hard deadline or proof of cause; occupancy, idle gaps and timings cover different boundaries. Target follow-up is per-client attribution and these controlled comparisons, not buffer, priority, scheduler or CPU tuning.
+
+## Resources and validation handoff
+
+Two fixed web-slot diagnostic records plus epoch/timestamp/state fields and per-client broker HWM fields add bounded storage; there are no new allocations, tasks, queues or payload buffers for this instrumentation. Disabled is not zero overhead: relaxed gate loads/branches remain, and broker occupancy sampling/HWM maintenance is always active. Enabled timestamping and short locked aggregate updates are bounded. Target runtime overhead, stack margins and throughput effects remain unmeasured.
+
+Parent integrated validation PASS: `python3 tests/session_broker_diagnostics/run.py` (7 groups), `python3 tests/web_serial_performance/run.py` (production queue/send/drain paths with host doubles, epochs, idle, retirement and saturation; suite-level PASS), console boundary `run.py`/`lifecycle.py`, `tests/admin_ssh_policy/run.py`, admin transport `--tickets` (25 transport/12 ticket groups), server lifecycle (21), admission diagnostics (12+guard) and cookie `--network`. Parent updated the extracted console fixture for the new command and added explicit SSH-allowed/browser-denied performance command policy regressions. Independent review found no confirmed actionable production bugs. Synthetic reproduction is not target measurement; real scheduling/critical-section overhead and delayed original-pointer reuse still rely on existing owner lifecycle guarantees.
+
+Final parent `pio run` **PASS, 9.07 s**, **100,100 B RAM / 1,748,161 B flash**, **+552 B RAM / +3,836 B flash** against the recorded Settings-presentation build. These are whole-build figures, not isolated record sizes or runtime heap measurements. Parent diff check passed; subsequent policy-test additions change no production source and their reruns passed. Host sanitizer libraries are unavailable; no sanitizer pass is claimed. No upload, erase, commit, generated assets, target acceptance, measured enabled/disabled overhead, lossless paired baseline or tuning result.