ESP32_Serial_Swiss_Army_Knife/docs/web_throughput_diagnostics.md

Web serial throughput diagnostics

Current contract and accepted fix

Default-disabled instrumentation measures broker fan-out and binary WebSocket TX, not CPU usage, peer receipt or pure scheduler latency. The user separately accepted the combined binary header/payload-send fix at 160 MHz / 230400 baud with full mix including browser admin. Acceptance evidence distinguishes that report from the latest capture and unmeasured reserves. The old frequency-only experiment is not the retained solution.

web_httpd_ws_send_binary is an IDF-5.5.0-pinned owner-only session-override send with a bounded 516-byte header/payload scratch copy and 512-byte payload. Generation validation and one outstanding item per slot remain. A non-full send installs a reject-only override before close/shutdown, preventing SDK automatic PONG/CLOSE from reentering TLS with different arguments after incomplete output, even if shutdown fails. No retry/replay; HTTPD owns TLS destruction. Text/control/admin retain the SDK sender. One API call does not promise one TLS record/packet or peer receipt. Keep CPU160, priorities, scheduling and 4096-byte broker/512-byte web payload bounds unless separately changing them with evidence.

Broker accounting

broker counters retains global totals and adds one row per active client: generation-safe ID, type, pending bytes, output high-water mark (HWM), UART bytes considered for that client, queued bytes, read bytes and dropped bytes. broker clients helps map the IDs to active transports; web performance show supplies each web slot's broker ID.

• HWM is peak output occupancy, bounded by 4,096 bytes. broker clear-counters clears totals but seeds each active HWM with current pending occupancy, not zero. It does not drain queues or change ownership.
• read means handed to a transport, not received or rendered by a peer. Queued counts successful copies; UART counts bytes considered while the client was connected.
• Disconnected per-client rows disappear and slot/generation reuse resets them. Global counters retain disconnected traffic until cleared, including accepted-but-unread output discarded on disconnect. Thus global drops need not mean only overflow, and queued plus dropped need not equal UART fan-out after disconnect discards.
• Capture before disconnect. Do not use broker read for observation: it consumes data intended for that transport. Snapshot/counter commands do not consume serial data.

Independent web performance capture

Use physical UART0 for the least intrusive capture. Commands are also available through the authenticated admin SSH registry. web performance enable|disable|show|clear is separate from admission tracing (web diagnostics enable|disable|show|clear) and from ordinary web counters|clear-counters.

Performance capture defaults disabled. Enable resumes aggregates; disable freezes aggregates; clear resets aggregates while preserving enable state. Each control operation advances a nonwrapping epoch and fences in-flight samples. Exhaustion fails closed. Slot generation and broker identity also fence reuse; this is not a disconnected-session archive.

Each active one of two fixed slots shows fd, generation, broker ID, pending/measured-pending/executing state and current-epoch pending_age_us. Age starts at reservation-path entry and includes an executing send. If measured_pending=0, zero age is unavailable, not evidence of immediate service. Disabling invalidates current-epoch pending age; capture a live show only if investigating a stuck queue, accepting its perturbation.

Only serial binary TX is measured. Text/control frames are not samples, though they can delay subsequent binary work. No payloads, passwords, keys, cookies, tickets, verifier material or other authentication secrets are retained or printed.

Printed timing	Exact interpretation
queue->callback-entry	Reservation-path entry to HTTPD callback entry, timestamped before the callback takes the transport lock. Includes submission-path work, not just HTTPD queue residence; owned callbacks retired without sending can count.
send-call	Time around the actual synchronous HTTPD-owner send API call. Return is not peer acknowledgement, browser receipt or rendering.
completion->first-drain-attempt-return	Successful send-call completion to return of the first subsequent broker read attempt, including empty/error attempts.
completion->next-nonempty-drain-return (includes idle)	Completion to the next successful nonempty broker read return, possibly after empty attempts and source idle time.
completion->first-attempt-nonempty-return	Subset where the first subsequent read returns data. Excludes observed empty attempts, but does not prove backlog existed at send completion.

Each timing reports count, sum in microseconds, integer estimated average (sum/count, zero for no samples) and maximum. Completion intervals are not scheduler-only: they include transport work, broker read/mutex time and intervening control-frame work, and may include idle time even in the first-attempt-nonempty subset. They do not isolate a particular task, TLS operation or network cause.

queued_frames/bytes counts transport reservations, including reported queue failures; queue_errors records those failures. sent_frames/bytes counts successful send-call returns. send_errors includes owner-context rejection, whereas send-call timings require an actual API call. retired records measured callbacks retired without sending. Saturating aggregates set saturated=1; totals and averages then cease to be reliable. Epoch fencing intentionally prevents old work contaminating a new capture, so do not demand equality across toggles or in-flight clears.

Reproducible UART0 capture

1. Record firmware/build identity, baud/framing/flow control, source burst size, browser versions, client IDs/roles and full-mix topology. Establish all serial clients and both admin routes first. Quiet the UART source and allow queues and connection activity to settle. Do not reconnect/login during the trace.

2. Optionally stop independent admission tracing before the timed capture. Then reset in this order, while quiet:

   web diagnostics disable
   web performance clear
   web performance enable
   broker clear-counters
   serial clear-counters
   web clear-counters
   

   The first command is optional; record whether admission tracing was enabled. Resets are sequential, not an atomic cross-service snapshot.

3. Send the known burst once, stop the source and wait for transport queues to drain. Avoid repeated console/status printing during the burst. A quiet final broker pending count of zero alone does not prove browser receipt; retain peer byte counts too.

4. Stop performance capture first, then capture all counters before disconnecting any client:

   web performance disable
   web performance show
   broker counters
   serial counters
   web counters
   broker clients
   

   Disable already prints the performance snapshot; show is an explicit repeatable frozen-aggregate record. Keep connection counts and source/peer totals with the sample.

5. Only after saving evidence, change to one browser, keeping the other full-mix participants and serial settings equivalent. Quiet/settle and repeat the complete reset/burst/drain/stop/capture sequence. Do not compare a fresh capture to lifetime counters.

6. Separately compare equivalent performance-enabled and performance-disabled bursts with fresh ordinary counter resets. Keep performance disabled for the latter; do not treat retained performance aggregates as that run's measurements. Keep admission tracing state equal and record it.

At 230400 baud, 8N1, 4,096 bytes represents approximately 177.8 ms of continuous input retention from an empty broker queue, ignoring flow-control pauses. Compare per-client drops/HWM with latency maxima against this scale, not as a hard deadline or proof of cause; occupancy, idle gaps and timings cover different boundaries. For future regressions, gather per-client attribution and controlled comparisons before changing buffers, priorities, scheduling or CPU.

Resources and regression references

Two fixed diagnostic records plus nonwrapping epoch/timestamp/slot fences and broker HWM fields add bounded storage, no new instrumentation allocations/tasks/queues/payloads. Disabled is not zero overhead: gate branches remain and broker occupancy/HWM is always active. Enabled timing and short aggregate locks have runtime costs not established by host tests. The combined-send local scratch adds HTTPD stack use; actual reserve remains a target measurement.

tests/session_broker_diagnostics/run.py covers counters and generation-safe management; tests/web_serial_performance/run.py covers production send/drain paths, epochs, retirement, SDK wire compatibility and failed-send/control-reply isolation. tests/admin_ssh_policy/run.py covers SSH-allowed/browser-denied tracing commands. Prior focused/broad host validation and firmware builds passed; no tests were run during documentation consolidation. These harnesses do not prove real TLS/scheduler overhead, peer-byte integrity or long-soak behavior. Use the capture procedure above for future attribution, retaining explicit workload/duration/counter provenance.