Send binary WebSocket frames in one write
This commit is contained in:
@@ -84,6 +84,8 @@ UART RX -> serial-service RX stream -> broker task
|
||||
|
||||
The broker drains serial input even with no clients. A full client output stream drops only that client's copy and updates drop counters; it does not block UART reception or other clients.
|
||||
|
||||
Binary serial WebSocket output uses the IDF-5.5.0-pinned `web_httpd_ws_send_binary` adapter: one owner-only session-override send of a bounded header+payload copy (516 bytes of local scratch, 512-byte payload). Existing generation validation and one outstanding work item per slot remain required. Non-full sends immediately replace that session's send override with a reject-only guard, mark it closing and shut down the socket before deferred cleanup; this prevents automatic SDK control replies from reentering TLS after incomplete output. HTTPD retains TLS destruction ownership. Text/control and admin output retain the SDK sender. One send call does not imply one TLS record, packet or peer receipt; the user signed off drop-free230400-baud full-client-mix operation at160MHz. Runtime stack-margin validation remains a separate follow-up; evidence and acceptance scope are in `current-state.md`.
|
||||
|
||||
Active-client counter snapshots expose ID/type/pending/HWM/UART/queued/read/dropped. HWM is maintained independently of web tracing and counter clear seeds current occupancy. Read means transport handoff, not peer receipt. Disconnect removes the client row while global totals retain traffic and unread-output discards. Independent default-disabled web performance capture uses two fixed slot records and nonwrapping epoch/generation fences; toggles fence in-flight samples, disable freezes aggregates, clear preserves enable state. Binary-only timestamps bound reservation-entry to callback-entry before locking, synchronous send calls and completion to broker-read return; idle and intervening work prevent scheduler-only/backlog-at-completion claims. No new instrumentation allocations or scheduling/buffer changes. Exact fields, overhead and capture contracts: `docs/web_throughput_diagnostics.md`.
|
||||
|
||||
### Clients to RS-232
|
||||
|
||||
@@ -4,6 +4,18 @@ This file is working memory. Update it during active work and before handoff; do
|
||||
|
||||
## Development state
|
||||
|
||||
- **Throughput fix signed off at160MHz (2026-09-08):** User explicitly validates drop-free230400-baud operation with the full client mix, including browser admin, at160MHz. Current defaults, active sdkconfig and generated sdkconfig.h independently confirm160MHz selection. This supersedes earlier frequency uncertainty and pending functional throughput validation below. Retain combined binary header/payload TLS send and bounded failure isolation; no further tuning needed for this issue. Latest full-mix160MHz result is user-reported without additional raw counters or duration; no invented long-soak, peer-byte comparison or HTTPD-stack reserve validation. Documentation-only acceptance update; no source/config/build/upload/erase/commit action.
|
||||
|
||||
- **Combined binary WS send functionally validated by user (2026-09-08):** User confirms no drops with browser admin also connected and explicitly signs off. Prior detailed two-browser/SSH/USB capture (browser admin absent) had UART71,292 B, all four clients read71,292/drop0, global285,168, web430 frames/142,584 B, HWM2,499/2,563; send averages4,215/4,522us and callback5,691/6,253us, no transport errors/disconnects. Full-mix follow-up is user-reported, no additional counters/duration supplied. Supersedes target-pending functional status below; no sustained-soak/HTTPD-stack reserve approval inferred. User also reports commenting out CPU240 default and assumes160MHz; inspection shows tracked default commented but active sdkconfig and generated sdkconfig.h STILL select240MHz. Thus full-mix functional sign-off stands, but160MHz operation is not verified. Preserve user config edit; no automatic rollback/build/upload. Next optional step is explicit160MHz selection in defaults and active config, rebuild/generated-setting verification, then target retest. Documentation-only sign-off update.
|
||||
|
||||
- **Combined binary WS send experiment implemented (2026-09-08), target pending:** User authorized after CPU240 target run still dropped6,318 browser bytes. `web_httpd_adapter` now assembles FIN/binary/unmasked header+payload in516 B owner-local scratch, sends once through the existing HTTPS session override; serial binary only, controls/admin remain SDK path. IDF5.5.0-pinned owner/current-context/upgraded/nonclosing/nonasync checks,512 B payload/one pending slot/generation fencing retained. Full-length return required; short/zero/error installs reject-only send override BEFORE ws_close/shutdown so SDK automatic PONG/CLOSE cannot reenter TLS with different arguments before deferred cleanup, even if shutdown fails. Independent review found that race in initial implementation; implementer fixed, parent inspected correction and reran performance suite PASS. No retry/replay; normal HTTPD owns TLS destruction. One TLS API call is not a guarantee of one TLS record/packet or peer receipt. No priority/affinity/TCP/buffer/task/config/assets changes; CPU240 retained. Implementer final pio run PASS22.08s,100,100 B RAM/1,748,513 B flash (+352 vs CPU-only baseline); helper local stack frame560 B, runtime HTTPD margin unmeasured. Implementer tests PASS performance (513 installed-SDK wire comparisons, short/zero/TLS errors, actual SDK automatic PING/CLOSE after failure including shutdown failure, fd reuse/slot isolation/diagnostic epochs), cookie base/settings/admin, idle18, lifecycle21, store serial. Parent performance/diff check PASS. Host TLS/scheduler doubles are not target timing/fault proof. No upload/erase/commit. Next: same settled full-mix capture and dmesg burst at240MHz, memory and HTTPD stack margin, then sustained traffic; no throughput fix claimed.
|
||||
|
||||
- **CPU240-only target result (2026-09-08):** UART71,311 B/no loss/errors; SSH/USB each71,311 read/drop0, HWM704/449. Web10 read68,504/drop2,807; web11 read67,800/drop3,511; both HWM4096, pending0. Global278,926 read+6,318 dropped=4×71,311; web277 frames/136,304 B equals browser reads. No connect/disconnect or send/queue/protocol errors;17 authenticated status requests. Callback wait avg11,704/11,939us, send8,292/8,547us, first drain3,332/3,257us. DCD116/RI4 transitions retained without diagnosis. Roughly68% fewer dropped copies than160MHz paired run, but still overflowing: frequency-only not a fix or proof CPU is dominant. User authorized combined-header/payload experiment above; supersedes prior target-pending/next-flash CPU entry below.
|
||||
|
||||
- **240 MHz-only throughput experiment prepared (2026-09-08), user authorized; target pending:** Set `CONFIG_ESP_DEFAULT_CPU_FREQ_MHZ_240=y` in tracked `sdkconfig.defaults` and updated the existing ignored `sdkconfig.esp32-s3-devkitc-1-n16r8` selection/compatibility aliases so the current build actually changes from160 to240MHz. Generated `.pio/build/esp32-s3-devkitc-1-n16r8/config/sdkconfig.h` confirms CPU240 and FreeRTOS100Hz. No transport, priority, affinity, buffer, TCP, asset or task changes. `pio run` PASS56.42s,100,100 B RAM/1,748,161 B flash (unchanged); existing nonfatal SDK FATFS_PRINT_FLOAT bool-default warning. No upload/erase/commit or target throughput-fix claim. Higher power/temperature possible. Next: flash normally, establish and settle full mix including two web serial clients and both admins, then repeat quiet UART0 performance capture and same dmesg burst; compare drops/HWM/send/wait timings and memory, then longer sustained output. Defaults alone do not override an existing sdkconfig on other worktrees; explicitly update its CPU choice too. Rollback requires160MHz in both defaults and active config, then rebuild.
|
||||
|
||||
- **Paired 160 MHz target throughput evidence (2026-09-08):** Full mix UART71,317 B, serial loss/errors0; SSH/USB each read71,317/drop0/HWM704/640, web10 read61,801/drop9,516 and web11 read61,234/drop10,083, both HWM4096. Global queued/read265,669/drop19,599; web253 frames/123,035 B matches browser reads, send/queue/protocol errors0; no disconnects,14 status requests only. Web callback wait averages12,954/13,278us, sends9,315/9,134us, first drain3,422/3,624us. Paired one-browser run UART71,292 B, SSH/USB/web11 each read71,292/drop0, HWM706/449/1536; global213,876/drop0, web222 frames/71,292 B, wait3,338us/send7,604us/first drain2,709us, errors0. One browser disconnected after capture began,20 status requests and one denied writer request: not perfectly settled, but byte accounting exact. Both admin routes retained per user. Loss localized to browser broker queues under shared HTTPD service deficit; elapsed send time is not CPU usage and successful sends are not peer receipt. Idle-inclusive maxima (~0.95s two-browser/~2.59s one-browser) are not scheduler-stall evidence. CPU saturation/TCP blocking not distinguished; single-variable frequency experiment now authorized, superseding diagnostic-only/no-target-evidence wording below. Do not enlarge queues or alter affinity/priority/TCP alongside it.
|
||||
|
||||
- **Web throughput diagnostics handoff (2026-09-08), target unmeasured:** Broker active-client ID/type/pending/HWM/UART/queued/read/dropped rows and independent default-disabled `web performance enable|disable|show|clear` now documented against source. Two fixed web slots, epoch/generation fencing, binary-only queue-entry/callback, synchronous-send and completion/drain timing aggregates; nonempty intervals include idle and are not scheduler-only or proof of completion-time backlog. No CPU160MHz/scheduling/priority/4096-byte broker/512-byte web buffer tuning or new instrumentation allocations. User burst71,292 × four=285,168 expected,265,852 queued,19,316 dropped; browser142,584−123,268 matches, without reported UART loss/disconnect/new login. One-browser no-drop report is not a measured paired sample. Parent broker7/performance suite, console boundary/lifecycle/policy (new SSH-allowed/browser-denied command tests), transport25/tickets12, server lifecycle21, admission diagnostics12+guard, cookie Network+shared PASS; review no confirmed actionable bugs. Parent pio run PASS9.07s,100,100 B RAM/1,748,161 B flash (+552/+3,836 vs Settings presentation), diff check PASS. Fixed extracted console fixture missing new handler double; no production behavior fix. Host sanitizer libraries missing, no sanitizer pass. Real scheduling/overhead/stack margins unmeasured. UART0 quiet/settled full-mix and one-browser capture recipe, before-disconnect attribution, enabled/disabled comparison, 177.8ms retention scale and limits: `docs/web_throughput_diagnostics.md`. Documentation-only pass; no source/tests/assets/device/build/commit action.
|
||||
|
||||
- **8D.12/8D.13 + Settings presentation functionally signed off (2026-09-08):** User explicitly accepts after 60-second boot/full-mix telemetry. Web writer8, web17/SSH10/USB11 observers, both admins, two cookies at230400 8N1 RTS/CTS DTR active. Web send/queue/protocol/close-failure0, SSH handshake/auth/IO failures0; retain SSH rejected1 byte, web rejected1 frame/1 byte, serial connect3/disconnect1, cookie login3/logout1, authfailures2 boot/3 loaded without causal diagnosis. Loaded free internal/DMA/PSRAM31,512/23,756/8,112,140 B; minima2,276/156/8,070,736 B; largest20,480/20,480/7,995,392 B. SSH stack min18,468 boot/16,276 loaded. Exact samples/counters/limits: `docs/phase8d12_13_implementation.md`. Supersedes target-pending/no-signoff wording below for implemented phases and visual refinements. Very low lifetime internal/DMA minima remain transient-headroom follow-up (conservative non-simultaneous region sums, not proof allocation failed); HTTPD/dispatcher floors/peak correlation and numeric reserves unapproved, not reopening functional acceptance. No invented individual checklist/soak/full-M3 pass, no 8D.14 authorization. Documentation only, no source/config/build/test/device/assets/commit action.
|
||||
|
||||
@@ -28,7 +28,7 @@ Only constraints supported by implementation or current project documentation be
|
||||
|
||||
**Consequence for future changes:** Do not replace fan-out with a blocking shared queue. Any added transport must tolerate partial/no-progress reads and expose drop/backpressure counters.
|
||||
|
||||
**Throughput observation, not tuning:** Preserve CPU160MHz, scheduling/priorities and 4096/512-byte broker/web buffers while gathering per-client HWM/drop attribution and independent opt-in web binary-TX timing. Fixed-slot epoch/generation-fenced aggregates avoid stale attribution; no new runtime allocations. Clear preserves queued data and seeds broker HWM; disconnected rows disappear while global discard counts remain. Callback timestamps precede the transport lock; synchronous send return is not peer receipt. Completion-to-read intervals include broker/mutex/control work and possible idle, even when the first read is nonempty; never label them pure scheduling latency or proof of backlog at completion. Compare enabled/disabled target captures before drawing overhead conclusions. Contracts and reproduction: `docs/web_throughput_diagnostics.md`.
|
||||
**Throughput observation and controlled experiments:** The initial diagnostic baseline used CPU160MHz; a CPU240MHz-only experiment reduced but did not eliminate browser queue overflow. Combining binary WebSocket header/payload into one bounded session-override send eliminated reported drops, and the user signed off230400-baud full-client-mix operation after returning to160MHz. Retain the combined send, not the frequency increase; evidence and limits are in `current-state.md`. Preserve scheduling/priorities and 4096/512-byte broker/web buffers while gathering per-client HWM/drop attribution and independent opt-in web binary-TX timing. Fixed-slot epoch/generation-fenced aggregates avoid stale attribution; no new runtime allocations. Clear preserves queued data and seeds broker HWM; disconnected rows disappear while global discard counts remain. Callback timestamps precede the transport lock; synchronous send return is not peer receipt. Completion-to-read intervals include broker/mutex/control work and possible idle, even when the first read is nonempty; never label them pure scheduling latency or proof of backlog at completion. Compare enabled/disabled target captures before drawing overhead conclusions. Contracts and reproduction: `docs/web_throughput_diagnostics.md`.
|
||||
|
||||
**Relevant files:** `src/session_broker.c`, `src/session_broker.h`, `docs/roadmap.md`
|
||||
|
||||
|
||||
@@ -1,4 +1,6 @@
|
||||
# ESP32-S3-WROOM-1-N16R8 hardware configuration
|
||||
# Single-variable web throughput experiment; target validation pending.
|
||||
CONFIG_ESP_DEFAULT_CPU_FREQ_MHZ_160=y
|
||||
CONFIG_ESPTOOLPY_FLASHSIZE_16MB=y
|
||||
CONFIG_SPIRAM=y
|
||||
CONFIG_SPIRAM_MODE_OCT=y
|
||||
|
||||
+49
-1
@@ -11,7 +11,7 @@
|
||||
#include "secure_random.h"
|
||||
|
||||
#if ESP_IDF_VERSION != ESP_IDF_VERSION_VAL(5, 5, 0)
|
||||
#error "Reaudit HTTPD headers, upgrade, request completion and idle cleanup for this IDF"
|
||||
#error "Reaudit HTTPD headers, upgrade, idle cleanup and WS/TLS send contracts for this IDF"
|
||||
#endif
|
||||
|
||||
/* IDF 5.5.0 httpd_sess_process increments lru_counter only AFTER successful
|
||||
@@ -60,6 +60,54 @@ void web_httpd_idle_sweep(httpd_handle_t server,
|
||||
}
|
||||
}
|
||||
|
||||
static int web_httpd_aborted_send(httpd_handle_t server, int fd,
|
||||
const char *buffer, size_t length, int flags)
|
||||
{
|
||||
(void)server; (void)fd; (void)buffer; (void)length; (void)flags;
|
||||
return HTTPD_SOCK_ERR_FAIL;
|
||||
}
|
||||
|
||||
esp_err_t web_httpd_ws_send_binary(httpd_handle_t server, int fd,
|
||||
const void *expected_context,
|
||||
const uint8_t *payload, size_t length)
|
||||
{
|
||||
struct httpd_data *hd = server;
|
||||
if (!hd || fd < 0 || !expected_context || (!payload && length) ||
|
||||
length > WEB_HTTPD_WS_BINARY_MAX_PAYLOAD) return ESP_ERR_INVALID_ARG;
|
||||
if (httpd_os_thread_handle() != hd->hd_td.handle || hd->hd_req_aux.sd)
|
||||
return ESP_ERR_INVALID_STATE;
|
||||
struct sock_db *sd = httpd_sess_get(hd, fd);
|
||||
if (!sd || sd->ctx != expected_context || !sd->ws_handshake_done ||
|
||||
sd->ws_close || sd->for_async_req || !sd->send_fn)
|
||||
return ESP_ERR_INVALID_STATE;
|
||||
|
||||
/* IDF 5.5.0 httpd_ws_send_frame_async emits header/payload separately.
|
||||
* Keep the existing send override (HTTPS -> esp_tls_conn_write), not raw
|
||||
* socket IO. Owner-local scratch lives through the synchronous call only. */
|
||||
uint8_t wire[WEB_HTTPD_WS_BINARY_MAX_PAYLOAD + 4U];
|
||||
size_t header = length <= 125U ? 2U : 4U;
|
||||
wire[0] = 0x82; /* FIN, binary; server frames are never masked. */
|
||||
wire[1] = header == 2U ? (uint8_t)length : 126U;
|
||||
if (header == 4U) {
|
||||
wire[2] = (uint8_t)(length >> 8U);
|
||||
wire[3] = (uint8_t)length;
|
||||
}
|
||||
if (length) memcpy(wire + header, payload, length);
|
||||
size_t total = header + length;
|
||||
int sent = sd->send_fn(hd, fd, (const char *)wire, total, 0);
|
||||
if (sent == (int)total) return ESP_OK;
|
||||
/* Owner retains this validated session across the synchronous send. TLS
|
||||
* may hold pending output after short/zero/WANT/error: never retry it with
|
||||
* different arguments. Deferred close alone permits SDK automatic PONG or
|
||||
* CLOSE first. Shutdown alone cannot block buffered-input TLS calls either.
|
||||
* Reject all sends before shutdown, even if shutdown fails. ws_close also
|
||||
* skips SDK request processing; normal HTTPD still owns TLS destruction. */
|
||||
sd->send_fn = web_httpd_aborted_send;
|
||||
sd->ws_close = true;
|
||||
(void)shutdown(fd, SHUT_RDWR);
|
||||
return ESP_FAIL;
|
||||
}
|
||||
|
||||
bool web_httpd_headers_valid(httpd_req_t *request)
|
||||
{
|
||||
if (!request || !request->aux) return false;
|
||||
|
||||
@@ -26,6 +26,15 @@ void web_httpd_wipe_request(httpd_req_t *request, bool closing);
|
||||
esp_err_t web_httpd_upgrade(httpd_req_t *request,
|
||||
esp_err_t (*handler)(httpd_req_t *));
|
||||
|
||||
#define WEB_HTTPD_WS_BINARY_MAX_PAYLOAD 512U
|
||||
/* HTTPD-owner work only; caller retains generation/lifecycle and one-work-slot
|
||||
* ownership. Synchronous FIN/binary/unmasked send through the session override.
|
||||
* Non-full writes synchronously block further session sends and abort the socket;
|
||||
* caller must retain normal failure/cleanup accounting, never replay the frame. */
|
||||
esp_err_t web_httpd_ws_send_binary(httpd_handle_t server, int fd,
|
||||
const void *expected_context,
|
||||
const uint8_t *payload, size_t length);
|
||||
|
||||
/* Serialized server startup only, exact-match ordinary GET, URI <= 127 bytes.
|
||||
* Stage both allocations before publication; HTTPD owns/frees them on success. */
|
||||
esp_err_t web_httpd_register_optional_get(httpd_handle_t server, const httpd_uri_t *uri);
|
||||
|
||||
@@ -65,6 +65,9 @@ typedef struct {
|
||||
uint32_t generation;
|
||||
} web_serial_work_t;
|
||||
|
||||
_Static_assert(WEB_SERIAL_TRANSPORT_TX_PAYLOAD_SIZE == WEB_HTTPD_WS_BINARY_MAX_PAYLOAD,
|
||||
"Reaudit bounded binary WS send when changing serial payload size");
|
||||
|
||||
typedef struct web_serial_slot {
|
||||
web_serial_slot_state_t state;
|
||||
httpd_handle_t server;
|
||||
@@ -1057,7 +1060,9 @@ static void web_serial_send_work(void *argument)
|
||||
};
|
||||
send_called = true;
|
||||
if (sample_epoch && performance_gate() == sample_epoch) send_start = esp_timer_get_time();
|
||||
result = httpd_ws_send_frame_async(server, socket_fd, &frame);
|
||||
result = type == HTTPD_WS_TYPE_BINARY
|
||||
? web_httpd_ws_send_binary(server, socket_fd, slot, slot->tx_data, length)
|
||||
: httpd_ws_send_frame_async(server, socket_fd, &frame);
|
||||
if (sample_epoch && performance_gate() == sample_epoch) send_end = esp_timer_get_time();
|
||||
if (result == ESP_OK) {
|
||||
(void)httpd_sess_update_lru_counter(server, socket_fd);
|
||||
|
||||
@@ -20,6 +20,7 @@ from serial_headers import SERIAL_HEADERS
|
||||
|
||||
HEADERS.update(SERIAL_HEADERS)
|
||||
HEADERS["esp_http_server.h"] += """
|
||||
#define HTTPD_SOCK_ERR_FAIL -1
|
||||
#define ESP_ERR_HTTPD_INVALID_REQ 0x200
|
||||
#define ESP_ERR_HTTPD_RESULT_TRUNC 0x201
|
||||
#define ESP_ERR_HTTPD_RESP_HDR 0x202
|
||||
@@ -35,6 +36,7 @@ HEADERS["esp_httpd_priv.h"] = """#pragma once
|
||||
#include "esp_http_server.h"
|
||||
static inline void *httpd_os_thread_handle(void) { return (void *)1; }
|
||||
struct sock_db { int fd; bool for_async_req, ws_close; uint64_t lru_counter;
|
||||
void *ctx; int (*send_fn)(httpd_handle_t, int, const char *, size_t, int);
|
||||
int (*pending_fn)(httpd_handle_t, int);
|
||||
bool ws_handshake_done; esp_err_t (*ws_handler)(httpd_req_t *);
|
||||
bool ws_control_frames; void *ws_user_ctx; char pending_data[128]; size_t pending_len; };
|
||||
@@ -44,6 +46,12 @@ struct httpd_req_aux { struct sock_db *sd; char *scratch; size_t scratch_cur_siz
|
||||
struct httpd_data { struct { unsigned max_resp_headers, max_uri_handlers, max_open_sockets; void *uri_match_fn; } config;
|
||||
struct { void *handle; } hd_td; struct httpd_req_aux hd_req_aux; struct sock_db *hd_sd;
|
||||
httpd_uri_t **hd_calls; };
|
||||
static inline struct sock_db *httpd_sess_get(httpd_handle_t h, int fd) {
|
||||
struct httpd_data *hd = h;
|
||||
for (unsigned i = 0; i < hd->config.max_open_sockets; ++i)
|
||||
if (hd->hd_sd[i].fd == fd) return &hd->hd_sd[i];
|
||||
return NULL;
|
||||
}
|
||||
esp_err_t httpd_ws_respond_server_handshake(httpd_req_t *, const char *);
|
||||
"""
|
||||
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
"""Compile actual transport callbacks using the existing serial/store doubles."""
|
||||
import os
|
||||
import pathlib
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
@@ -13,9 +14,32 @@ sys.path.insert(0, str(BASE))
|
||||
from run import HEADERS
|
||||
from serial_headers import SERIAL_HEADERS
|
||||
os.environ['CCACHE_DISABLE'] = '1'
|
||||
IDF = pathlib.Path(os.environ.get('IDF_PATH', str(pathlib.Path.home() / '.platformio/packages/framework-espidf')))
|
||||
|
||||
def function(source, name):
|
||||
match = re.search(r'^(?:static )?(?:esp_err_t|int|ssize_t) ' + name + r'\(.*?^\}', source, re.M | re.S)
|
||||
assert match, name
|
||||
return match.group() + '\n'
|
||||
|
||||
adapter = (ROOT / 'src/web_httpd_adapter.c').read_text()
|
||||
assert 'ESP_IDF_VERSION_VAL(5, 5, 0)' in adapter
|
||||
ws_source = (IDF / 'components/esp_http_server/src/httpd_ws.c').read_text()
|
||||
ws = function(ws_source, 'httpd_ws_send_frame_async')
|
||||
main = (IDF / 'components/esp_http_server/src/httpd_main.c').read_text()
|
||||
assert main.index('/* Case0:') < main.index('httpd_process_ctrl_msg(hd);') < main.index('/* Case1:')
|
||||
parse = (IDF / 'components/esp_http_server/src/httpd_parse.c').read_text()
|
||||
assert parse.index('if (sd->ws_close == true)') < parse.index('ret = httpd_ws_get_frame_type(r);')
|
||||
ssl_msg = (IDF / 'components/mbedtls/mbedtls/library/ssl_msg.c').read_text()
|
||||
assert 'to be called with the same parameters' in ssl_msg
|
||||
assert ws.count('sess->send_fn(') == 2
|
||||
https = function((IDF / 'components/esp_https_server/src/https_server.c').read_text(), 'httpd_ssl_send')
|
||||
assert 'esp_tls_conn_write(tls, buf, buf_len)' in https and 'return ret;' in https
|
||||
tls = function((IDF / 'components/esp-tls/esp_tls_mbedtls.c').read_text(), 'esp_mbedtls_write')
|
||||
assert 'return (written > 0) ? written : ret;' in tls and 'return written;' in tls
|
||||
with tempfile.TemporaryDirectory(prefix='web-performance-') as directory:
|
||||
tmp = pathlib.Path(directory)
|
||||
headers = HEADERS | SERIAL_HEADERS
|
||||
headers['esp_http_server.h'] += '\n#define HTTPD_SOCK_ERR_FAIL -1\n'
|
||||
headers['freertos/FreeRTOS.h'] = headers['freertos/FreeRTOS.h'].replace(
|
||||
'extern int host_lock_depth;', 'extern int host_lock_depth; void host_before_lock(void);').replace(
|
||||
'(void)(m); assert(host_lock_depth++', '(void)(m); host_before_lock(); assert(host_lock_depth++')
|
||||
@@ -27,6 +51,14 @@ with tempfile.TemporaryDirectory(prefix='web-performance-') as directory:
|
||||
fixture = fixture.replace('#include "test.c"', '#include "' + str(BASE / 'test.c') + '"')
|
||||
fixture = fixture.replace('../../src/web_serial_transport.c', str(ROOT / 'src/web_serial_transport.c'))
|
||||
(tmp / 'fixture.c').write_text(fixture)
|
||||
(tmp / 'binary.inc').write_text(function(adapter, 'web_httpd_aborted_send') + function(adapter, 'web_httpd_ws_send_binary'))
|
||||
(tmp / 'sdk_ws.inc').write_text(ws.replace('httpd_ws_send_frame_async(', 'sdk_ws_send_frame('))
|
||||
automatic = ''.join(function(ws_source, name) for name in
|
||||
('httpd_ws_check_req', 'httpd_ws_send_frame', 'httpd_ws_get_frame_type'))
|
||||
automatic = automatic.replace('httpd_ws_send_frame_async(', 'sdk_ws_send_frame(')
|
||||
automatic = automatic.replace('httpd_ws_send_frame(', 'sdk_ws_send_request(')
|
||||
automatic = automatic.replace('httpd_ws_recv_frame(', 'sdk_control_recv(')
|
||||
(tmp / 'sdk_automatic.inc').write_text(automatic)
|
||||
console = (ROOT / 'src/web_console.c').read_text()
|
||||
start = console.index('static void print_performance_time(')
|
||||
end = console.index('static int command_web(', start)
|
||||
@@ -39,3 +71,4 @@ with tempfile.TemporaryDirectory(prefix='web-performance-') as directory:
|
||||
'-Wl,--gc-sections', str(HERE / 'test.c'), str(ROOT / 'src/web_session_store.c'),
|
||||
str(ROOT / 'src/web_auth_parse.c'), '-o', str(tmp / 'test')], check=True, timeout=30)
|
||||
subprocess.run([str(tmp / 'test')], check=True, timeout=15)
|
||||
print('PASS installed IDF WS two-write / HTTPS forwarding / TLS partial-return contract guards')
|
||||
|
||||
@@ -6,7 +6,83 @@ static void *queued_arg;
|
||||
static bool early;
|
||||
static esp_err_t queue_result, send_result;
|
||||
static size_t read_bytes;
|
||||
static unsigned sends;
|
||||
static unsigned sends, control_sends, lru_updates;
|
||||
/* Only private HTTPD layout/owner and the low-level TLS send are doubled. */
|
||||
struct sock_db {
|
||||
int fd;
|
||||
void *ctx;
|
||||
bool ws_handshake_done, ws_close, for_async_req, ws_control_frames;
|
||||
int (*send_fn)(httpd_handle_t, int, const char *, size_t, int);
|
||||
};
|
||||
struct httpd_req_aux {
|
||||
struct sock_db *sd;
|
||||
bool ws_final;
|
||||
httpd_ws_type_t ws_type;
|
||||
};
|
||||
struct httpd_data {
|
||||
struct { unsigned handle; } hd_td;
|
||||
struct { struct sock_db *sd; } hd_req_aux;
|
||||
};
|
||||
static struct httpd_data host_hd;
|
||||
static struct sock_db host_sd;
|
||||
static unsigned owner;
|
||||
static int raw_result;
|
||||
static bool override_result;
|
||||
static uint8_t captured[516];
|
||||
static size_t captured_length;
|
||||
static unsigned httpd_os_thread_handle(void) { return owner; }
|
||||
static struct sock_db *httpd_sess_get(httpd_handle_t h, int fd)
|
||||
{ assert(h == &host_hd); return fd == host_sd.fd ? &host_sd : NULL; }
|
||||
static unsigned shutdown_calls, shutdown_delay;
|
||||
static int shutdown_result;
|
||||
#define SHUT_RDWR 2
|
||||
static int host_shutdown(int fd, int how)
|
||||
{
|
||||
assert(!host_lock_depth && owner == host_hd.hd_td.handle);
|
||||
assert(fd == host_sd.fd && how == SHUT_RDWR && host_sd.ws_close);
|
||||
/* The send barrier must already hold even when shutdown fails. */
|
||||
assert(host_sd.send_fn(&host_hd, fd, "ignored", 7, 0) == HTTPD_SOCK_ERR_FAIL);
|
||||
++shutdown_calls; now += shutdown_delay;
|
||||
return shutdown_result;
|
||||
}
|
||||
#define shutdown host_shutdown
|
||||
#include "binary.inc"
|
||||
#undef shutdown
|
||||
#define HTTPD_WS_FIN_BIT 0x80
|
||||
#define HTTPD_WS_CONTINUE 0
|
||||
#define HTTPD_WS_MASK_BIT 0x80
|
||||
#ifndef ESP_LOGW
|
||||
#define ESP_LOGW(...) ((void)0)
|
||||
#endif
|
||||
#include "sdk_ws.inc"
|
||||
#define HTTPD_WS_OPCODE_BITS 0x0f
|
||||
#define HTTPD_WS_TYPE_CLOSE 8
|
||||
#define HTTPD_WS_TYPE_PING 9
|
||||
#define HTTPD_WS_TYPE_PONG 10
|
||||
static uint8_t incoming_opcode;
|
||||
static unsigned automatic_reads;
|
||||
static int httpd_recv_with_opt(httpd_req_t *r, char *out, size_t n, bool peek)
|
||||
{
|
||||
(void)r; assert(n == 1 && !peek); *out = 0x80 | incoming_opcode;
|
||||
++automatic_reads; return 1;
|
||||
}
|
||||
static esp_err_t sdk_control_recv(httpd_req_t *r, httpd_ws_frame_t *f, size_t n)
|
||||
{
|
||||
(void)r; assert(n == 126); f->len = 1; f->payload[0] = 0x5a;
|
||||
++automatic_reads; return ESP_OK;
|
||||
}
|
||||
#define ESP_LOGD(...) ((void)0)
|
||||
#include "sdk_automatic.inc"
|
||||
|
||||
static int tls_send(httpd_handle_t h, int fd, const char *data, size_t n, int flags)
|
||||
{
|
||||
assert(!host_lock_depth && h == &host_hd && fd == host_sd.fd && flags == 0);
|
||||
assert(owner == host_hd.hd_td.handle && captured_length + n <= sizeof(captured));
|
||||
memcpy(captured + captured_length, data, n); captured_length += n;
|
||||
++sends; now += 11;
|
||||
if (send_hook) send_hook();
|
||||
return override_result ? raw_result : send_result == ESP_OK ? (int)n : -1;
|
||||
}
|
||||
static unsigned lock_delay;
|
||||
static bool disconnect_stalled;
|
||||
void host_before_lock(void) { now += lock_delay; lock_delay = 0; }
|
||||
@@ -25,18 +101,19 @@ esp_err_t httpd_queue_work(httpd_handle_t h, void (*fn)(void *), void *arg)
|
||||
(void)h; assert(!host_lock_depth);
|
||||
queued = fn; queued_arg = arg;
|
||||
if (queue_hook) queue_hook();
|
||||
if (early && queue_result == ESP_OK) { now += 7; fn(arg); }
|
||||
if (early && queue_result == ESP_OK) { captured_length = 0; now += 7; fn(arg); }
|
||||
return queue_result;
|
||||
}
|
||||
esp_err_t httpd_ws_send_frame_async(httpd_handle_t h, int fd, httpd_ws_frame_t *f)
|
||||
{
|
||||
(void)h; (void)fd; assert(!host_lock_depth); assert(f->payload == s_slots[0].tx_data);
|
||||
++sends; now += 11;
|
||||
assert(f->type == HTTPD_WS_TYPE_TEXT);
|
||||
++control_sends; now += 11;
|
||||
if (send_hook) send_hook();
|
||||
return send_result;
|
||||
}
|
||||
esp_err_t httpd_sess_update_lru_counter(httpd_handle_t h, int fd)
|
||||
{ (void)h; (void)fd; assert(!host_lock_depth); now += 3; return ESP_OK; }
|
||||
{ (void)h; (void)fd; assert(!host_lock_depth); ++lru_updates; now += 3; return ESP_OK; }
|
||||
esp_err_t session_broker_read(session_broker_client_id_t id, uint8_t *data, size_t n, size_t *out)
|
||||
{
|
||||
(void)id; (void)data; assert(!host_lock_depth && read_bytes <= n);
|
||||
@@ -45,7 +122,13 @@ esp_err_t session_broker_read(session_broker_client_id_t id, uint8_t *data, size
|
||||
}
|
||||
static web_serial_slot_t *setup(void)
|
||||
{
|
||||
request.handle = &host_hd;
|
||||
serial_reset(); issued_t a = mint(&alice); web_serial_slot_t *s = connect_session(&a);
|
||||
host_hd = (struct httpd_data){.hd_td.handle = 1}; owner = 1;
|
||||
host_sd = (struct sock_db){.fd = 10, .ctx = s, .ws_handshake_done = true, .send_fn = tls_send};
|
||||
override_result = false; captured_length = 0; control_sends = 0; lru_updates = 0;
|
||||
shutdown_calls = shutdown_delay = automatic_reads = 0; shutdown_result = 0;
|
||||
memset(&s_counters, 0, sizeof(s_counters));
|
||||
early = false; disconnect_stalled = false; lock_delay = 0; queue_result = send_result = ESP_OK;
|
||||
send_hook = queue_hook = read_hook = NULL; read_bytes = 0; sends = 0;
|
||||
assert(web_serial_performance_enable(false) == ESP_OK);
|
||||
@@ -56,7 +139,7 @@ static void enable(void) { assert(web_serial_performance_enable(true) == ESP_OK)
|
||||
static void clear(void) { assert(web_serial_performance_clear() == ESP_OK); }
|
||||
static void disable(void) { assert(web_serial_performance_enable(false) == ESP_OK); }
|
||||
static void cycle(void) { disable(); enable(); }
|
||||
static void run_work(void) { assert(queued); queued(queued_arg); }
|
||||
static void run_work(void) { assert(queued); captured_length = 0; queued(queued_arg); }
|
||||
static void queue_binary(web_serial_slot_t *s)
|
||||
{ assert(queue_slot_frame(s, s->generation, HTTPD_WS_TYPE_BINARY, 512) == ESP_OK); }
|
||||
static web_serial_performance_session_t row(void)
|
||||
@@ -65,9 +148,133 @@ static web_serial_performance_session_t row(void)
|
||||
assert(web_serial_performance_snapshot(&s) == ESP_OK);
|
||||
return s.sessions[0];
|
||||
}
|
||||
static void binary_tests(void)
|
||||
{
|
||||
/* Compare every supported length and arbitrary bytes with installed IDF's
|
||||
* actual serializer, but require one override call instead of two. */
|
||||
for (size_t length = 0; length <= 512; ++length) {
|
||||
web_serial_slot_t *s = setup();
|
||||
for (size_t i = 0; i < sizeof(s->tx_data); ++i) s->tx_data[i] = (uint8_t)i;
|
||||
httpd_ws_frame_t frame = {.final = true, .type = HTTPD_WS_TYPE_BINARY,
|
||||
.payload = s->tx_data, .len = length};
|
||||
assert(sdk_ws_send_frame(request.handle, 10, &frame) == ESP_OK);
|
||||
uint8_t expected[516]; size_t expected_length = captured_length;
|
||||
memcpy(expected, captured, expected_length);
|
||||
assert(sends == (length ? 2U : 1U));
|
||||
captured_length = 0; sends = 0;
|
||||
assert(web_httpd_ws_send_binary(request.handle, 10, s, s->tx_data, length) == ESP_OK);
|
||||
assert(sends == 1 && captured_length == expected_length);
|
||||
assert(!memcmp(captured, expected, expected_length));
|
||||
for (size_t i = 0; i < sizeof(s->tx_data); ++i) assert(s->tx_data[i] == (uint8_t)i);
|
||||
}
|
||||
/* Every short result, including an entirely consumed header, fails closed.
|
||||
* No second call/replay, no successful byte/frame or LRU accounting. */
|
||||
for (int result = -4; result <= 517; ++result) {
|
||||
if (result == 516) continue;
|
||||
web_serial_slot_t *s = setup(); enable(); override_result = true; raw_result = result;
|
||||
queue_binary(s); run_work();
|
||||
assert(sends == 1 && !control_sends && s->close_requested && !s->work_pending);
|
||||
assert(row().send_errors == 1 && row().send_call.sum_us == 11 && !row().sent_frames);
|
||||
assert(s_counters.send_failures == 1 && !s_counters.tx_binary_bytes && !lru_updates);
|
||||
assert(queue_slot_frame(s, s->generation, HTTPD_WS_TYPE_BINARY, 1) == ESP_ERR_INVALID_STATE);
|
||||
run_work(); assert(sends == 1);
|
||||
}
|
||||
const int tls_errors[] = {-0x6880, -0x6900, -0x0050, -0x004e};
|
||||
for (size_t i = 0; i < sizeof(tls_errors)/sizeof(tls_errors[0]); ++i) {
|
||||
web_serial_slot_t *s = setup(); enable(); override_result = true; raw_result = tls_errors[i];
|
||||
queue_binary(s); run_work(); assert(sends == 1 && s->close_requested && row().send_errors == 1);
|
||||
}
|
||||
for (unsigned invalid = 0; invalid < 7; ++invalid) {
|
||||
web_serial_slot_t *s = setup(); enable(); queue_binary(s);
|
||||
switch (invalid) {
|
||||
case 0: owner = 2; break;
|
||||
case 1: host_hd.hd_req_aux.sd = &host_sd; break;
|
||||
case 2: host_sd.fd = 11; break;
|
||||
case 3: host_sd.ctx = &s_slots[1]; break;
|
||||
case 4: host_sd.ws_handshake_done = false; break;
|
||||
case 5: host_sd.ws_close = true; break;
|
||||
case 6: host_sd.for_async_req = true; break;
|
||||
}
|
||||
run_work(); assert(!sends && s->close_requested && row().send_errors == 1);
|
||||
assert(!shutdown_calls && host_sd.send_fn == tls_send);
|
||||
}
|
||||
web_serial_slot_t *s = setup();
|
||||
host_sd.send_fn = NULL;
|
||||
assert(web_httpd_ws_send_binary(request.handle, 10, s, s->tx_data, 1) == ESP_ERR_INVALID_STATE);
|
||||
host_sd.send_fn = tls_send;
|
||||
assert(web_httpd_ws_send_binary(NULL, 10, s, s->tx_data, 1) == ESP_ERR_INVALID_ARG);
|
||||
assert(web_httpd_ws_send_binary(request.handle, -1, s, s->tx_data, 1) == ESP_ERR_INVALID_ARG);
|
||||
assert(web_httpd_ws_send_binary(request.handle, 10, NULL, s->tx_data, 1) == ESP_ERR_INVALID_ARG);
|
||||
assert(web_httpd_ws_send_binary(request.handle, 10, s, NULL, 1) == ESP_ERR_INVALID_ARG);
|
||||
assert(web_httpd_ws_send_binary(request.handle, 10, s, s->tx_data, 513) == ESP_ERR_INVALID_ARG);
|
||||
assert(!sends);
|
||||
s = setup(); enable(); queue_binary(s);
|
||||
assert(queue_slot_frame(s, s->generation, HTTPD_WS_TYPE_BINARY, 512) == ESP_ERR_INVALID_STATE);
|
||||
run_work(); assert(sends == 1 && row().sent_bytes == 512 && lru_updates == 1);
|
||||
assert(queue_slot_frame(s, s->generation, HTTPD_WS_TYPE_TEXT, 4) == ESP_OK); run_work();
|
||||
assert(control_sends == 1 && sends == 1 && row().sent_frames == 1);
|
||||
/* Both fixed slots can retain one independent payload. HTTPD still sends
|
||||
* serially; failure of the first must not retire the second reservation. */
|
||||
s = setup(); issued_t other = mint(&bob); web_serial_slot_t *second = connect_session(&other);
|
||||
memset(s->tx_data, 0xa5, sizeof(s->tx_data));
|
||||
memset(second->tx_data, 0x5a, sizeof(second->tx_data));
|
||||
queue_binary(s); queue_binary(second);
|
||||
request.sess_ctx = s; host_sd.ctx = s; override_result = true; raw_result = 2;
|
||||
web_serial_send_work(&s->work);
|
||||
assert(s->close_requested && second->work_pending && !second->close_requested);
|
||||
captured_length = 0; request.sess_ctx = second; override_result = false;
|
||||
host_sd = (struct sock_db){.fd = 10, .ctx = second, .ws_handshake_done = true, .send_fn = tls_send};
|
||||
web_serial_send_work(&second->work);
|
||||
assert(sends == 2 && !second->work_pending && !second->close_requested);
|
||||
for (size_t i = 4; i < captured_length; ++i) assert(captured[i] == 0x5a);
|
||||
assert(s_counters.tx_binary_bytes == 512 && s_counters.send_failures == 1);
|
||||
printf("PASS binary single-write: 513 SDK wire comparisons, short/zero/TLS errors, owner/session gates, two-slot isolation, one outstanding, text isolation\n");
|
||||
}
|
||||
|
||||
static void abort_tests(void)
|
||||
{
|
||||
const int results[] = {-0x6880, -0x6900, -0x0050, 0, 2, 515};
|
||||
for (unsigned i = 0; i < sizeof(results)/sizeof(results[0]); ++i) {
|
||||
for (int failure = 0; failure < 2; ++failure) {
|
||||
web_serial_slot_t *s = setup(); enable();
|
||||
override_result = true; raw_result = results[i];
|
||||
shutdown_result = failure ? -1 : 0; shutdown_delay = 5;
|
||||
unsigned closes_before = closes;
|
||||
queue_binary(s); run_work();
|
||||
assert(sends == 1 && shutdown_calls == 1 && closes == closes_before);
|
||||
assert(s->close_requested && !s->work_pending && host_sd.ws_close);
|
||||
assert(row().send_call.sum_us == 16 && row().send_errors == 1 && !lru_updates);
|
||||
/* Deliberately invoke actual SDK automatic control processing even
|
||||
* though ws_close would normally skip it in httpd_req_new. Model
|
||||
* buffered input after shutdown, including failed shutdown. No
|
||||
* deferred close or transport-owner iteration has run yet. */
|
||||
struct httpd_req_aux aux = {.sd = &host_sd};
|
||||
httpd_req_t req = {.handle = &host_hd, .aux = &aux};
|
||||
const uint8_t opcodes[] = {HTTPD_WS_TYPE_PING, HTTPD_WS_TYPE_CLOSE};
|
||||
override_result = false; captured_length = 0;
|
||||
for (unsigned j = 0; j < sizeof(opcodes); ++j) {
|
||||
incoming_opcode = opcodes[j];
|
||||
assert(httpd_ws_get_frame_type(&req) == ESP_FAIL);
|
||||
assert(sends == 1 && !captured_length && closes == closes_before);
|
||||
assert(aux.ws_type == incoming_opcode && aux.ws_final);
|
||||
}
|
||||
assert(automatic_reads == 4 && shutdown_calls == 1);
|
||||
/* Ordinary HTTPD teardown/new-session initialization resets the
|
||||
* per-session override: no global fd poison survives reuse. */
|
||||
host_sd = (struct sock_db){.fd = 10, .ctx = &s_slots[1],
|
||||
.ws_handshake_done = true, .send_fn = tls_send};
|
||||
incoming_opcode = HTTPD_WS_TYPE_PING;
|
||||
assert(httpd_ws_get_frame_type(&req) == ESP_OK && sends == 3);
|
||||
}
|
||||
}
|
||||
puts("PASS owner abort before return: WANT/error/zero/short, shutdown success/failure, actual SDK automatic PING/CLOSE blocked before deferred cleanup, fd reuse");
|
||||
}
|
||||
|
||||
int main(void)
|
||||
{
|
||||
assert(serial_tests() == 0);
|
||||
binary_tests();
|
||||
abort_tests();
|
||||
web_serial_slot_t *s = setup();
|
||||
queue_binary(s); run_work(); assert(row().sent_frames == 0); /* default off */
|
||||
enable(); early = true; queue_binary(s);
|
||||
|
||||
Reference in New Issue
Block a user