Commander1024/ESP32_Serial_Swiss_Army_Knife
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Migrate to IDF 5.5.3 candidate

Browse Source 
Pin PlatformIO packages and toolchains, rebase protected SDK
overrides, and add WebSocket receive regression coverage. Document
isolated candidate validation, archive provenance, and remaining gates.
This commit is contained in:
Commander1024
2026-09-18 14:23:13 +02:00
parent cdc4d4a8df
commit 797d2681ac
32 changed files with 1394 additions and 123 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/sdk_security_overrides/README.md
+41 -4
View File
@@ -7,7 +7,7 @@ client ciphersuite setting is changed.
## Build contract
`tools/security_overrides.py` requires the installed ESP-IDF version header to
identify **5.5.0**, and checks each complete original source against its reviewed
identify **5.5.3**, and checks each complete original source against its reviewed
SHA256. Every text substitution must match **exactly once**. The entire input
plan is validated before any output is written. A changed SDK, missing source,
ambiguous edit, duplicate source, or missing/ambiguous component target fails
@@ -80,8 +80,9 @@ Source generator expressions are rejected rather than guessed through.
[bfaf4a47fd33da860796feaba6235847acb71127](https://github.com/Mbed-TLS/mbedtls/commit/bfaf4a47fd33da860796feaba6235847acb71127.patch).
These three patches were fetched from the official repositories and compared
with the installed pinned sources on 2026-09-15. No dependency versions or
existing original-source hashes changed. The optional WS subprotocol backport
with the installed pinned sources on 2026-09-15. That original implementation did not change dependency versions or source hashes;
the current 5.5.3 rebase has separately reviewed original hashes (see
`docs/idf_553_rebase_review.md`). The optional WS subprotocol backport
and separate ASN.1 repeated-OID/empty-value correction are **not** implemented.
Clients that cannot negotiate this server profile will no longer connect.
@@ -109,7 +110,7 @@ entry tuple as well as the default registry. The manifest maps each entry to
its component, optional nested target, original and derived source. CMake's
`sak_security_replace_source(component original generated nested_target)` handles
replacement on the actual source owner, not the IDF mbedTLS wrapper. The current
registry has six IDF sources and one project-managed wolfSSH source. Update the
registry has seven IDF C sources, two project-managed wolfSSH C sources and one wolfSSH header overlay. Update the
corresponding library-specific feature/behavior tests when extending the registry. Multiple
sources in the same real component are supported by the replacement function.
@@ -130,6 +131,42 @@ old generated copies lack the new entries/notice and must not pass. It checks re
Ninja registration: exactly one compilation of each derived source, no original
compilation, and exact generated bytes. It does **not** run a firmware build.
### Candidate WebSocket receive regression
```sh
CCACHE_DISABLE=1 python3 -B tests/sdk_security_overrides/run.py --idf-path .pio/idf-candidate-5.5.3/core/packages/framework-espidf
CCACHE_DISABLE=1 python3 -B tests/web_serial_performance/run.py --idf-path .pio/idf-candidate-5.5.3/core/packages/framework-espidf
```
Either runner accepts `--build-dir .pio/idf-candidate-5.5.3/app/.pio/build/esp32-s3-devkitc-1-n16r8`
for strict existing-build evidence. The performance runner otherwise renders the
current hash-verified WS override; it never falls back to vendor WS code.
Its build mode verifies exactly one generated compilation input, its HTTPD owner,
original-source absence and byte equality with the current override. A stale build
missing the WS replacement must fail; parent reconfiguration/build is separate.
`ws.c` executes complete generated `httpd_ws_get_frame_type`,
`httpd_ws_recv_frame`, unmask/check/send-wrapper functions and complete hash-pinned
vendor `httpd_recv_with_opt`/`httpd_recv_pending`, not copied conditional snippets.
The send endpoint and socket receive callback are bounded doubles. Its 982 cases
cover all five header sites (first/second byte, 2/8-byte length, 4-byte mask),
negative failure/timeout, EOF, every shorter prefix, split socket reads, pending
prefixes plus timeout (positive short returns), successful binary decode, length
probe/resume, nontrivial extended lengths, automatic PING/PONG and CLOSE.
Input ends at a guard page; output/mask canaries, exact byte consumption and
send/callback counts enforce no payload read or output after framing failure.
UBSan trap instrumentation is enabled.
The tests deliberately do **not** assert transactional rollback: failed first-byte
reads return `ESP_OK` with final/CLOSE metadata; other failed reads retain already
decoded type/final/length or partially received mask bytes. They assert that exact
state and that no automatic reply follows failed control framing. Five individual
cast removals must fail strict compilation with `sign-compare`; five explicit
unsigned-promotion equivalents must compile with warnings-as-errors and then fail
behavioral assertions. No diagnostic suppression is used for these mutations.
This is host framing evidence, not live sockets, target ABI/timing or firmware
integration evidence.
Coverage:
- Generator full-source hashes, version, missing/duplicate/ambiguous inputs,
tests/sdk_security_overrides/run.py
+43 -4
View File
@@ -41,8 +41,8 @@ def source_path(entry, idf, project=ROOT):
AUXILIARY = {
"components/esp_http_server/src/httpd_main.c": "a16ef65069dda13889c67b922f25eb566573983d6c24f01c089a902d5fd26149",
"components/esp_http_server/src/httpd_txrx.c": "7659ad52c32f29b9a08208dc8b22d023edf274047835ed58107d82a47ccce00e",
"components/esp_http_server/src/httpd_main.c": "55fccf1ec01265dd9be45609c4d8eeb5d9f80da99fee23b2309a6b372c56e24f",
"components/esp_http_server/src/httpd_txrx.c": "f0978034ae0acc7e5f5c52fcb4aabc424afd403d433a417e04ff70319a7dd140",
}
FEATURES = ["MBEDTLS_SSL_PROTO_TLS1_2", "MBEDTLS_SSL_SRV_C",
"MBEDTLS_KEY_EXCHANGE_ECDHE_ECDSA_ENABLED", "MBEDTLS_ECDH_C",
@@ -145,8 +145,8 @@ def generator_tests(idf, work):
expect_error(lambda: sdk.generate(fake, ROOT, binary), "SHA256 mismatch")
assert before == {p: (p.read_bytes(), p.stat().st_mtime_ns) for p in before}
shutil.copyfile(idf / TLS_ENTRY.source, last)
(fake / version).write_text((fake / version).read_text().replace("VERSION_PATCH 0", "VERSION_PATCH 1"))
expect_error(lambda: sdk.generate(fake, ROOT, failed), "5.5.0")
(fake / version).write_text((fake / version).read_text().replace("VERSION_PATCH 3", "VERSION_PATCH 1"))
expect_error(lambda: sdk.generate(fake, ROOT, failed), "5.5.3")
shutil.copyfile(idf / version, fake / version)
last.unlink()
expect_error(lambda: sdk.generate(fake, ROOT, failed), "No such file")
@@ -215,6 +215,44 @@ def extracted_tests(idf, binary, work):
"-fsanitize=undefined", "-fsanitize-undefined-trap-on-error"])
def websocket_tests(idf, binary, work):
entry = next(e for e in sdk.ENTRIES if e.name == "httpd_ws")
text = generated_path(binary, entry).read_text()
assert text.encode() == sdk.render_entry(entry, {"idf": idf, "project": ROOT})[1]
txrx = (idf / "components/esp_http_server/src/httpd_txrx.c").read_text()
assert hashlib.sha256(txrx.encode()).hexdigest() == AUXILIARY["components/esp_http_server/src/httpd_txrx.c"]
private = (idf / "components/esp_http_server/src/esp_httpd_priv.h").read_text()
options = re.search(r"typedef enum \{[^}]*\} httpd_recv_opt_t;", private)
assert options
functions = "".join(extract(txrx, n) for n in ("httpd_recv_pending", "httpd_recv_with_opt"))
functions += "".join(extract(text, n) for n in (
"httpd_ws_check_req", "httpd_ws_unmask_payload", "httpd_ws_recv_frame",
"httpd_ws_send_frame", "httpd_ws_get_frame_type"))
template = (HERE / "ws.c").read_text().replace("/* SDK_OPTIONS */", options.group())
source = template.replace("/* SDK_FUNCTIONS */", functions)
flags = ["-fsanitize=undefined", "-fsanitize-undefined-trap-on-error"]
compile_run("ws", source, work, flags)
# Independently enumerate all five sites, rather than trusting the edit registry.
sites = list(re.finditer(r"httpd_recv_with_opt\([^\n]+HTTPD_RECV_OPT_BLOCKING\) < \(int\)sizeof\(([^)]+)\)", functions))
assert [m[1] for m in sites] == ["second_byte", "length_bytes", "length_bytes", "aux->mask_key", "first_byte"]
for i, match in enumerate(sites):
removed = functions[:match.start()] + match[0].replace("(int)sizeof", "sizeof") + functions[match.end():]
c = work / f"ws_removed_cast_{i}.c"
c.write_text(template.replace("/* SDK_FUNCTIONS */", removed))
output = run(["cc", "-std=gnu11", "-Wall", "-Wextra", "-Werror", "-fsyntax-only", c], ok=False)
assert "sign-compare" in output, output
# Explicit unsigned conversion reproduces the vendor's implicit promotion
# without disabling sign-compare diagnostics: rejection must be behavioral.
mutant = functions[:match.start()] + "(size_t)" + match[0].replace("(int)sizeof", "sizeof") + functions[match.end():]
c = work / f"ws_mutant_{i}.c"
exe = work / f"ws_mutant_{i}"
c.write_text(template.replace("/* SDK_FUNCTIONS */", mutant))
run(["cc", "-std=gnu11", "-O2", "-Wall", "-Wextra", "-Werror", *flags, c, "-o", exe])
output = run([exe], ok=False)
assert "Assertion" in output or "assertion" in output, output
print("WS five removed casts rejected by strict compilation and five unsigned-comparison behavioral mutations rejected PASS")
def compile_run(name, source, work, flags=()):
c = work / (name + ".c"); exe = work / name
c.write_text("/* Extracted SDK sections retain their upstream Apache-2.0 license. */\n" + source)
@@ -410,6 +448,7 @@ def main():
work = Path(tmp)
binary = generator_tests(idf, work)
extracted_tests(idf, binary, work)
websocket_tests(idf, binary, work)
cmake_fixture_tests(idf, work)
extension_fixture_tests(idf, work)
if args.build_dir: build_registration(args.build_dir.resolve(), idf)
tests/sdk_security_overrides/ws.c
+215
View File
@@ -0,0 +1,215 @@
/* SPDX-License-Identifier: GPL-3.0-only */
#define _GNU_SOURCE
#include <assert.h>
#include <stdbool.h>
#include <stdint.h>
#include <stdio.h>
#include <string.h>
#include <sys/mman.h>
#include <unistd.h>
typedef int esp_err_t;
typedef void *httpd_handle_t;
typedef int httpd_ws_type_t;
#define ESP_OK 0
#define ESP_FAIL -1
#define ESP_ERR_INVALID_ARG -2
#define ESP_ERR_INVALID_STATE -3
#define ESP_ERR_INVALID_SIZE -4
#define HTTPD_SOCK_ERR_TIMEOUT -3
#define HTTPD_WS_FIN_BIT 0x80
#define HTTPD_WS_MASK_BIT 0x80
#define HTTPD_WS_LENGTH_BITS 0x7f
#define HTTPD_WS_OPCODE_BITS 0x0f
#define HTTPD_WS_TYPE_CLOSE 8
#define HTTPD_WS_TYPE_PING 9
#define HTTPD_WS_TYPE_PONG 10
#define MIN(a,b) ((a) < (b) ? (a) : (b))
#define ESP_LOGW(...) ((void)0)
#define ESP_LOGD(...) ((void)0)
/* SDK_OPTIONS */
struct sock_db {
httpd_handle_t handle;
int fd;
bool ws_handshake_done, ws_control_frames;
unsigned char pending_data[32];
size_t pending_len;
int (*recv_fn)(httpd_handle_t, int, char *, size_t, int);
};
struct httpd_req_aux {
struct sock_db *sd;
bool ws_final;
httpd_ws_type_t ws_type;
unsigned char before[8], mask_key[4], after[8];
};
typedef struct { void *aux; httpd_handle_t handle; } httpd_req_t;
typedef struct {
bool final, fragmented;
httpd_ws_type_t type;
unsigned char *payload;
size_t len;
} httpd_ws_frame_t;
static unsigned char *wire;
static size_t position, limit, chunk;
static int terminal_result, terminal_calls, sends, sent_type;
static size_t sent_len;
static unsigned char sent_payload[128];
static int receive(httpd_handle_t h, int fd, char *buf, size_t len, int flags)
{
(void)h; (void)fd; (void)flags;
assert(len > 0);
if (position == limit) {
/* Any continuation past failed framing is an observable failure. */
assert(terminal_calls++ == 0);
return terminal_result;
}
size_t n = MIN(len, MIN(chunk, limit - position));
memcpy(buf, wire + position, n);
position += n;
return (int)n;
}
static int httpd_req_to_sockfd(httpd_req_t *req) { (void)req; return 7; }
static esp_err_t httpd_ws_send_frame_async(httpd_handle_t h, int fd, httpd_ws_frame_t *f)
{
(void)h; assert(fd == 7); assert(f->len <= sizeof(sent_payload));
sends++; sent_type = f->type; sent_len = f->len;
if (f->len) memcpy(sent_payload, f->payload, f->len);
return ESP_OK;
}
/* SDK_FUNCTIONS */
static struct sock_db sd;
static struct httpd_req_aux aux;
static httpd_req_t req;
static unsigned char packet[32];
static size_t packet_len, header_len;
static const unsigned char plain[] = {0, 0xff, 0x80};
static void setup(int encoding, int opcode, size_t available, int result, size_t split, size_t pending)
{
memset(&sd, 0, sizeof(sd)); memset(&aux, 0, sizeof(aux));
memset(aux.before, 0xa5, sizeof(aux.before));
memset(aux.mask_key, 0xcc, sizeof(aux.mask_key));
memset(aux.after, 0x5a, sizeof(aux.after));
sd.ws_handshake_done = true; sd.recv_fn = receive; aux.sd = &sd;
aux.ws_type = 2; req.aux = &aux;
size_t n = 0;
packet[n++] = 0x80 | opcode;
packet[n++] = 0x80 | (encoding ? (encoding == 2 ? 126 : 127) : sizeof(plain));
for (int i = 0; i < encoding; i++) packet[n++] = i == encoding - 1 ? sizeof(plain) : 0;
for (int i = 0; i < 4; i++) packet[n++] = (unsigned char)(0x10 + i);
header_len = n;
for (size_t i = 0; i < sizeof(plain); i++) packet[n++] = plain[i] ^ (0x10 + i);
packet_len = n;
limit = MIN(available, n); position = pending; chunk = split;
assert(pending <= limit && pending <= sizeof(sd.pending_data));
memcpy(sd.pending_data + sizeof(sd.pending_data) - pending, packet, pending);
sd.pending_len = pending;
/* Input ends exactly at an inaccessible page, including partial headers. */
long page = sysconf(_SC_PAGESIZE);
wire = mmap(NULL, (size_t)page * 2, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
assert(wire != MAP_FAILED);
assert(mprotect(wire + page, (size_t)page, PROT_NONE) == 0);
wire += page - limit;
memcpy(wire, packet, limit);
terminal_result = result; terminal_calls = sends = 0;
}
static void finish(void)
{
for (size_t i = 0; i < 8; i++) assert(aux.before[i] == 0xa5 && aux.after[i] == 0x5a);
long page = sysconf(_SC_PAGESIZE);
assert(munmap(wire + limit - page, (size_t)page * 2) == 0);
}
static unsigned checks;
static void failure(int encoding, int stage, size_t partial, int error, bool pending, int opcode)
{
/* stage: first byte, second byte, extended length, mask. */
size_t start = stage == 0 ? 0 : stage == 1 ? 1 : stage == 2 ? 2 : 2 + encoding;
setup(encoding, opcode, start + partial, error, 1, pending ? start + partial : 0);
unsigned char output[5]; memset(output, 0xab, sizeof(output));
httpd_ws_frame_t frame = {.payload = output + 1};
esp_err_t ret = httpd_ws_get_frame_type(&req);
if (stage == 0) {
assert(ret == ESP_OK && aux.ws_final && aux.ws_type == HTTPD_WS_TYPE_CLOSE);
assert(frame.len == 0);
} else if (opcode == 2) {
assert(ret == ESP_OK && aux.ws_type == 2 && aux.ws_final);
assert(httpd_ws_recv_frame(&req, &frame, 3) == ESP_FAIL);
assert(frame.type == 2 && frame.final);
assert(frame.len == (stage == 3 ? 3u : 0u));
} else {
assert(ret == ESP_ERR_INVALID_STATE);
assert(aux.ws_type == opcode && aux.ws_final);
}
assert(position == limit && terminal_calls == 1 && sends == 0);
assert(sd.pending_len == 0 && sd.ws_handshake_done && !sd.ws_control_frames);
assert(req.aux == &aux && aux.sd == &sd);
for (size_t i = 0; i < sizeof(output); i++) assert(output[i] == 0xab);
for (size_t i = 0; i < sizeof(aux.mask_key); i++)
assert(aux.mask_key[i] == (stage == 3 && i < partial ? 0x10 + i : 0xcc));
finish(); checks++;
}
static void valid(int encoding, size_t split, size_t pending, int opcode, bool probe)
{
setup(encoding, opcode, 32, 0, split, pending);
unsigned char output[5]; memset(output, 0xab, sizeof(output));
httpd_ws_frame_t frame = {.payload = output + 1};
assert(httpd_ws_get_frame_type(&req) == ESP_OK);
if (opcode == 2) {
if (probe) {
assert(httpd_ws_recv_frame(&req, &frame, 0) == ESP_OK);
assert(frame.len == 3 && position == header_len);
assert(output[1] == 0xab);
}
assert(httpd_ws_recv_frame(&req, &frame, 3) == ESP_OK);
assert(frame.len == 3 && frame.type == 2 && frame.final);
assert(memcmp(output + 1, plain, 3) == 0 && sends == 0);
} else {
assert(sends == 1 && sent_type == (opcode == 9 ? 10 : 8));
assert(sent_len == (opcode == 9 ? 3u : 0u));
if (opcode == 9) assert(memcmp(sent_payload, plain, 3) == 0);
}
assert(output[0] == 0xab && output[4] == 0xab);
assert(position == packet_len && sd.pending_len == 0 && terminal_calls == 0);
finish(); checks++;
}
static void extended_length(int encoding, uint64_t length)
{
setup(encoding, 2, (size_t)(6 + encoding), 0, 1, 0);
for (int i = 0; i < encoding; i++)
wire[2 + i] = (unsigned char)(length >> (8 * (encoding - i - 1)));
httpd_ws_frame_t frame = {0};
assert(httpd_ws_get_frame_type(&req) == ESP_OK);
assert(httpd_ws_recv_frame(&req, &frame, 0) == ESP_OK);
assert(frame.len == length && frame.type == 2 && frame.final);
assert(position == header_len && sends == 0 && terminal_calls == 0);
finish(); checks++;
}
int main(void)
{
const int errors[] = {-1, HTTPD_SOCK_ERR_TIMEOUT, 0};
const int encodings[] = {0, 2, 8};
const int opcodes[] = {2, 9, 8};
for (size_t e = 0; e < 3; e++) {
int encoding = encodings[e];
for (size_t o = 0; o < 3; o++) {
for (int stage = 0; stage < 4; stage++) {
if (stage == 2 && encoding == 0) continue;
size_t extent = stage < 2 ? 1 : stage == 2 ? (size_t)encoding : 4;
for (size_t part = 0; part < extent; part++)
for (size_t err = 0; err < 3; err++)
for (int pending = 0; pending < 2; pending++)
failure(encoding, stage, part, errors[err], pending, opcodes[o]);
}
for (size_t split = 1; split <= 16; split *= 2)
for (size_t pending = 0; pending <= (size_t)(6 + encoding); pending++)
valid(encoding, split, pending, opcodes[o], false);
valid(encoding, 1, 0, opcodes[o], true);
}
}
extended_length(2, 126);
extended_length(2, 65535);
extended_length(8, 65536);
extended_length(8, UINT64_C(0x0102030405060708));
printf("WS actual full receive/blocking/pending functions: %u guard-page/canary cases PASS\n", checks);
return 0;
}