ESP32_Serial_Swiss_Army_Knife/docs/idf_553_rebase_review.md

IDF 5.5.3 uncommitted rebase review

Review date: 2026-09-18. Baseline Git HEAD: cdc4d4a8df02c2c44180b9b6d17bf6104c9fbcd2.

Current disposition: existing protected corrections are semantically retained; the initial stale compiled WS input and missing receive-error regression blockers are resolved by the supplied final fresh-build/24-suite evidence below. No evidence found that a protected correction should be removed as superseded. This is a bounded rebase review, not advisory recertification, device acceptance or release clearance.

Initial reviewer-owned change: this document only. Existing working changes were preserved. No SDK installation, firmware build, upload, erase, asset regeneration or production edit was performed. A concurrent change to third_party/release-notices/inputs.json appeared during review; it was left untouched and was not reviewed in that initial pass. The later supplied notice review semantically rebased nine entries, retaining the other 66 unchanged; this does not turn the initial review into a catalog audit.

Final follow-up — 2026-09-18

Supplied final parent command, run from the repository root:

CCACHE_DISABLE=1 python3 -B tools/validate_phase9.py --build-dir .pio/idf-candidate-5.5.3/app-validated/.pio/build/esp32-s3-devkitc-1-n16r8 --idf-path .pio/idf-candidate-5.5.3/core/packages/framework-espidf --platformio-core-dir .pio/idf-candidate-5.5.3/core --interop --web-performance

PASS 24/24. This selects the actual fresh isolated build, SDK and toolchain core; it is not a default-root-build execution. The separate fresh .pio/idf-candidate-5.5.3/app-validated build PASS reports 95,552 B linked RAM / 1,749,493 B flash, versus historical 94,340 / 1,768,901 B (+1,212 B RAM / −19,408 B flash). The default parent pio run timed out after 200 seconds during installation, before compilation; no normal root build PASS is claimed.

Pre/post source equality: 3,237 files, SHA-256 3a1af78c15cfdd02da1055a8957b4086f9018862b7aa1c7c52fd2401a1a0a031. Actual generated-input registration covers nine C sources plus one forced header. WS receive tests exercise generated code: 982 cases / 10 mutation checks, including the five signed sizeof corrections. The stale web-cookie fixture asserting IDF 5.5.0 was corrected, not bypassed. Historical stale-WS compilation/coverage failures in the rebase review are resolved by this final snapshot, not hidden or retroactively called passes.

The fix-bearing Wi-Fi bundle is integrated in the candidate, with unchanged PMF/WPA3; radio-hardware vulnerability closure and full target/resource/recovery gates remain pending. Nine notice catalog entries were semantically rebased, the other 66 unchanged (75 total); supplied notice evidence is 36 fixtures PASS, two actual deterministic bundles each 77 files / 4,433,930 bytes. Archive pins and source equality do not prove complete immutable root/ancillary/Python dependency closure, legal clearance or Phase 9 acceptance. This documentation update records supplied parent evidence; it did not rerun these builds, suites, bundles or hardware tests.

1. Historical review inputs, not assumed SDK paths

Paths below are relative to the repository except the explicitly absolute shared SDK:

Role	Observed path / identity
Old SDK	/home/mscholz/.platformio/packages/framework-espidf, package 3.50500.0 / IDF 5.5.0
Candidate archive	.pio/idf-candidate-5.5.3/archives/framework-espidf-3.50503.0.tar.gz
Candidate installed SDK	.pio/idf-candidate-5.5.3/core/packages/framework-espidf, package 3.50503.0 / IDF 5.5.3
Isolated application	.pio/idf-candidate-5.5.3/app
Existing candidate build	.pio/idf-candidate-5.5.3/app/.pio/build/esp32-s3-devkitc-1-n16r8
Candidate compiler in compile database	.pio/idf-candidate-5.5.3/core/packages/toolchain-xtensa-esp-elf/bin/xtensa-esp32s3-elf-gcc

There is no literal .pio/idf-candidate directory. The shared SDK is still old, not an installed production 5.5.3 SDK. There was no root .pio/build/esp32-s3-devkitc-1-n16r8 build to validate. Tests defaulting to the shared SDK must explicitly select the candidate.

Recomputed size and SHA-256 of all four complete archives against tools/idf_candidate/artifacts.json: all matched (platform 6.13.0, framework 3.50503.0, Xtensa and RISC-V 14.2.0+20251107). Framework archive SHA-256: 8353f6fd5030dd7e662500891428fad15d46efd7e4b718cab2fe6bfb9e7f13fc. Current platformio.ini selects that version pairing. Registry version pins in the root configuration are not themselves enforcement of these archive hashes; the isolated application uses local archive URLs.

Loaded the protected-entry registry from Git HEAD, verified every old original against its HEAD hash, then compared it with the working registry and actual candidate sources. Every working original hash and exact edit application passed. Also compared candidate protected IDF originals and esp_httpd_priv.h directly with members of the hash-verified framework archive: byte-identical. This separates old baseline, downloaded package, installed candidate and generated compilation inputs.

2. Historical blockers — resolved in final follow-up

B1 — Historical candidate firmware does not contain the new WS correction

The staged app/tools/security_overrides.py lacks Entry("httpd_ws", ...); the working generator includes it. The staged test runner also differs from the working web-performance runner. The candidate compile database compiles:

.../core/packages/framework-espidf/components/esp_http_server/src/httpd_ws.c

not security_overrides/httpd_ws/httpd_ws.c. The map includes libesp_http_server.a(httpd_ws.c.o). The working SDK override test with this actual build fails:

AssertionError: ('httpd_ws', [])

All eight previously protected generated C files and the wolfSSH forced header independently match the current generator's bytes. Their compile paths are present; representative wolfSSH and application commands retain the joined forced-header and crypto-policy includes. This is a specific stale ninth-C-source build, not evidence that all replacements are missing.

An ELF/bin/map exists, but this reviewer did not build it. Snapshot identities:

• firmware.elf: 15,558,488 bytes; SHA-256 39d12988d80c44787f1d9587d7ba685c990bfac940ba10180b86e60313b60fad.
• firmware.bin: 1,749,888 bytes; SHA-256 0477958591e0e21269b21fdd62ba855d507a6dd460107647f4f38739f9264e96.
• esp32_serial_swiss_army_knife.map: SHA-256 346abf8ce619742edd38274eef6267d6d8e0750fea6b1e2c8ffdcc34e5bc0085.

Required: stage the final reviewed inputs in an ownership-safe isolated workspace, rebuild, and verify all nine C overrides plus one forced header, exact owners, original-source absence and generated bytes against that build. Do not attribute the current WS correction to the existing binary.

B2 — Historical vendor-code tests and missing signed-error behavioral matrix

tests/web_serial_performance/run.py reads installed httpd_ws.c, not generated httpd_ws.c. Its enum-aware receive double is a necessary adaptation, but the suite fails compiling httpd_ws_get_frame_type against candidate vendor source:

error: comparison of integer expressions of different signedness
[-Werror=sign-compare]

The new five-edit override is included in generic generator/CMake fixtures. However, tests/sdk_security_overrides/run.py has no dedicated WS receive behavior or mutation suite. The web-performance receive double always supplies a successful first byte and the control-frame receiver is a double; it cannot establish the newly changed real frame-receive error paths.

Required: consume verified generated WS code without suppressing the warning or editing vendor files. Exercise actual httpd_ws_get_frame_type, httpd_ws_recv_frame and, where needed, the real httpd_recv_with_opt against bounded IO doubles: negative fail/timeout, EOF, short lengths and success at all five fixed-header reads; split/pending bytes; two- and eight-byte lengths; four-byte mask; automatic control-frame failure/close and no subsequent payload/send on failed framing. Require each removed signed cast to be detected by a negative regression/mutation. Passing generic generation alone is insufficient.

B3 — Historical documentation gap: final snapshot required

At the initial review, docs/idf_candidate_integration.md described preparation only, production untouched and smoke not run, while the worktree already had production configuration/guard changes and an isolated application ELF. The generated modification notice pointed to that preparation document as rebase evidence without per-entry dispositions. The integration record now links the semantic dispositions and supplied final build/test evidence. This review's initial artifact inspection is not retroactively a fresh build pass.

Broader wolf ordering/parser/crypto/authentication, browser admission and full phase validation were not rerun by this reviewer. Existing historical 5.5.0 results and a stale candidate build cannot stand in for final-snapshot integration validation. Hardware/radio/resource/recovery and distribution/source obligations remain separate established gates, not reasons to repeat completed broad advisory research.

3. Per-entry semantic disposition

All old entries preserve exactly their HEAD edit sequences; changed IDF file hashes reflect the vendor differences below. Exact successful replacement is necessary but was not the only check: old/new source differences and relevant ownership/error paths were inspected, and existing behavioral fixtures were executed against candidate-generated sources.

Protected entry	Disposition and reason
dhcpserver	Retain unchanged original and six edits. Vendor source is byte-identical across these SDKs; bounded option header/body, PAD/END handling and minimum lengths are not supplied by this upgrade. Extracted parser passes 196,623 guard-page/equivalence cases.
mbedtls_ssl_tls	Rebase original hash; retain EMS error return. Vendor changes include handshake-state helpers, dynamic-buffer reset handling, PRF label-length plumbing and exporter support. The calc_verify error branch still logs without returning; the local return remains necessary before PRF use. SHA256/SHA384 failures, success, PRF failure, non-EMS and resumption fixtures pass. Dynamic TLS buffers remain rejected. This is not a review of every new exporter/TLS1.3 path.
mbedtls_x509_create	Rebase original hash; retain OID allocation check. Vendor now rejects a prepopulated output list instead of clearing it; it still dereferences an unchecked newly allocated oid.p. Local OOM return remains needed. Subject/issuer OOM, cleanup and retry fixtures pass. Project certificate generation sets subject and issuer separately once, consistent with the changed empty-list precondition; do not generalize this to repeated setter calls elsewhere.
https_server	Rebase original hash; retain all four edits. Vendor delta adds hardware ECDSA curve/block configuration, not post-handshake allocation-failure cleanup, copied-key wiping or failed-start secure-context destruction. Local TLS delete, wipe-before-free and restored/null-cleared ownership still apply. Existing allocation/handshake/start/stop matrix passes. Failed stop continues to retain live ownership.
httpd_parse	Rebase original hash; retain all six edits. Vendor delta fixes cookie length reporting on truncation, not scratch ownership. Null initial pointer arithmetic, failed realloc ownership and unwiped retired scratch still require local fixes. Allocate/copy/wipe/free, preserve-old-on-failure, nullable parser pointer and final wipe remain; null/grow/shrink/failure/pending-unread fixtures pass.
esp_tls_mbedtls	Rebase original hash; retain guards and server-only policy. Vendor adds dynamic-buffer strategy, hardware ECDSA configuration, RSA-ALT cleanup and error/certificate reporting changes. None supplies the project server TLS1.2 ECDHE-ECDSA AES-GCM allowlist or no-renegotiation policy. Injection remains in server configuration; clients retain their defaults. Server/client isolation, feature and dynamic-buffer guard fixtures pass.
wolfssh_internal	Retain original hash and all 53 edits. Managed source matches HEAD. Ordering delta, restricted no-EXT_INFO profile, password bounds/wiping, parser/name/service/signature/window corrections are unchanged; this IDF migration does not supersede them. The initial semantic review did not rerun full wolf behavior; the final supplied 24/24 run includes the established wolf suites.
wolfssh_ssh	Retain original hash and ordering edits. Generated candidate bytes match working output; exact generated compilation path present.
wolfssh_internal_header	Retain original hash and ABI/order overlay. Generated header bytes match; candidate library and representative application commands retain the forced include and overlay include path. PUBLIC direct/transitive consumer fixture passes.
httpd_ws (new)	New candidate-specific correction, not a repin. IDF 5.5.3 changes fixed header reads from nonblocking <= 0 checks to blocking < sizeof(...). int negative receive results convert to unsigned size_t, making the checks false. The five (int)sizeof(...) changes correctly restore signed rejection for sizes 1, 2, 8, 4 and 1; they retain vendor blocking reads and do not alter payload framing policy. Behavioral regression and current-build evidence were initially blocked by B1/B2; the final follow-up resolves both with 982 cases / 10 mutation checks and actual generated-source registration.

No protected correction was demonstrated to be superseded. Native pre-handshake support is not a reason to delete the application's admission adapter: it is disabled in the candidate configuration and the application deliberately registers the two upgrade routes as ordinary GET handlers.

4. Heap extent and HTTPD private boundary

Heap: guarded 5.5.3 extent contract is supported

components/heap/heap_caps.c, multi_heap.c, multi_heap_platform.h and tlsf/tlsf.c are byte-identical old-to-candidate. The audited chain remains:

1. heap_caps_get_allocated_size removes any owner offset, locates the containing heap and subtracts owner metadata from the reported size.
2. Unpoisoned multi_heap_get_allocated_size aliases the implementation returning tlsf_block_size.
3. TLSF obtains the allocated payload block's rounded size, not a containing-block extent for an arbitrary interior pointer.

Candidate resolved config enables CONFIG_HEAP_POISONING_DISABLED, disables task tracking and leaves dynamic mbedTLS buffers off. ssh_memory still rejects poisoning and all other IDF versions. Actual candidate-source contract checks and host allocation/free/realloc wipe tests pass, including rounded capacity, retained shrink, growth and allocation failure. No new allocator layout/header mechanism was introduced. Target wipe latency and old-plus-new allocation peaks remain unmeasured.

HTTPD: relevant layouts retained; receive API changed

The private-header diff adds shared control-message declarations, HTTPD_RECV_OPT_*, the enum receive parameter and a direct-LRU-close declaration. It does not change the adapter-used httpd_data, httpd_req_aux or sock_db field layouts. The application compiles against candidate headers rather than embedding an old SDK struct definition.

Relevant behavior review:

• Session completion still performs request creation/handling, request deletion/purge, then updates the LRU completion marker. New sessions initialize the counter from the server; the adapter already tracks session identity. All 18 idle lifecycle groups pass against candidate completion/purge sources.
• httpd_main still processes control work before connection data. New direct LRU close is conditional; application lru_purge_enable remains false.
• Header/scratch parser delta is the cookie reporting fix, not a layout/termination change. Right-aligned pending bytes and application wipe preservation remain applicable.
• Optional URI registration still publishes before strdup failure cleanup; the application's allocate-both-before-publication correction remains necessary.
• Vendor URI dispatch adds optional pre-handshake callback support. Candidate config disables it; application ordinary-GET upgrade handlers continue to authorize/admit before manually sending 101.
• SDK WS sending remains separate header/payload sends through session send_fn; the bounded combined binary adapter still uses that TLS override and replaces it with reject-only behavior after incomplete output. No raw socket bypass was introduced.
• httpd_recv_with_opt changes from Boolean to enum and loops for blocking fixed-header reads; httpd_recv uses NONE. Async completion now wakes the owner with a control message. Application owner/async exclusion rules remain relevant. The receive double was correctly changed to the enum, and the initial WS receive-error/test-input blockers B1/B2 are now resolved as recorded in the final follow-up.

This supports the narrow guard update; it does not certify arbitrary configurations, async use or every HTTPD API.

5. Whole radio bundle identity and actual link inputs

Compared every regular archive member under these component trees with the corresponding installed candidate file; all were byte-identical:

Component	Files compared
esp_wifi	163
esp_phy	114
esp_coex	41
wpa_supplicant	301

This includes packaged source, headers, CMake inputs and all target/optional archives, not only the seven currently discussed S3 Wi-Fi blobs. It establishes coherent delivery from the one locked framework archive, not independent upstream PHY/coexistence Git submodule provenance, vendor signature validation or opaque binary correctness.

S3 Git blob identities recomputed from installed bytes:

Archive	Git blob SHA-1
Wi-Fi libcore.a	9f7b14a8bf6eec64973da8adc65d35b5ba9bee49
Wi-Fi libespnow.a	132b4f67e339ca2081d2add91c14eefa39476ff9
Wi-Fi libmesh.a	2e9dc1c8c7afbf033337b4175032e9b1161e3262
Wi-Fi libnet80211.a	2800d447ec385d33869373696ba8191292647694
Wi-Fi libpp.a	8944bcad7371621045f376cf74c62fde6f368cbb
Wi-Fi libsmartconfig.a	4dc759e25617aa00b9e12887fd092a1d5780a170
Wi-Fi libwapi.a	65655b6feab0572a6e8a1200946d53a21f3f4722
PHY libphy.a	75d3c8756c8e0310066fa9a1e5c24df50e6c04e3
PHY libbtbb.a	e94e3c2b476d8abb2757bcb53519ebcfba92b213
PHY libbttestmode.a	06389c391ea55278df6205b6195b6d8182c7b8be
PHY librfate.a	05dc4b69e02dac8e6c75ab8f9dc49c1d33d66c32
PHY librftest.a	45c6dc240287332c8c8790e84a80475dbfe3a842
Coexistence libcoexist.a	21fb35634fbdd421519bc985072fdc0ab54dd1df

The seven Wi-Fi identities agree with the already recorded 5.5.3 release-point column in docs/wifi_security_update_plan.md; they are not a mixed fix-point/release-point bundle. Candidate esp_wifi_driver.h declares wpa_ap_get_wpa_ie(size_t *len) and esp_wpa_main.c defines wpa_ap_get_wpa_ie(size_t *ie_len) and assigns that callback. The compile database uses candidate supplicant sources.

Actual candidate map LOAD entries select all seven Wi-Fi blobs plus libphy.a and libbtbb.a from the candidate SDK path, and the candidate-built component wrappers/supplicant archive. No old shared-SDK radio path was observed in those inputs. libcoexist.a is delivered and verified but is not a LOAD input in this configuration: esp_coex/CMakeLists.txt adds it only for software/external coexistence, not merely CONFIG_ESP_COEX_ENABLED. Optional RF/test PHY archives likewise are packaged identities, not claimed linked inputs. A map LOAD line does not prove all members survive link garbage collection.

The whole-package approach avoids a 5.5.0 C/header/blob transplant. This provides stronger delivery/path evidence than the earlier seven-Wi-Fi-only check, but does not demonstrate runtime PMF/WPA3 behavior, radio stability, target reserves or deployment vulnerability closure.

6. Historical validation executed and handoff

Commands run from the repository root:

CCACHE_DISABLE=1 python3 -B tests/ssh_memory/run.py --idf-path .pio/idf-candidate-5.5.3/core/packages/framework-espidf
IDF_PATH=/home/mscholz/Repos/ESP32_serial_swiss_army_knife/.pio/idf-candidate-5.5.3/core/packages/framework-espidf CCACHE_DISABLE=1 python3 -B tests/web_httpd_idle/run.py
IDF_PATH=/home/mscholz/Repos/ESP32_serial_swiss_army_knife/.pio/idf-candidate-5.5.3/core/packages/framework-espidf CCACHE_DISABLE=1 python3 -B tests/web_serial_performance/run.py
CCACHE_DISABLE=1 python3 -B tests/sdk_security_overrides/run.py --idf-path .pio/idf-candidate-5.5.3/core/packages/framework-espidf --build-dir .pio/idf-candidate-5.5.3/app/.pio/build/esp32-s3-devkitc-1-n16r8

Results:

• PASS: SSH allocator behavior, six invalid poisoning profiles, explicit disabled profile, seven rejected IDF versions, real candidate extent-source checks and compile with candidate version header.
• PASS: all 18 idle lifecycle groups and candidate completion/owner-order guards.
• FAIL: web serial performance compile against original WS source, B2.
• PARTIAL PASS / overall FAIL: SDK generator, HTTPS/scratch/TLS/DHCP/EMS/X509 behavior and CMake replacement/forced-header/extension fixtures pass; actual build registration rejects missing generated WS input, B1. Do not report this command as passing.
• PASS: read-only HEAD/current original-hash and edit comparisons; all four archive identities; protected candidate/archive comparisons; all four radio component trees; candidate compile/map path inspection.

The initial next actions were generated-WS regression coverage, a fresh staged snapshot/build and final-input integration validation. Those actions are now complete in the supplied final follow-up; the historical failures above remain failures of the old snapshot, not current blockers or retroactive passes. No production defect was found in the retained old overrides or reviewed heap/private-layout assumptions. Normal root build completion, immutable full dependency closure, target/radio/resource/recovery and release gates remain open. Preserve PMF/WPA3, broker isolation, native USB independence and UART0 recovery.