Complete Phase 12 dual-stack networking

Add IPv6-aware Wi-Fi state, HTTPS/SSH listeners, mDNS service reconciliation, and browser Wi-Fi administration. Include a guarded build-local fix for mDNS 1.12.0 membership handling, focused regression suites, and Phase 12 acceptance documentation.
2026-09-20 22:35:34 +02:00
parent ece4ba77e3
commit 8902b25d78
52 changed files with 3042 additions and 163 deletions
@@ -0,0 +1,145 @@
+# mDNS 1.12.0 multicast membership regression
+
+This is a narrow dependency correctness fix for the lwIP backend, not a general
+patch framework or a revival of the abandoned Phase 9 patches. Dependency
+versions, the component manifest/lock, managed sources, socket backend, and
+application mDNS lifecycle policy are unchanged.
+
+## Defects and fix
+
+In the inspected `mdns_networking_lwip.c`, upstream `pcb_if_deinit()` (original
+lines 267–277) only leaves a group when the interface's **last** protocol bit is
+cleared. Removing one family while the other remains active therefore leaks a
+membership reference. Restoration joins again; repeated partial transitions
+accumulate references in lwIP's bounded group-use counter. Deinitializing an
+already-disabled family can also issue an unmatched leave.
+
+The generated copy checks the requested family's active bit, returns immediately
+if absent, and leaves that family's group before clearing its bit. It clears
+`ready` only after the last family on that interface is removed, and frees the
+shared PCB only if no other interface remains ready.
+
+In `pcb_if_init()` (original lines 282–301), a successful group join followed by
+`pcb_init()` failure was not unwound. The copy attempts a leave before returning
+the **original** PCB error. No readiness or protocol bit is published on failure.
+
+These operations still run through upstream's existing lwIP-thread wrappers.
+No new locks, allocations, task ownership, retry state, or public API are added.
+Upstream `join_group()` is unchanged, including its refusal to act when the netif
+is absent/down. Leaves remain best effort: the patch guarantees balanced leave
+**attempts**, not successful lwIP cleanup after an interface disappears. It does
+not retry a failed leave or retain a PCB solely because leave failed; physical
+netif teardown remains responsible for its own membership cleanup.
+
+## Build integration
+
+1. Root `CMakeLists.txt` includes `cmake/mdns_membership.cmake` **after** IDF's
+   `project()`, when component targets and resolved versions are available.
+2. The include obtains `espressif__mdns`'s `COMPONENT_DIR`, `COMPONENT_LIB`, and
+   `COMPONENT_VERSION` via IDF component properties. Socket-backend builds skip
+   the overlay entirely; the fix is only relevant to the lwIP source.
+3. The resolved component version and manifest version must both be exactly
+   `1.12.0`. Using IDF's configured Python interpreter, the helper verifies the
+   complete source SHA-256, then performs two exact, unique replacements.
+4. It writes only `${CMAKE_BINARY_DIR}/mdns_membership/mdns_networking_lwip.c`.
+   Unchanged output is not rewritten, avoiding gratuitous rebuilds. Everything
+   outside the two replacement regions, including provenance/license headers,
+   remains byte-for-byte intact. The actual upstream file identifies itself as
+   Apache-2.0, copyright 2022–2025 Espressif; it is not relabeled as GPL. Existing
+   project GPL/license material is untouched.
+5. The include replaces exactly one matching entry in the existing component
+   target's `SOURCES` property, accepting absolute or component-relative paths.
+   All other source entries and all component compile definitions, include paths,
+   dependencies, and target linkage are retained. The original file is not also
+   compiled. An unexpected source list fails configuration.
+6. The helper, original source, manifest, and generated copy are registered as
+   configure dependencies. Reconfiguration regenerates/verifies the copy; a
+   clean build simply recreates it. The CMake include itself is automatically a
+   CMake input. Version/hash/replacement errors fail configure rather than
+   silently compiling an unpatched dependency.
+
+The integration uses the component's existing target; no whole-component copy,
+managed in-place edits, dependency overrides, manifest changes, or extra source
+library are involved. The inspected upstream CMake file offers no per-source
+substitution option; replacing the target source after `project()` is the local
+integration point.
+
+### Exact reviewed baseline
+
+- Component: `espressif/mdns`, version `1.12.0`.
+- Manifest repository: `espressif/esp-protocols`, `components/mdns`.
+- Manifest commit: `db06b19b7be729c163d346f62ec0eba01047b7f1` (provenance;
+  guards are the resolved/manifest version and complete source hash).
+- Source: `managed_components/espressif__mdns/mdns_networking_lwip.c`.
+- SHA-256: `adc139fa504a925ab644f21f8dce3659927f534e390a176b72b0ae3206c6a3ea`.
+
+## Run the host tests
+
+From the repository root, on a POSIX host with Python 3, CMake >= 3.16, and a C11
+compiler:
+
+```sh
+./tests/mdns_membership/run.py
+# Optional compiler selection (CC is an executable, not a shell command):
+CC=clang ./tests/mdns_membership/run.py
+```
+
+The managed component must already be installed. The runner uses temporary
+directories, does not fetch dependencies, does not build firmware, and disables
+core dumps for deliberate negative tests. It generates the overlay with the
+**same helper used by configure**, extracts the actual patched state declarations
+and six functions (`pcb_init`, `pcb_deinit`, `mdns_priv_if_ready`,
+`is_any_pcb_in_use`, `pcb_if_init`, `pcb_if_deinit`), and compiles them with
+`-std=c11 -Wall -Wextra -Werror -pedantic`. Only group operations and low-level
+UDP APIs are mocked; the lifecycle logic under test is not reimplemented.
+
+Coverage:
+
+- 512 IPv4 loss/restoration cycles while IPv6 stays active, and 512 inverse
+  cycles; exact group references and readiness checked after each transition.
+- Repeated disabled-family deinit, final-family teardown, and duplicate init.
+- Join failure without an unmatched leave or PCB allocation.
+- 512 allocation failures and 512 bind failures per family, with successful
+  joins unwound, no state publication, original error preserved, and recovery.
+- Leave failure on teardown and unwind: one attempt, no repeated disabled leave,
+  original allocation error retained.
+- 512 cycles per family with a second interface holding the shared PCB; exactly
+  one allocation, no premature free, and one final removal.
+- Separate negative controls restoring each original buggy function must fail
+  the very same executable harness.
+- Mock-IDF CMake configure fixtures for absolute/relative source replacement,
+  unchanged unrelated source, repeat configure without rewriting, socket bypass,
+  missing/duplicate networking source rejection, resolved-version rejection,
+  and full-source hash rejection. Manifest-version rejection is tested through
+  the helper CLI. Rejected fresh helper runs must not produce output.
+- Original managed networking bytes are checked unchanged after the run.
+
+These are host unit/configure tests, **not** a real ESP-IDF integration build or
+hardware/network test. The real `join_group()`, lwIP IGMP/MLD counters, scheduling,
+and network teardown are not exercised by mocks. Run the normal `pio run`
+separately (never concurrently with another full build) to validate actual IDF
+integration. In its generated `compile_commands.json`, the mDNS networking entry
+must point to the build-local `mdns_membership/mdns_networking_lwip.c`, with no
+original managed networking entry; other mDNS sources must remain managed paths.
+On device, exercise repeated IPv4-only and IPv6-only loss/restoration while the
+other family remains active, final network teardown/recovery, and continued
+mDNS discovery. No hardware result is implied by the host PASS output.
+
+## Maintenance / removal
+
+Do **not** update the expected hash merely to make a new dependency build. On
+any mismatch, review the entire changed networking source and component CMake,
+especially protocol state, join/leave behavior, netif-down semantics, and shared
+PCB lifetime. Establish whether upstream has fixed both defects first. A
+version bump is a separate explicitly reviewed change; this overlay does not
+select or upgrade the dependency.
+
+If upstream fixes both paths, remove the root include and these narrowly scoped
+helper/test files (or replace the tests with suitable upstream coverage), then
+clean/reconfigure and check that only the upstream source compiles. If a local
+fix is still necessary, re-audit the exact new source/version, update the guards
+and exact replacements together, update this provenance record, rerun the host
+suite with its negative controls, and run a serial full firmware build. Never
+fall back to unguarded search/replace or silently skip a failed patch. A stale
+build-local copy is not authority: it must always be reproducible from the
+managed source plus the reviewed helper.
@@ -0,0 +1,157 @@
+#!/usr/bin/env python3
+"""Host-only regression and configure integration tests; no firmware build."""
+import hashlib
+import os
+import resource
+from pathlib import Path
+import shutil
+import subprocess
+import sys
+import tempfile
+
+ROOT = Path(__file__).resolve().parents[2]
+HERE = Path(__file__).resolve().parent
+COMPONENT = ROOT / "managed_components/espressif__mdns"
+HELPER = ROOT / "cmake/mdns_membership.py"
+
+
+def run(command, *, succeeds=True):
+    result = subprocess.run([str(item) for item in command], capture_output=True, text=True)
+    if (result.returncode == 0) != succeeds:
+        raise AssertionError(f"Unexpected result: {command}\n{result.stdout}\n{result.stderr}")
+    return result.stdout + result.stderr
+
+
+def function(source, signature):
+    start = source.index(signature)
+    brace = source.index("{", start)
+    depth = 1
+    end = brace + 1
+    while depth:
+        depth += (source[end] == "{") - (source[end] == "}")
+        end += 1
+    return source[start:end] + "\n"
+
+
+def extract(source):
+    # Keep the original provenance header and actual state declarations too.
+    header = source[:source.index("#include")]
+    state = source[source.index("enum interface_protocol"):source.index("static const char *TAG")]
+    signatures = ["static esp_err_t pcb_init(void)", "static void pcb_deinit(void)",
+                  "bool mdns_priv_if_ready(", "static bool is_any_pcb_in_use(void)",
+                  "static void pcb_if_deinit(", "static esp_err_t pcb_if_init("]
+    return header + state + "\n".join(function(source, name) for name in signatures)
+
+
+def cmake_fixture(work, component, mode, succeeds=True):
+    fixture = work / f"cmake-{mode}"
+    fixture.mkdir()
+    (fixture / "dummy.c").write_text("int dummy;\n")
+    source = component / "mdns_networking_lwip.c"
+    sources = f'"{source}"'
+    if mode == "missing":
+        sources = ""
+    if mode == "duplicate":
+        sources += " " + sources
+    # Test relative as well as absolute target source properties.
+    if mode == "relative":
+        shutil.copyfile(source, fixture / source.name)
+        shutil.copyfile(component / "idf_component.yml", fixture / "idf_component.yml")
+        component = fixture
+        sources = source.name
+    version = "1.13.0" if mode == "version" else "1.12.0"
+    (fixture / "CMakeLists.txt").write_text(f'''cmake_minimum_required(VERSION 3.16)
+project(mdns_overlay_fixture C)
+add_library(mdns STATIC dummy.c {sources})
+function(idf_component_get_property output component property)
+    if(property STREQUAL "COMPONENT_DIR")
+        set(value "{component}")
+    elseif(property STREQUAL "COMPONENT_LIB")
+        set(value mdns)
+    elseif(property STREQUAL "COMPONENT_VERSION")
+        set(value "{version}")
+    else()
+        message(FATAL_ERROR "Unexpected component property")
+    endif()
+    set(${{output}} "${{value}}" PARENT_SCOPE)
+endfunction()
+function(idf_build_get_property output property)
+    if(NOT property STREQUAL "PYTHON")
+        message(FATAL_ERROR "Unexpected build property")
+    endif()
+    set(${{output}} "{sys.executable}" PARENT_SCOPE)
+endfunction()
+set(CONFIG_MDNS_NETWORKING_SOCKET {"ON" if mode == "socket" else "OFF"})
+include("{ROOT / 'cmake/mdns_membership.cmake'}")
+get_target_property(sources mdns SOURCES)
+file(WRITE "${{CMAKE_BINARY_DIR}}/selected.txt" "${{sources}}")
+''')
+    build = fixture / "build"
+    output = run(["cmake", "-S", fixture, "-B", build], succeeds=succeeds)
+    if not succeeds:
+        assert "mDNS" in output, output
+        return
+    selected = (build / "selected.txt").read_text().split(";")
+    if mode == "socket":
+        assert str(source) in selected
+        assert not (build / "mdns_membership").exists()
+    else:
+        overlay = build / "mdns_membership/mdns_networking_lwip.c"
+        assert selected == ["dummy.c", str(overlay)], selected
+        before = overlay.stat().st_mtime_ns
+        run(["cmake", "-S", fixture, "-B", build])
+        assert overlay.stat().st_mtime_ns == before
+
+
+def main():
+    resource.setrlimit(resource.RLIMIT_CORE, (0, 0))
+    original = (COMPONENT / "mdns_networking_lwip.c").read_bytes()
+    with tempfile.TemporaryDirectory(prefix="mdns-membership-") as directory:
+        work = Path(directory)
+        overlay = work / "overlay/mdns_networking_lwip.c"
+        run([sys.executable, HELPER, COMPONENT, overlay])
+        patched = overlay.read_text()
+        assert patched[:patched.index("#include")] == original.decode()[:original.decode().index("#include")]
+        timestamp = overlay.stat().st_mtime_ns
+        run([sys.executable, HELPER, COMPONENT, overlay])
+        assert overlay.stat().st_mtime_ns == timestamp
+        (work / "actual_functions.inc").write_text(extract(patched))
+        executable = work / "test"
+        command = [os.environ.get("CC", "cc"), "-std=c11", "-Wall", "-Wextra", "-Werror",
+                   "-pedantic", "-I", work, HERE / "test.c", "-o", executable]
+        run(command)
+        print(run([executable]), end="")
+
+        # Prove the harness detects each original bug independently.
+        for name in ("pcb_if_deinit", "pcb_if_init"):
+            signature = ("static void " if name.endswith("deinit") else "static esp_err_t ") + name + "("
+            mutated = patched.replace(function(patched, signature), function(original.decode(), signature))
+            (work / "actual_functions.inc").write_text(extract(mutated))
+            run(command)
+            run([executable], succeeds=False)
+        print("PASS: both original defects independently fail the same harness")
+
+        copied = work / "component"
+        copied.mkdir()
+        shutil.copyfile(COMPONENT / "idf_component.yml", copied / "idf_component.yml")
+        (copied / "mdns_networking_lwip.c").write_bytes(original + b"\n")
+        output = run([sys.executable, HELPER, copied, work / "rejected.c"], succeeds=False)
+        assert "SHA-256 mismatch" in output and not (work / "rejected.c").exists()
+        (copied / "mdns_networking_lwip.c").write_bytes(original)
+        manifest = (copied / "idf_component.yml").read_text()
+        (copied / "idf_component.yml").write_text(manifest.replace("version: 1.12.0", "version: 1.13.0"))
+        output = run([sys.executable, HELPER, copied, work / "rejected.c"], succeeds=False)
+        assert "exactly version 1.12.0" in output and not (work / "rejected.c").exists()
+        (copied / "idf_component.yml").write_text(manifest)
+        for mode in ("absolute", "relative", "socket", "missing", "duplicate", "version"):
+            cmake_fixture(work, copied, mode, succeeds=mode not in ("missing", "duplicate", "version"))
+        # Also exercise hash failure through configure, not only the helper CLI.
+        (copied / "mdns_networking_lwip.c").write_bytes(original + b"\n")
+        cmake_fixture(work, copied, "hash", succeeds=False)
+        print("PASS: source/version guards, CMake replacement, relative paths, socket bypass, repeat configure")
+    assert hashlib.sha256((COMPONENT / "mdns_networking_lwip.c").read_bytes()).digest() == hashlib.sha256(original).digest()
+    print("PASS: managed networking source unchanged")
+
+
+if __name__ == "__main__":
+    main()
@@ -0,0 +1,174 @@
+#include <assert.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <string.h>
+
+typedef int esp_err_t;
+typedef int mdns_if_t;
+typedef int mdns_ip_protocol_t;
+enum { ESP_OK, ESP_ERR_NO_MEM, ESP_ERR_INVALID_STATE };
+enum { MDNS_IP_PROTOCOL_V4, MDNS_IP_PROTOCOL_V6, MDNS_IP_PROTOCOL_MAX };
+#define MDNS_MAX_INTERFACES 2
+#define MDNS_SERVICE_PORT 5353
+struct udp_pcb { int mcast_ttl, remote_port, remote_ip; };
+static int any_address;
+#define IP_ANY_TYPE (&any_address)
+#define ip_addr_copy(to, from) ((to) = (from))
+static void receive(void) {}
+static struct udp_pcb storage;
+static bool allocated, fail_alloc, fail_bind, fail_join, fail_leave;
+static int allocations, removals, joins[2][2], leaves[2][2], references[2][2];
+static struct udp_pcb *udp_new(void)
+{
+    if (fail_alloc) return NULL;
+    assert(!allocated);
+    allocated = true;
+    allocations++;
+    return &storage;
+}
+static int udp_bind(struct udp_pcb *pcb, const int *address, int port)
+{
+    assert(pcb == &storage && address == IP_ANY_TYPE && port == 5353);
+    return fail_bind;
+}
+static void udp_remove(struct udp_pcb *pcb)
+{
+    assert(pcb == &storage && allocated);
+    allocated = false;
+    removals++;
+}
+static void udp_recv(struct udp_pcb *pcb, void (*callback)(void), void *arg)
+{ assert(pcb == &storage); (void)callback; (void)arg; }
+static void udp_disconnect(struct udp_pcb *pcb) { assert(pcb == &storage); }
+static esp_err_t join_group(mdns_if_t interface, mdns_ip_protocol_t family, bool join)
+{
+    if (join) {
+        joins[interface][family]++;
+        if (fail_join) return ESP_ERR_INVALID_STATE;
+        references[interface][family]++;
+        /* A real lwIP group has a bounded use count: never accumulate it. */
+        assert(references[interface][family] == 1);
+    } else {
+        leaves[interface][family]++;
+        if (fail_leave) return ESP_ERR_INVALID_STATE;
+        assert(references[interface][family] == 1);
+        references[interface][family]--;
+    }
+    return ESP_OK;
+}
+
+#include "actual_functions.inc"
+
+static void reset(void)
+{
+    assert(!allocated && s_pcb_main == NULL);
+    memset(s_interfaces, 0, sizeof(s_interfaces));
+    memset(joins, 0, sizeof(joins));
+    memset(leaves, 0, sizeof(leaves));
+    memset(references, 0, sizeof(references));
+    allocations = removals = 0;
+    fail_alloc = fail_bind = fail_join = fail_leave = false;
+}
+static void check(int interface, int family, bool active)
+{
+    assert(!!mdns_priv_if_ready(interface, family) == active);
+    assert(references[interface][family] == (int)active);
+    assert(joins[interface][family] - leaves[interface][family] == (int)active);
+}
+static void transitions(int family)
+{
+    reset();
+    int other = 1 - family;
+    assert(pcb_if_init(0, family) == ESP_OK);
+    assert(pcb_if_init(0, other) == ESP_OK);
+    for (int i = 0; i < 512; i++) {
+        pcb_if_deinit(0, family);
+        check(0, family, false);
+        check(0, other, true);
+        assert(s_interfaces[0].ready && allocated && removals == 0);
+        pcb_if_deinit(0, family); /* Already-disabled family must not leave again. */
+        check(0, family, false);
+        assert(pcb_if_init(0, family) == ESP_OK);
+        check(0, family, true);
+        int before = joins[0][family];
+        assert(pcb_if_init(0, family) == ESP_ERR_INVALID_STATE);
+        assert(joins[0][family] == before);
+    }
+    pcb_if_deinit(0, other);
+    assert(s_interfaces[0].ready && allocated);
+    pcb_if_deinit(0, family);
+    assert(!s_interfaces[0].ready && !allocated);
+    assert(allocations == 1 && removals == 1);
+    pcb_if_deinit(0, family);
+    pcb_if_deinit(0, other);
+    check(0, family, false);
+    check(0, other, false);
+}
+static void failures(int family)
+{
+    reset();
+    pcb_if_deinit(0, family);
+    assert(leaves[0][family] == 0 && removals == 0);
+    fail_join = true;
+    assert(pcb_if_init(0, family) == ESP_ERR_INVALID_STATE);
+    assert(!s_interfaces[0].ready && s_interfaces[0].proto == 0);
+    assert(allocations == 0 && leaves[0][family] == 0);
+    fail_join = false;
+    for (int mode = 0; mode < 2; mode++) {
+        fail_alloc = mode == 0;
+        fail_bind = mode == 1;
+        for (int i = 0; i < 512; i++) {
+            int before = leaves[0][family];
+            assert(pcb_if_init(0, family) == (fail_alloc ? ESP_ERR_NO_MEM : ESP_ERR_INVALID_STATE));
+            assert(leaves[0][family] == before + 1);
+            assert(references[0][family] == 0 && !allocated && !s_pcb_main);
+            assert(!s_interfaces[0].ready && s_interfaces[0].proto == 0);
+            pcb_if_deinit(0, family);
+            assert(leaves[0][family] == before + 1);
+        }
+    }
+    fail_alloc = fail_bind = false;
+    assert(pcb_if_init(0, family) == ESP_OK);
+    pcb_if_deinit(0, family);
+    assert(!allocated && references[0][family] == 0);
+
+    /* Preserve upstream best-effort leave semantics when netif has gone down. */
+    reset();
+    assert(pcb_if_init(0, family) == ESP_OK);
+    fail_leave = true;
+    pcb_if_deinit(0, family);
+    pcb_if_deinit(0, family);
+    assert(leaves[0][family] == 1 && !s_interfaces[0].ready && !allocated);
+    references[0][family] = 0; /* Model netif teardown clearing its memberships. */
+    reset();
+    fail_alloc = fail_leave = true;
+    assert(pcb_if_init(0, family) == ESP_ERR_NO_MEM);
+    assert(leaves[0][family] == 1 && !s_interfaces[0].ready && !allocated);
+    references[0][family] = 0;
+}
+static void shared_pcb(int family)
+{
+    reset();
+    assert(pcb_if_init(1, 1 - family) == ESP_OK);
+    for (int i = 0; i < 512; i++) {
+        assert(pcb_if_init(0, family) == ESP_OK);
+        pcb_if_deinit(0, family);
+        pcb_if_deinit(0, family);
+        check(0, family, false);
+        check(1, 1 - family, true);
+        assert(!s_interfaces[0].ready && allocated && removals == 0);
+    }
+    assert(allocations == 1);
+    pcb_if_deinit(1, 1 - family);
+    assert(!allocated && removals == 1);
+}
+int main(void)
+{
+    for (int family = 0; family < 2; family++) {
+        transitions(family);
+        failures(family);
+        shared_pcb(family);
+    }
+    puts("PASS: actual mDNS functions, 512 cycles per family/scenario, failure unwind and shared PCB");
+    return 0;
+}