# ESP32 Serial Swiss Army Knife ![ESP32 Serial Swiss Army Knife logo](/images/logo.png "ESP32 Serial Swiss Army Knife") Universal wireless serial adaptor firmware for the ESP32-S3. ## Initial hardware target - ESP32-S3-DevKitC-1-compatible development board - ESP32-S3-WROOM-1-N16R8 module - 16 MB flash - 8 MB octal PSRAM - Adafruit MAX3243 full-pinout RS-232 breakout, product 5988 The firmware has completed **Phase 0 hardware characterization**, the **Phase 1 serial-core foundation**, the **Phase 2 transport-neutral session broker**, native USB CDC-ACM, and the **Phase 4 Wi-Fi foundation**. The current Phase 5A foundation adds authenticated HTTPS with a persistent device-specific identity and physical-console recovery. WebSocket/xterm.js serial transport is deliberately deferred to the next web phase. The MAX3243 diagnostics and recovery consoles remain available. No electrical test starts automatically; UART1 starts when requested explicitly or when a host opens native USB CDC. ## Hardware wiring See [`wiring.md`](wiring.md) for the hardware profile, GPIO assignments, loopback diagrams, safety notes, and the recommended test sequence. The initial profile covers the ESP32-S3-DevKitC-1 N16R8 and the Adafruit MAX3243 full-pinout RS-232 breakout. ## Flash partition layout The N16R8 target has 16 MiB flash and 8 MiB octal PSRAM. PlatformIO uses the custom [`partitions.csv`](partitions.csv) layout: | Partition | Offset | Size | Purpose | |---|---:|---:|---| | `nvs` | `0x009000` | 512 KiB | Serial, Wi-Fi, HTTPS credential, and certificate/key data | | `otadata` | `0x089000` | 8 KiB | Active OTA-slot selection metadata | | `phy_init` | `0x08B000` | 4 KiB | Optional PHY initialization data | | `nvs_key` | `0x08C000` | 4 KiB | Reserved for future encrypted-NVS keys | | `coredump` | `0x08D000` | 128 KiB | Reserved for flash core dumps | | `ota_0` | `0x0B0000` | 4 MiB | Primary application/OTA slot | | `ota_1` | `0x4B0000` | 4 MiB | Alternate application/OTA slot | | `storage` | `0x8B0000` | 7488 KiB | Future LittleFS web assets, logs, and files | Application offsets are aligned to the ESP32-S3's required 64 KiB boundary. The final storage partition ends at `0x1000000`, exactly the end of the 16 MiB flash chip. The partition table reserves OTA and LittleFS space but does not by itself implement OTA downloads, rollback confirmation, core-dump handling, NVS encryption, or filesystem mounting. Those features will be enabled deliberately in later phases. ### One-time migration from the default partition table The previous 1 MiB factory application began at `0x10000`, which is now inside the enlarged NVS address range. A normal upload does not erase all stale bytes there. Perform a full flash erase once when first switching to this layout: ```sh pio run --target erase pio run --target upload pio device monitor -b 115200 ``` This erases the currently saved serial configuration and all other flash contents. The firmware will boot with safe serial defaults and recreate NVS. Subsequent ordinary uploads do not require another full erase. PlatformIO's application-size report should now use the 4 MiB `ota_0` slot instead of the previous 1 MiB factory partition. ## Build ```sh pio run ``` ## Upload and monitor Connect the board's **USB-to-UART** port for firmware upload and the UART0 development console, then run: ```sh pio run --target upload pio device monitor -b 115200 ``` The firmware starts an interactive console on UART0 with the prompt `serial-tool>`. Type `help` to display concise root-command descriptions. This USB-to-UART device normally appears as `/dev/ttyUSB*`; it is separate from the native USB CDC serial transport described below. The console provides line editing, history for the current session, terminal-aware cursor movement, and Tab completion. ESP-IDF supplies completion for root commands; the project extends it to nested `debug`, `serial`, `broker`, `usb`, `wifi`, and `web` subcommands plus safe fixed values such as AP policy and serial framing. Password values are intentionally never completion candidates. ESP-IDF normally probes terminal cursor support once while constructing the UART REPL. If the board boots without a terminal attached, that probe times out and selects dumb mode. The DevKit's USB-to-UART bridge exposes no host-open signal that firmware can monitor, and entering enhanced mode before a terminal exists would block linenoise while it waits for a cursor-position response. The project therefore preserves safe dumb mode until the first real UART byte arrives. Attach picoterm, picocom, minicom, PuTTY, or another ANSI-capable terminal and press Enter once. That empty line is consumed by the basic reader and promotes the next prompt to enhanced mode, enabling history, Delete, cursor keys, and completion without rebooting. This detects terminal activity rather than electrical USB attachment; a genuinely non-ANSI terminal is not automatically distinguishable on this hardware profile. The root-level lifecycle command is: ```text reboot ``` It acknowledges the request, waits briefly for UART output to drain, and calls the ESP-IDF software restart. RAM-only serial or Wi-Fi changes are lost unless they were persisted first with `serial save` or `wifi save`. ### Phase 1 serial service The `serial` command manages the working configuration and UART1 service: ```text serial status serial start serial stop serial set serial save serial load serial defaults serial reset serial counters serial clear-counters ``` Safe defaults are 115200 baud, 8 data bits, no parity, one stop bit, no flow control, and inactive DTR. Supported configuration values are: | Parameter | Values | |---|---| | `baud` | 110–1000000 | | `data-bits` | `7`, `8` | | `parity` | `none`, `even`, `odd` | | `stop-bits` | `1`, `2` | | `flow` | `none`, `rts-cts` | | `dtr` | `inactive`, `active`, `on-connect` | | `rts-threshold` | 1–127 bytes | `serial set` changes the working configuration and safely restarts UART1 if the service is running. It does not write flash; use `serial save` to commit the current configuration to NVS. `serial defaults` changes RAM only, while `serial reset` applies and persists defaults. The firmware never erases the shared NVS partition automatically when storage is incompatible or unavailable. The service uses independent software RX and TX streams. Calls into those streams are nonblocking, and a deasserted CTS cannot block service shutdown. UART data access is intentionally reserved for the session broker; the `serial` command controls configuration and lifecycle only. UART1 has exclusive ownership while the service runs. Phase 0 commands will refuse to touch the port until `serial stop` releases it. ### Phase 2 session broker The broker is initialized at boot and continuously drains the serial service whenever UART1 is running. It is transport-neutral: console test clients and native USB CDC use the same API that WebSocket and SSH transports will use later. ```text broker status broker clients broker counters broker clear-counters broker connect broker disconnect broker request-writer broker release-writer broker force-writer broker send-hex broker read [maximum-bytes] broker events ``` Each connection receives a generation-safe numeric ID. Stale IDs from disconnected clients cannot address a newly reused slot. Up to eight clients may connect, each with a bounded 4096-byte output queue and a 16-entry event queue. UART RX is copied to every connected client. A full observer queue drops bytes only for that observer and records the loss; it never blocks UART reception or another client. With no clients, the broker still drains UART data and records it as unobserved. Exactly one client may hold the writer lease. Competing requests are denied and generate events. Administrative forced reassignment atomically revokes the old writer and grants the new one. Bytes already accepted before revocation remain queued for transmission; revocation prevents future admission rather than purging the UART TX stream. Connect, disconnect, writer grant, release, revoke, and denial events have a broker-global sequence number. Event queues are intentionally bounded, so future transports should reconcile sequence gaps against broker snapshots. The first and last broker connection also drive the Phase 1 `DTR=on-connect` policy. ### Native USB CDC-ACM transport The ESP32-S3's native USB OTG peripheral presents one CDC-ACM serial interface through the development board's connector labelled **USB**. It uses GPIO19 (`USB D-`) and GPIO20 (`USB D+`) and normally appears on Linux as `/dev/ttyACM*`. It is not the USB-to-UART bridge used for upload and logs. The UART0 development console provides these diagnostics and controls: ```text usb usb help usb status usb counters usb clear-counters usb request-writer usb release-writer ``` Both `usb` and `usb help` print the same multi-line command summary and return successfully; runtime state is shown explicitly with `usb status`. Opening the CDC port with DTR asserted automatically starts UART1, connects a broker client named `usb-cdc`, and requests the writer lease. If another client already owns the lease, USB remains connected as a read-only observer; `usb status` reports its current role. Closing the port or unplugging native USB disconnects that broker client and discards transport-local pending data. The serial service itself remains running until it is stopped explicitly with `serial stop`. The data path is binary-transparent. UTF-8 bytes, NUL bytes, terminal escape sequences, and color sequences are passed unchanged; interpretation remains the terminal application's responsibility. USB output is bounded and nonblocking, so a host that stops reading can lose only its own observer data rather than stall UART1 or another client. Host line coding is accepted for baud rates 110–1000000 with 7 or 8 data bits, none/odd/even parity, and 1 or 2 stop bits. USB's 1.5 stop bits and mark/space parity are rejected. Supported settings are applied to the working UART configuration only when USB owns the writer lease and queued UART TX has drained. They are not saved to NVS automatically; use `serial save` deliberately if the setting should survive reboot. USB RTS is reported as host status only. It does not drive the physical RS-232 RTS line, which remains controlled by UART1's configured RTS/CTS flow control. The development VID/PID comes from Espressif's TinyUSB defaults. The USB serial-number string is derived from the ESP32-S3 station MAC so multiple adapters can be distinguished consistently. #### Linux loopback validation Keep the USB-to-UART cable connected for logs and commands, and connect a second data-capable cable to the native **USB** connector. On the host, identify the new CDC device: ```sh dmesg ls -l /dev/ttyACM* ``` With power removed and no external RS-232 peer attached, connect only DE-9 pin 3 (`TX`) to pin 2 (`RX`), then power the board. Open the native port with a serial terminal such as: ```sh picocom -b 115200 /dev/ttyACM0 ``` Use the actual device path assigned by the host. Typed data should return through USB → broker → UART1 → MAX3243 loopback → broker → USB. On the UART0 console, verify `usb status`, `usb counters`, `broker clients`, and `serial status`. The USB client should normally be the writer and counters should increase without drops. For a binary check, install PySerial on the host and send all byte values: ```python import serial payload = bytes(range(256)) with serial.Serial("/dev/ttyACM0", 115200, timeout=2) as port: port.reset_input_buffer() port.write(payload) echoed = port.read(len(payload)) assert echoed == payload, (len(echoed), echoed.hex()) print("256-byte binary USB/RS-232 loopback passed") ``` Close the terminal and check `usb status` and `broker clients`; DTR-aware applications should cause the USB broker client to disconnect. Physically unplugging the native USB cable is the definitive detach test. To test observer mode, assign a console test client as writer before opening `/dev/ttyACM0`; USB should connect as an observer, receive UART output, and discard host-originated input until ownership is granted. Power down and remove the DE-9 pin 3-to-2 jumper before connecting an external serial peer. ### Wi-Fi foundation Wi-Fi is managed independently of the serial-session broker. It provides network connectivity and recovery access-point policy for the HTTPS service. The firmware does not run plaintext HTTP, DNS interception, a captive portal, NAPT, or any TCP serial listener. Configuration uses four fixed station-profile slots. Lower numeric priority values are tried first, with slot number breaking ties. Profiles support WPA2/WPA3 mixed operation or require WPA3-SAE. ESP-IDF's station threshold can express “WPA2 or stronger” but not a strict WPA2-only maximum, so the configuration does not pretend to offer a distinct WPA2-only mode. Each profile attempt has a 12-second association/DHCP deadline. After all enabled profiles fail, the manager uses exponential retry delays from 2 to 60 seconds. AP policy is independent of the station profiles: | Policy | Behavior | |---|---| | `off` | Station only; never start the fallback AP | | `fallback` | Start the AP immediately when no profiles exist, or after one failed profile cycle; disable it after station connectivity has remained stable for 30 seconds | | `always` | Keep AP and station active concurrently | Fresh defaults enable Wi-Fi with `fallback` policy, AP channel 6, a MAC-suffixed SSID such as `ESP32-SAK-A1B2C3`, and a randomly generated 16-character password. The initial random credential is saved to NVS automatically when possible so it remains stable across reboot. Retrieve it deliberately from the physical UART0 administration console with `wifi ap show-secret`. The `wifi` command provides: ```text wifi status wifi profiles wifi start|stop|reconnect wifi profile set wifi profile secret wifi profile enable|disable|delete wifi ap policy wifi ap ssid wifi ap channel <1..11> wifi ap secret|show-secret wifi save|load|defaults|reset wifi counters|clear-counters wifi ping [count] wifi nslookup wifi traceroute [max-hops] ``` The network diagnostics are also registered as root aliases, so `ping`, `nslookup`, and `traceroute` are equivalent to their `wifi`-prefixed forms. `ping` accepts 1–20 probes and supports IPv4 or IPv6. `nslookup` prints unique numeric IPv4/IPv6 results. `traceroute` is currently IPv4-only, sends one ICMP Echo probe per hop, accepts 1–30 hops, and uses a one-second timeout per hop; routers that suppress ICMP replies appear as `*`. Ordinary status and profile output never displays passwords. `wifi profile secret` and `wifi ap secret` read through a dedicated no-echo UART0 prompt, keeping credentials out of the command line and its history. SSIDs containing spaces can be quoted. Profile and AP edits apply to the working RAM configuration and restart Wi-Fi asynchronously if it is running; use `wifi save` explicitly to persist them. `wifi start` and `wifi stop` also change the working `enabled-at-boot` setting, which becomes persistent only after `wifi save`. A typical station setup is: ```text wifi profile set 0 10 mixed "your SSID" wifi profile secret 0 wifi profile enable 0 wifi save wifi reconnect wifi status ``` The fallback AP uses Espressif's default `192.168.4.1/24` network for now. AP clients receive addresses through its DHCP server but are not routed to the station network. ESP32-S3 has one 2.4 GHz radio, so in AP+STA mode the AP follows the connected station's channel. Station connection attempts and scans can temporarily increase AP latency, and clients can briefly reconnect when the channel moves. Wi-Fi credentials currently reside as plaintext in the application-owned `wifi_app/config` NVS blob. Selecting `WIFI_STORAGE_RAM` prevents the ESP-IDF driver from creating a second persistent credential copy, but it does not encrypt the application's blob. The reserved `nvs_key` partition alone does not enable encryption. NVS encryption, secure boot, flash encryption, and core-dump credential exposure require a deliberate later security phase. #### Wi-Fi validation 1. Boot with no station profiles. `wifi status` should report `ap-only`, and the generated SSID should be visible from another device. 2. Use `wifi ap show-secret`, join the AP, confirm a `192.168.4.x` lease, and run `wifi ping 192.168.4.1`. The authenticated HTTPS page should be reachable at `https://192.168.4.1/`. 3. Configure and enable a WPA2/WPA3 station profile using the example above. `wifi status` should progress through `connecting`, `waiting-ip`, and `online` and display the acquired address, channel, RSSI, and negotiated authentication. 4. Reboot and verify profile and AP credential persistence. 5. Configure two profiles with different priorities, make the first unavailable, and verify failover to the second after its timeout. 6. Make all profiles unavailable and verify fallback AP startup plus increasing retry delays in `wifi status`/`wifi counters`. 7. Test `wifi ap policy always` while online and confirm both interfaces remain available; expect the AP channel to follow the station. 8. Test `wifi stop`, `wifi start`, and `wifi reconnect` while confirming UART0 and native USB serial operation remain unaffected. 9. If available, test a WPA3-only profile and a wrong password, then inspect the disconnect reason and counters. ### Phase 5A authenticated HTTPS foundation One ESP-IDF HTTPS server listens on TCP port 443 across whichever AP and station interfaces are active. There is no plaintext port 80 listener. This slice intentionally exposes only: ```text GET / GET /api/status ``` Both endpoints require HTTP Basic authentication over TLS. `/` is a small self-contained status landing page; `/api/status` returns JSON containing uptime plus non-secret Wi-Fi, serial-service, broker, native-USB, and HTTPS state/counters. No WebSocket, xterm.js terminal, or web broker client exists in Phase 5A. On first boot, the device generates and persists: - username `admin` and a random 24-character Base64URL-safe password; - an ECDSA P-256 private key; - a device-specific self-signed SHA-256 certificate valid from 2025-01-01 through 2049-12-31; - certificate SANs for `192.168.4.1` and a MAC-suffixed name such as `esp32-sak-a1b2c3.local`. The certificate and credentials remain stable across ordinary reboot and OTA-slot changes until explicitly rotated. The `.local` name is included for future hostname discovery, but this phase does not yet advertise mDNS; use the AP address or the station address reported by `wifi status`. The physical UART0 administration console provides: ```text web web help web status web start|stop web counters|clear-counters web credentials show web credentials rotate --force web certificate info web certificate rotate --force web reset --force ``` `web` and `web help` print the same usage summary. `web credentials show` is the intended first-boot credential-retrieval path. Rotation and reset operations write NVS immediately rather than creating RAM-only secrets. Destructive operations require a literal `--force`; `web reset --force` is also the recovery path for an incompatible or damaged `web_sec/material` blob and starts HTTPS with the recovered material. Invalid stored material is never overwritten automatically, and HTTPS failure never disables UART0, native USB, the serial core, or Wi-Fi recovery. The self-signed certificate is not trusted by browsers or host tools by default. After retrieving the password, validate from a host connected to the fallback AP with: ```sh curl -k -u 'admin:YOUR_24_CHARACTER_PASSWORD' https://192.168.4.1/ curl -k -u 'admin:YOUR_24_CHARACTER_PASSWORD' https://192.168.4.1/api/status ``` A request without credentials should return `401 Unauthorized` and a `WWW-Authenticate` challenge: ```sh curl -k -i https://192.168.4.1/api/status ``` Inspect and compare the live certificate with `web certificate info`: ```sh openssl s_client -connect 192.168.4.1:443 -servername esp32-sak-device.local /dev/null \ | openssl x509 -noout -subject -issuer -dates -fingerprint -sha256 ``` Replace the example SNI name with the DNS SAN printed by `web certificate info`. SNI is not required for this single-certificate server, but supplying the device name makes the test representative of future hostname use. Repeat the fingerprint check after reboot to confirm persistence, then optionally test each explicit rotation command and verify that only the requested material changes. HTTPS is memory-bounded to two simultaneous client sockets; ESP-IDF documents approximately 40 KiB per TLS socket. Basic authentication is acceptable here only because plaintext HTTP is disabled. It is an initial administration mechanism, not the final authorization design. **Current security limitation:** the web password and ECDSA private key are stored as plaintext in the application-owned `web_sec/material` NVS blob, just as Wi-Fi credentials are currently plaintext in `wifi_app/config`. The reserved `nvs_key` partition does not activate NVS encryption. ESP-IDF 5.5 also keeps an internal heap copy of the active TLS private key and does not guarantee zeroization when that allocation is freed. Do not treat the current firmware as resistant to physical flash or RAM extraction; NVS encryption, flash encryption, secure boot, protected OTA, secret-aware core-dump handling, and framework-level key zeroization belong to the later hardening phase. ### Phase 0 diagnostics The top-level `status` command retains quick MAX3243 signal-state inspection. Potentially disruptive hardware-characterization operations are grouped below `debug` so the primary help page stays concise: ```text status debug transceiver debug drivers debug loopback-a debug loopback-b debug valid-test debug uart-loopback [8N1|8E1|8O1|8N2|7E1|7O1] [bytes] debug uart-suite debug cts-flow-test debug rts-flow-test ``` Run `debug` without a subcommand for its usage summary. `debug uart-loopback` defaults to `8N1` and 256 bytes. Its accepted payload range is 1–512 bytes. `debug uart-suite` covers 300 through 250000 baud and all supported frame formats. `debug cts-flow-test` verifies transmit gating and exact resumption, while `debug rts-flow-test` uses UART2 as an internal traffic generator to verify automatic receive backpressure. Follow the command-specific loopback wiring in [`wiring.md`](wiring.md) before invoking any test. A mutex-protected port lease prevents diagnostics, UART1 service startup, and future clients from reconfiguring the same GPIOs concurrently. If a UART driver cannot be removed during cleanup, the firmware keeps the MAX3243 shut down and marks the port faulted until reboot rather than exposing an ambiguous hardware state. The onboard RGB LED reports the most recent test-harness state: | Color | Meaning | |---|---| | Blue | Idle; waiting for a command | | Yellow/orange | Test running | | Green | Last test passed | | Red | Last test failed | This hardware profile uses the onboard RGB LED on GPIO48. Official ESP32-S3-DevKitC-1 v1.1 boards commonly use GPIO38 instead, and compatible boards or clones may vary. A different board revision requires an adjusted board pin profile before running this firmware. ## License This project is licensed under the [GNU General Public License version 3 only](LICENSE) (`GPL-3.0-only`). This is compatible with using the GPLv3 releases of wolfSSL and wolfSSH later. Third-party components remain subject to their respective licenses.