ESP32_Serial_Swiss_Army_Knife/README.md

ESP32 Serial Swiss Army Knife

Universal wireless serial adaptor firmware for the ESP32-S3.

Initial hardware target

• ESP32-S3-DevKitC-1-compatible development board
• ESP32-S3-WROOM-1-N16R8 module
• 16 MB flash
• 8 MB octal PSRAM
• Adafruit MAX3243 full-pinout RS-232 breakout, product 5988

The firmware has completed Phase 0 hardware characterization, the Phase 1 serial-core foundation, the Phase 2 transport-neutral session broker, native USB CDC-ACM, the Phase 4 Wi-Fi foundation, and the Phase 5 authenticated HTTPS web terminal. Phase 5B adds an offline xterm.js interface and bounded WebSocket transport to the persistent HTTPS identity and recovery foundation from Phase 5A. The MAX3243 diagnostics and recovery consoles remain available. No electrical test starts automatically; UART1 starts when requested explicitly or when a native USB or authenticated web-terminal session opens.

Hardware wiring

See wiring.md for the hardware profile, GPIO assignments, loopback diagrams, safety notes, and the recommended test sequence. The initial profile covers the ESP32-S3-DevKitC-1 N16R8 and the Adafruit MAX3243 full-pinout RS-232 breakout.

Flash partition layout

The N16R8 target has 16 MiB flash and 8 MiB octal PSRAM. PlatformIO uses the custom partitions.csv layout:

Partition	Offset	Size	Purpose
nvs	0x009000	512 KiB	Serial, Wi-Fi, HTTPS credential, and certificate/key data
otadata	0x089000	8 KiB	Active OTA-slot selection metadata
phy_init	0x08B000	4 KiB	Optional PHY initialization data
nvs_key	0x08C000	4 KiB	Reserved for future encrypted-NVS keys
coredump	0x08D000	128 KiB	Reserved for flash core dumps
ota_0	0x0B0000	4 MiB	Primary application/OTA slot
ota_1	0x4B0000	4 MiB	Alternate application/OTA slot
storage	0x8B0000	7488 KiB	Future LittleFS web assets, logs, and files

Application offsets are aligned to the ESP32-S3's required 64 KiB boundary. The final storage partition ends at 0x1000000, exactly the end of the 16 MiB flash chip.

The partition table reserves OTA and LittleFS space but does not by itself implement OTA downloads, rollback confirmation, core-dump handling, NVS encryption, or filesystem mounting. Those features will be enabled deliberately in later phases.

One-time migration from the default partition table

The previous 1 MiB factory application began at 0x10000, which is now inside the enlarged NVS address range. A normal upload does not erase all stale bytes there. Perform a full flash erase once when first switching to this layout:

pio run --target erase
pio run --target upload
pio device monitor -b 115200

This erases the currently saved serial configuration and all other flash contents. The firmware will boot with safe serial defaults and recreate NVS. Subsequent ordinary uploads do not require another full erase.

PlatformIO's application-size report should now use the 4 MiB ota_0 slot instead of the previous 1 MiB factory partition.

Build

pio run

Upload and monitor

Connect the board's USB-to-UART port for firmware upload and the UART0 development console, then run:

pio run --target upload
pio device monitor -b 115200

The firmware starts an interactive console on UART0 with the prompt serial-tool>. Type help to display concise root-command descriptions. This USB-to-UART device normally appears as /dev/ttyUSB*; it is separate from the native USB CDC serial transport described below.

The console provides line editing, history for the current session, terminal-aware cursor movement, and Tab completion. ESP-IDF supplies completion for root commands; the project extends it to nested debug, serial, broker, usb, wifi, and web subcommands plus safe fixed values such as AP policy and serial framing. Password values are intentionally never completion candidates.

ESP-IDF normally probes terminal cursor support once while constructing the UART REPL. If the board boots without a terminal attached, that probe times out and selects dumb mode. The DevKit's USB-to-UART bridge exposes no host-open signal that firmware can monitor, and entering enhanced mode before a terminal exists would block linenoise while it waits for a cursor-position response.

The project therefore preserves safe dumb mode until the first real UART byte arrives. Attach picoterm, picocom, minicom, PuTTY, or another ANSI-capable terminal and press Enter once. That empty line is consumed by the basic reader and promotes the next prompt to enhanced mode, enabling history, Delete, cursor keys, and completion without rebooting. This detects terminal activity rather than electrical USB attachment; a genuinely non-ANSI terminal is not automatically distinguishable on this hardware profile.

The root-level lifecycle command is:

reboot

It acknowledges the request, waits briefly for UART output to drain, and calls the ESP-IDF software restart. RAM-only serial or Wi-Fi changes are lost unless they were persisted first with serial save or wifi save.

Phase 1 serial service

The serial command manages the working configuration and UART1 service:

serial status
serial start
serial stop
serial set <baud|data-bits|parity|stop-bits|flow|dtr|rts-threshold> <value>
serial save
serial load
serial defaults
serial reset
serial counters
serial clear-counters

Safe defaults are 115200 baud, 8 data bits, no parity, one stop bit, no flow control, and inactive DTR. Supported configuration values are:

Parameter	Values
baud	110–1000000
data-bits	7, 8
parity	none, even, odd
stop-bits	1, 2
flow	none, rts-cts
dtr	inactive, active, on-connect
rts-threshold	1–127 bytes

serial set changes the working configuration and safely restarts UART1 if the service is running. It does not write flash; use serial save to commit the current configuration to NVS. serial defaults changes RAM only, while serial reset applies and persists defaults. The firmware never erases the shared NVS partition automatically when storage is incompatible or unavailable.

The service uses independent software RX and TX streams. Calls into those streams are nonblocking, and a deasserted CTS cannot block service shutdown. UART data access is intentionally reserved for the session broker; the serial command controls configuration and lifecycle only.

UART1 has exclusive ownership while the service runs. Phase 0 commands will refuse to touch the port until serial stop releases it.

Phase 2 session broker

The broker is initialized at boot and continuously drains the serial service whenever UART1 is running. It is transport-neutral: console test clients and native USB CDC use the same API that WebSocket and SSH transports will use later.

broker status
broker clients
broker counters
broker clear-counters
broker connect <name>
broker disconnect <client-id>
broker request-writer <client-id>
broker release-writer <client-id>
broker force-writer <client-id|none>
broker send-hex <client-id> <hex-bytes>
broker read <client-id> [maximum-bytes]
broker events <client-id>

Each connection receives a generation-safe numeric ID. Stale IDs from disconnected clients cannot address a newly reused slot. Up to eight clients may connect, each with a bounded 4096-byte output queue and a 16-entry event queue.

UART RX is copied to every connected client. A full observer queue drops bytes only for that observer and records the loss; it never blocks UART reception or another client. With no clients, the broker still drains UART data and records it as unobserved.

Exactly one client may hold the writer lease. Competing requests are denied and generate events. Administrative forced reassignment atomically revokes the old writer and grants the new one. Bytes already accepted before revocation remain queued for transmission; revocation prevents future admission rather than purging the UART TX stream.

Connect, disconnect, writer grant, release, revoke, and denial events have a broker-global sequence number. Event queues are intentionally bounded, so future transports should reconcile sequence gaps against broker snapshots. The first and last broker connection also drive the Phase 1 DTR=on-connect policy.

Native USB CDC-ACM transport

The ESP32-S3's native USB OTG peripheral presents one CDC-ACM serial interface through the development board's connector labelled USB. It uses GPIO19 (USB D-) and GPIO20 (USB D+) and normally appears on Linux as /dev/ttyACM*. It is not the USB-to-UART bridge used for upload and logs.

The UART0 development console provides these diagnostics and controls:

usb
usb help
usb status
usb counters
usb clear-counters
usb request-writer
usb release-writer

Both usb and usb help print the same multi-line command summary and return successfully; runtime state is shown explicitly with usb status.

Opening the CDC port with DTR asserted automatically starts UART1, connects a broker client named usb-cdc, and requests the writer lease. If another client already owns the lease, USB remains connected as a read-only observer; usb status reports its current role. Closing the port or unplugging native USB disconnects that broker client and discards transport-local pending data. The serial service itself remains running until it is stopped explicitly with serial stop.

The data path is binary-transparent. UTF-8 bytes, NUL bytes, terminal escape sequences, and color sequences are passed unchanged; interpretation remains the terminal application's responsibility. USB output is bounded and nonblocking, so a host that stops reading can lose only its own observer data rather than stall UART1 or another client.

Host line coding is accepted for baud rates 110–1000000 with 7 or 8 data bits, none/odd/even parity, and 1 or 2 stop bits. USB's 1.5 stop bits and mark/space parity are rejected. Supported settings are applied to the working UART configuration only when USB owns the writer lease and queued UART TX has drained. They are not saved to NVS automatically; use serial save deliberately if the setting should survive reboot. USB RTS is reported as host status only. It does not drive the physical RS-232 RTS line, which remains controlled by UART1's configured RTS/CTS flow control.

The development VID/PID comes from Espressif's TinyUSB defaults. The USB serial-number string is derived from the ESP32-S3 station MAC so multiple adapters can be distinguished consistently.

Linux loopback validation

Keep the USB-to-UART cable connected for logs and commands, and connect a second data-capable cable to the native USB connector. On the host, identify the new CDC device:

dmesg
ls -l /dev/ttyACM*

With power removed and no external RS-232 peer attached, connect only DE-9 pin 3 (TX) to pin 2 (RX), then power the board. Open the native port with a serial terminal such as:

picocom -b 115200 /dev/ttyACM0

Use the actual device path assigned by the host. Typed data should return through USB → broker → UART1 → MAX3243 loopback → broker → USB. On the UART0 console, verify usb status, usb counters, broker clients, and serial status. The USB client should normally be the writer and counters should increase without drops.

For a binary check, install PySerial on the host and send all byte values:

import serial

payload = bytes(range(256))
with serial.Serial("/dev/ttyACM0", 115200, timeout=2) as port:
    port.reset_input_buffer()
    port.write(payload)
    echoed = port.read(len(payload))

assert echoed == payload, (len(echoed), echoed.hex())
print("256-byte binary USB/RS-232 loopback passed")

Close the terminal and check usb status and broker clients; DTR-aware applications should cause the USB broker client to disconnect. Physically unplugging the native USB cable is the definitive detach test. To test observer mode, assign a console test client as writer before opening /dev/ttyACM0; USB should connect as an observer, receive UART output, and discard host-originated input until ownership is granted.

Power down and remove the DE-9 pin 3-to-2 jumper before connecting an external serial peer.

Wi-Fi foundation

Wi-Fi is managed independently of the serial-session broker. It provides network connectivity and recovery access-point policy for the HTTPS service. The firmware does not run plaintext HTTP, DNS interception, a captive portal, NAPT, or any TCP serial listener.

Configuration uses four fixed station-profile slots. Lower numeric priority values are tried first, with slot number breaking ties. Profiles support WPA2/WPA3 mixed operation or require WPA3-SAE. ESP-IDF's station threshold can express “WPA2 or stronger” but not a strict WPA2-only maximum, so the configuration does not pretend to offer a distinct WPA2-only mode. Each profile attempt has a 12-second association/DHCP deadline. After all enabled profiles fail, the manager uses exponential retry delays from 2 to 60 seconds.

AP policy is independent of the station profiles:

Policy	Behavior
off	Station only; never start the fallback AP
fallback	Start the AP immediately when no profiles exist, or after one failed profile cycle; disable it after station connectivity has remained stable for 30 seconds
always	Keep AP and station active concurrently

Fresh defaults enable Wi-Fi with fallback policy, AP channel 6, a MAC-suffixed SSID such as ESP32-SAK-A1B2C3, and a randomly generated 16-character password. The initial random credential is saved to NVS automatically when possible so it remains stable across reboot. Retrieve it deliberately from the physical UART0 administration console with wifi ap show-secret.

The wifi command provides:

wifi status
wifi profiles
wifi start|stop|reconnect
wifi profile set <slot> <priority> <mixed|wpa3> <ssid>
wifi profile secret <slot>
wifi profile enable|disable|delete <slot>
wifi ap policy <off|fallback|always>
wifi ap ssid <ssid>
wifi ap channel <1..11>
wifi ap secret|show-secret
wifi save|load|defaults|reset
wifi counters|clear-counters
wifi ping <host> [count]
wifi nslookup <host>
wifi traceroute <host> [max-hops]

The network diagnostics are also registered as root aliases, so ping, nslookup, and traceroute are equivalent to their wifi-prefixed forms. ping accepts 1–20 probes and supports IPv4 or IPv6. nslookup prints unique numeric IPv4/IPv6 results. traceroute is currently IPv4-only, sends one ICMP Echo probe per hop, accepts 1–30 hops, and uses a one-second timeout per hop; routers that suppress ICMP replies appear as *.

Ordinary status and profile output never displays passwords. wifi profile secret and wifi ap secret read through a dedicated no-echo UART0 prompt, keeping credentials out of the command line and its history. SSIDs containing spaces can be quoted. Profile and AP edits apply to the working RAM configuration and restart Wi-Fi asynchronously if it is running; use wifi save explicitly to persist them. wifi start and wifi stop also change the working enabled-at-boot setting, which becomes persistent only after wifi save.

A typical station setup is:

wifi profile set 0 10 mixed "your SSID"
wifi profile secret 0
wifi profile enable 0
wifi save
wifi reconnect
wifi status

The fallback AP uses Espressif's default 192.168.4.1/24 network for now. AP clients receive addresses through its DHCP server but are not routed to the station network. ESP32-S3 has one 2.4 GHz radio, so in AP+STA mode the AP follows the connected station's channel. Station connection attempts and scans can temporarily increase AP latency, and clients can briefly reconnect when the channel moves.

Wi-Fi credentials currently reside as plaintext in the application-owned wifi_app/config NVS blob. Selecting WIFI_STORAGE_RAM prevents the ESP-IDF driver from creating a second persistent credential copy, but it does not encrypt the application's blob. The reserved nvs_key partition alone does not enable encryption. NVS encryption, secure boot, flash encryption, and core-dump credential exposure require a deliberate later security phase.

Wi-Fi validation

1. Boot with no station profiles. wifi status should report ap-only, and the generated SSID should be visible from another device.
2. Use wifi ap show-secret, join the AP, confirm a 192.168.4.x lease, and run wifi ping 192.168.4.1. The authenticated HTTPS page should be reachable at https://192.168.4.1/.
3. Configure and enable a WPA2/WPA3 station profile using the example above. wifi status should progress through connecting, waiting-ip, and online and display the acquired address, channel, RSSI, and negotiated authentication.
4. Reboot and verify profile and AP credential persistence.
5. Configure two profiles with different priorities, make the first unavailable, and verify failover to the second after its timeout.
6. Make all profiles unavailable and verify fallback AP startup plus increasing retry delays in wifi status/wifi counters.
7. Test wifi ap policy always while online and confirm both interfaces remain available; expect the AP channel to follow the station.
8. Test wifi stop, wifi start, and wifi reconnect while confirming UART0 and native USB serial operation remain unaffected.
9. If available, test a WPA3-only profile and a wrong password, then inspect the disconnect reason and counters.

Phase 5 authenticated HTTPS web terminal

One ESP-IDF HTTPS server listens on TCP port 443 across whichever AP and station interfaces are active. There is no plaintext port 80 listener. Phase 5A established persistent authentication, certificate management, and recovery; Phase 5B adds these local-only browser resources and transport endpoints:

GET  /
GET  /api/status
POST /api/ws-ticket
WSS  /ws/serial?ticket=<one-time-ticket>
GET  /assets/xterm.css
GET  /assets/xterm.js
GET  /assets/addon-fit.js
GET  /assets/app.js

The page, status API, assets, and ticket endpoint require HTTP Basic authentication over TLS. / is now a responsive xterm.js serial workspace; /api/status returns JSON containing uptime plus non-secret Wi-Fi, serial-service, broker, native-USB, HTTPS, and WebSocket state/counters. xterm.js and FitAddon are pinned, vendored, compressed, and served by the ESP32 itself, so the terminal works while connected only to the fallback AP and never depends on a CDN. The terminal is constrained to the browser viewport, and fitting only resizes xterm when the host dimensions and resulting row/column count actually change; terminal rendering therefore cannot create a self-amplifying page-height loop.

The document CSP permits scripts only from the device itself. xterm.js requires inline styles for its DOM renderer, but inline scripts remain prohibited. A data-URL empty favicon avoids an unnecessary authenticated /favicon.ico request. A CSP warning naming a UUID or browser extension despite no inline script in the served document is browser-injected content and is intentionally not hash-whitelisted by the firmware.

On first boot, the device generates and persists:

• username admin and a random 24-character Base64URL-safe password;
• an ECDSA P-256 private key;
• a device-specific self-signed SHA-256 certificate valid from 2025-01-01 through 2049-12-31;
• certificate SANs for 192.168.4.1 and a MAC-suffixed name such as esp32-sak-a1b2c3.local.

The certificate and credentials remain stable across ordinary reboot and OTA-slot changes until explicitly rotated. The .local name is included for future hostname discovery, but this phase does not yet advertise mDNS; use the AP address or the station address reported by wifi status.

The physical UART0 administration console provides:

web
web help
web status
web start|stop
web counters|clear-counters
web credentials show
web credentials rotate --force
web certificate info
web certificate rotate --force
web reset --force

web and web help print the same usage summary. web credentials show is the intended first-boot credential-retrieval path. Rotation and reset operations write NVS immediately rather than creating RAM-only secrets. Destructive operations require a literal --force; web reset --force is also the recovery path for an incompatible or damaged web_sec/material blob and starts HTTPS with the recovered material. Invalid stored material is never overwritten automatically, and HTTPS failure never disables UART0, native USB, the serial core, or Wi-Fi recovery.

The self-signed certificate is not trusted by browsers or host tools by default. After retrieving the password, validate from a host connected to the fallback AP with:

curl -k -u 'admin:YOUR_24_CHARACTER_PASSWORD' https://192.168.4.1/
curl -k -u 'admin:YOUR_24_CHARACTER_PASSWORD' https://192.168.4.1/api/status

A request without credentials should return 401 Unauthorized and a WWW-Authenticate challenge:

curl -k -i https://192.168.4.1/api/status

Inspect and compare the live certificate with web certificate info:

openssl s_client -connect 192.168.4.1:443 -servername esp32-sak-device.local </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates -fingerprint -sha256

Replace the example SNI name with the DNS SAN printed by web certificate info. SNI is not required for this single-certificate server, but supplying the device name makes the test representative of future hostname use. Repeat the fingerprint check after reboot to confirm persistence, then optionally test each explicit rotation command and verify that only the requested material changes.

WebSocket authentication and broker behavior

Browser JavaScript cannot reliably attach a Basic Authorization header to a WebSocket constructor. The authenticated page therefore obtains a 192-bit random, RAM-only ticket with POST /api/ws-ticket, then presents that ticket once in the WSS URL. The server stores only its SHA-256 digest, accepts it once within 30 seconds, binds it to the current credential generation, and creates no broker client until validation succeeds. Credential rotation invalidates outstanding tickets and closes active web-terminal sessions.

ESP-IDF 5.5 sends the RFC 6455 101 Switching Protocols response before invoking the application WebSocket handler. Consequently, an invalid ticket receives the protocol upgrade and is then closed immediately rather than receiving an HTTP 401; it never gains a broker session, serial output, or writer access. Strict rejection before 101 would require a framework-level pre-handshake authorization hook that ESP-IDF 5.5 does not provide.

Each accepted browser becomes a normal SESSION_BROKER_CLIENT_WEB. Opening the first terminal starts UART1 if necessary and automatically requests the writer lease. A competing web or USB client remains a read-only observer when another client owns the lease. The page clearly reports its role and provides Request control, Release control, and Reconnect actions. Broker ownership remains authoritative even if a browser is stale or malicious.

Serial traffic uses binary WebSocket frames. Browser input is UTF-8 encoded and split into at most 1024-byte frames; output is drained in at most 512-byte frames. Each web session permits only one queued/in-flight TLS frame. If a browser stops reading, its own 4096-byte broker observer queue eventually drops data without blocking UART reception, USB, or another broker observer. ESP-IDF performs TLS sends on one shared HTTP task, so a slow TLS peer can delay other HTTPS work for at most the configured one-second socket timeout; this is bounded rather than absolute per-socket isolation.

A direct ticket diagnostic is available without exposing the ticket in firmware logs:

curl -k -u 'admin:YOUR_24_CHARACTER_PASSWORD' -X POST https://192.168.4.1/api/ws-ticket

For end-to-end validation:

1. Open https://192.168.4.1/, accept the device certificate warning, and authenticate as admin.
2. Confirm xterm.js loads without Internet access and the page reaches Connected / Writer when no other writer exists.
3. Send text, terminal escape sequences, UTF-8, and pasted input through an RS-232 loopback or peer; verify exact traffic through web counters, broker clients, and serial counters.
4. Open a second browser/private session. It should connect as an observer, receive the same UART output, and keep terminal input disabled.
5. Release control in the first browser, request it in the second, and verify the role badges, broker writer ID, and actual serial input ownership change together.
6. Open native USB while a web writer exists, then repeat with USB owning the lease. Confirm each losing transport remains an observer and cannot inject bytes.
7. Close/reload a browser and verify its broker client disappears, the writer lease is released when applicable, and reconnect uses a fresh ticket.
8. Run web credentials rotate --force; existing browsers should disconnect and old credentials must no longer mint tickets.
9. Exercise web stop, web start, and reboot while confirming UART0/native USB recovery remains available and no stale web broker clients survive.

Vendored browser sources, versions, hashes/provenance, deterministic gzip artifacts, and MIT license notices are recorded under web_assets/. Third-party code remains under its upstream license; project firmware code remains GPL-3.0-only.

HTTPS permits up to six simultaneous client sockets: two bounded persistent WebSocket terminals plus parallel browser asset, ticket, and status requests. ESP-IDF documents approximately 40 KiB per active TLS socket, so this is a concurrency ceiling rather than preallocated per-socket memory. WebSocket serial sessions themselves remain fixed at two. To preserve internal DRAM during concurrent handshakes, the firmware configures ESP-IDF's mbedTLS allocator for the board's external PSRAM. Basic authentication is acceptable here only because plaintext HTTP is disabled. It is an initial administration mechanism, not the final authorization design.

Current security limitation: the web password and ECDSA private key are stored as plaintext in the application-owned web_sec/material NVS blob, just as Wi-Fi credentials are currently plaintext in wifi_app/config. The reserved nvs_key partition does not activate NVS encryption. ESP-IDF 5.5 also keeps active mbedTLS allocations, including TLS key/session material, in external PSRAM under this memory configuration and does not guarantee zeroization when those allocations are freed. PSRAM encryption is not enabled in the current firmware. Do not treat the current firmware as resistant to physical flash or RAM extraction; NVS encryption, flash/PSRAM encryption, secure boot, protected OTA, secret-aware core-dump handling, and framework-level key zeroization belong to the later hardening phase.

Phase 0 diagnostics

The top-level status command retains quick MAX3243 signal-state inspection. Potentially disruptive hardware-characterization operations are grouped below debug so the primary help page stays concise:

status
debug transceiver <enable|disable>
debug drivers <tx 0|1> <dtr 0|1> <rts 0|1>
debug loopback-a
debug loopback-b
debug valid-test
debug uart-loopback <baud> [8N1|8E1|8O1|8N2|7E1|7O1] [bytes]
debug uart-suite
debug cts-flow-test
debug rts-flow-test

Run debug without a subcommand for its usage summary. debug uart-loopback defaults to 8N1 and 256 bytes. Its accepted payload range is 1–512 bytes. debug uart-suite covers 300 through 250000 baud and all supported frame formats. debug cts-flow-test verifies transmit gating and exact resumption, while debug rts-flow-test uses UART2 as an internal traffic generator to verify automatic receive backpressure. Follow the command-specific loopback wiring in wiring.md before invoking any test.

A mutex-protected port lease prevents diagnostics, UART1 service startup, and future clients from reconfiguring the same GPIOs concurrently. If a UART driver cannot be removed during cleanup, the firmware keeps the MAX3243 shut down and marks the port faulted until reboot rather than exposing an ambiguous hardware state.

The onboard RGB LED reports the most recent test-harness state:

Color	Meaning
Blue	Idle; waiting for a command
Yellow/orange	Test running
Green	Last test passed
Red	Last test failed

This hardware profile uses the onboard RGB LED on GPIO48. Official ESP32-S3-DevKitC-1 v1.1 boards commonly use GPIO38 instead, and compatible boards or clones may vary. A different board revision requires an adjusted board pin profile before running this firmware.

License

This project is licensed under the GNU General Public License version 3 only (GPL-3.0-only). This is compatible with using the GPLv3 releases of wolfSSL and wolfSSH later. Third-party components remain subject to their respective licenses.