ESP32 Serial Swiss Army Knife

Universal wireless serial adaptor firmware for the ESP32-S3.

Initial hardware target

• ESP32-S3-DevKitC-1-compatible development board
• ESP32-S3-WROOM-1-N16R8 module
• 16 MB flash
• 8 MB octal PSRAM
• Adafruit MAX3243 full-pinout RS-232 breakout, product 5988

The firmware has completed Phase 0 hardware characterization, the Phase 1 serial-core foundation, the Phase 2 transport-neutral session broker, native USB CDC-ACM, and the Phase 4 Wi-Fi foundation. The current Phase 5A foundation adds authenticated HTTPS with a persistent device-specific identity and physical-console recovery. WebSocket/xterm.js serial transport is deliberately deferred to the next web phase. The MAX3243 diagnostics and recovery consoles remain available. No electrical test starts automatically; UART1 starts when requested explicitly or when a host opens native USB CDC.

Hardware wiring

See wiring.md for the hardware profile, GPIO assignments, loopback diagrams, safety notes, and the recommended test sequence. The initial profile covers the ESP32-S3-DevKitC-1 N16R8 and the Adafruit MAX3243 full-pinout RS-232 breakout.

Flash partition layout

The N16R8 target has 16 MiB flash and 8 MiB octal PSRAM. PlatformIO uses the custom partitions.csv layout:

Partition	Offset	Size	Purpose
nvs	0x009000	512 KiB	Serial, Wi-Fi, HTTPS credential, and certificate/key data
otadata	0x089000	8 KiB	Active OTA-slot selection metadata
phy_init	0x08B000	4 KiB	Optional PHY initialization data
nvs_key	0x08C000	4 KiB	Reserved for future encrypted-NVS keys
coredump	0x08D000	128 KiB	Reserved for flash core dumps
ota_0	0x0B0000	4 MiB	Primary application/OTA slot
ota_1	0x4B0000	4 MiB	Alternate application/OTA slot
storage	0x8B0000	7488 KiB	Future LittleFS web assets, logs, and files

Application offsets are aligned to the ESP32-S3's required 64 KiB boundary. The final storage partition ends at 0x1000000, exactly the end of the 16 MiB flash chip.

The partition table reserves OTA and LittleFS space but does not by itself implement OTA downloads, rollback confirmation, core-dump handling, NVS encryption, or filesystem mounting. Those features will be enabled deliberately in later phases.

One-time migration from the default partition table

The previous 1 MiB factory application began at 0x10000, which is now inside the enlarged NVS address range. A normal upload does not erase all stale bytes there. Perform a full flash erase once when first switching to this layout:

pio run --target erase
pio run --target upload
pio device monitor -b 115200

This erases the currently saved serial configuration and all other flash contents. The firmware will boot with safe serial defaults and recreate NVS. Subsequent ordinary uploads do not require another full erase.

PlatformIO's application-size report should now use the 4 MiB ota_0 slot instead of the previous 1 MiB factory partition.

Build

pio run

Upload and monitor

Connect the board's USB-to-UART port for firmware upload and the UART0 development console, then run:

pio run --target upload
pio device monitor -b 115200

The firmware starts an interactive console on UART0 with the prompt serial-tool>. Type help to display concise root-command descriptions. This USB-to-UART device normally appears as /dev/ttyUSB*; it is separate from the native USB CDC serial transport described below.

The console provides line editing, history for the current session, terminal-aware cursor movement, and Tab completion. ESP-IDF supplies completion for root commands; the project extends it to nested debug, serial, broker, usb, wifi, and web subcommands plus safe fixed values such as AP policy and serial framing. Password values are intentionally never completion candidates.

ESP-IDF normally probes terminal cursor support once while constructing the UART REPL. If the board boots without a terminal attached, that probe times out and selects dumb mode. The DevKit's USB-to-UART bridge exposes no host-open signal that firmware can monitor, and entering enhanced mode before a terminal exists would block linenoise while it waits for a cursor-position response.

The project therefore preserves safe dumb mode until the first real UART byte arrives. Attach picoterm, picocom, minicom, PuTTY, or another ANSI-capable terminal and press Enter once. That empty line is consumed by the basic reader and promotes the next prompt to enhanced mode, enabling history, Delete, cursor keys, and completion without rebooting. This detects terminal activity rather than electrical USB attachment; a genuinely non-ANSI terminal is not automatically distinguishable on this hardware profile.

The root-level lifecycle command is:

reboot

It acknowledges the request, waits briefly for UART output to drain, and calls the ESP-IDF software restart. RAM-only serial or Wi-Fi changes are lost unless they were persisted first with serial save or wifi save.

Phase 1 serial service

The serial command manages the working configuration and UART1 service:

serial status
serial start
serial stop
serial set <baud|data-bits|parity|stop-bits|flow|dtr|rts-threshold> <value>
serial save
serial load
serial defaults
serial reset
serial counters
serial clear-counters

Safe defaults are 115200 baud, 8 data bits, no parity, one stop bit, no flow control, and inactive DTR. Supported configuration values are:

Parameter	Values
baud	110–1000000
data-bits	7, 8
parity	none, even, odd
stop-bits	1, 2
flow	none, rts-cts
dtr	inactive, active, on-connect
rts-threshold	1–127 bytes

serial set changes the working configuration and safely restarts UART1 if the service is running. It does not write flash; use serial save to commit the current configuration to NVS. serial defaults changes RAM only, while serial reset applies and persists defaults. The firmware never erases the shared NVS partition automatically when storage is incompatible or unavailable.

The service uses independent software RX and TX streams. Calls into those streams are nonblocking, and a deasserted CTS cannot block service shutdown. UART data access is intentionally reserved for the session broker; the serial command controls configuration and lifecycle only.

UART1 has exclusive ownership while the service runs. Phase 0 commands will refuse to touch the port until serial stop releases it.

Phase 2 session broker

The broker is initialized at boot and continuously drains the serial service whenever UART1 is running. It is transport-neutral: console test clients and native USB CDC use the same API that WebSocket and SSH transports will use later.

broker status
broker clients
broker counters
broker clear-counters
broker connect <name>
broker disconnect <client-id>
broker request-writer <client-id>
broker release-writer <client-id>
broker force-writer <client-id|none>
broker send-hex <client-id> <hex-bytes>
broker read <client-id> [maximum-bytes]
broker events <client-id>

Each connection receives a generation-safe numeric ID. Stale IDs from disconnected clients cannot address a newly reused slot. Up to eight clients may connect, each with a bounded 4096-byte output queue and a 16-entry event queue.

UART RX is copied to every connected client. A full observer queue drops bytes only for that observer and records the loss; it never blocks UART reception or another client. With no clients, the broker still drains UART data and records it as unobserved.

Exactly one client may hold the writer lease. Competing requests are denied and generate events. Administrative forced reassignment atomically revokes the old writer and grants the new one. Bytes already accepted before revocation remain queued for transmission; revocation prevents future admission rather than purging the UART TX stream.

Connect, disconnect, writer grant, release, revoke, and denial events have a broker-global sequence number. Event queues are intentionally bounded, so future transports should reconcile sequence gaps against broker snapshots. The first and last broker connection also drive the Phase 1 DTR=on-connect policy.

Native USB CDC-ACM transport

The ESP32-S3's native USB OTG peripheral presents one CDC-ACM serial interface through the development board's connector labelled USB. It uses GPIO19 (USB D-) and GPIO20 (USB D+) and normally appears on Linux as /dev/ttyACM*. It is not the USB-to-UART bridge used for upload and logs.

The UART0 development console provides these diagnostics and controls:

usb
usb help
usb status
usb counters
usb clear-counters
usb request-writer
usb release-writer

Both usb and usb help print the same multi-line command summary and return successfully; runtime state is shown explicitly with usb status.

Opening the CDC port with DTR asserted automatically starts UART1, connects a broker client named usb-cdc, and requests the writer lease. If another client already owns the lease, USB remains connected as a read-only observer; usb status reports its current role. Closing the port or unplugging native USB disconnects that broker client and discards transport-local pending data. The serial service itself remains running until it is stopped explicitly with serial stop.

The data path is binary-transparent. UTF-8 bytes, NUL bytes, terminal escape sequences, and color sequences are passed unchanged; interpretation remains the terminal application's responsibility. USB output is bounded and nonblocking, so a host that stops reading can lose only its own observer data rather than stall UART1 or another client.

Host line coding is accepted for baud rates 110–1000000 with 7 or 8 data bits, none/odd/even parity, and 1 or 2 stop bits. USB's 1.5 stop bits and mark/space parity are rejected. Supported settings are applied to the working UART configuration only when USB owns the writer lease and queued UART TX has drained. They are not saved to NVS automatically; use serial save deliberately if the setting should survive reboot. USB RTS is reported as host status only. It does not drive the physical RS-232 RTS line, which remains controlled by UART1's configured RTS/CTS flow control.

The development VID/PID comes from Espressif's TinyUSB defaults. The USB serial-number string is derived from the ESP32-S3 station MAC so multiple adapters can be distinguished consistently.

Linux loopback validation

Keep the USB-to-UART cable connected for logs and commands, and connect a second data-capable cable to the native USB connector. On the host, identify the new CDC device:

dmesg
ls -l /dev/ttyACM*

With power removed and no external RS-232 peer attached, connect only DE-9 pin 3 (TX) to pin 2 (RX), then power the board. Open the native port with a serial terminal such as:

picocom -b 115200 /dev/ttyACM0

Use the actual device path assigned by the host. Typed data should return through USB → broker → UART1 → MAX3243 loopback → broker → USB. On the UART0 console, verify usb status, usb counters, broker clients, and serial status. The USB client should normally be the writer and counters should increase without drops.

For a binary check, install PySerial on the host and send all byte values:

import serial

payload = bytes(range(256))
with serial.Serial("/dev/ttyACM0", 115200, timeout=2) as port:
    port.reset_input_buffer()
    port.write(payload)
    echoed = port.read(len(payload))

assert echoed == payload, (len(echoed), echoed.hex())
print("256-byte binary USB/RS-232 loopback passed")

Close the terminal and check usb status and broker clients; DTR-aware applications should cause the USB broker client to disconnect. Physically unplugging the native USB cable is the definitive detach test. To test observer mode, assign a console test client as writer before opening /dev/ttyACM0; USB should connect as an observer, receive UART output, and discard host-originated input until ownership is granted.

Power down and remove the DE-9 pin 3-to-2 jumper before connecting an external serial peer.

Wi-Fi foundation

Wi-Fi is managed independently of the serial-session broker. It provides network connectivity and recovery access-point policy for the HTTPS service. The firmware does not run plaintext HTTP, DNS interception, a captive portal, NAPT, or any TCP serial listener.

Configuration uses four fixed station-profile slots. Lower numeric priority values are tried first, with slot number breaking ties. Profiles support WPA2/WPA3 mixed operation or require WPA3-SAE. ESP-IDF's station threshold can express “WPA2 or stronger” but not a strict WPA2-only maximum, so the configuration does not pretend to offer a distinct WPA2-only mode. Each profile attempt has a 12-second association/DHCP deadline. After all enabled profiles fail, the manager uses exponential retry delays from 2 to 60 seconds.

AP policy is independent of the station profiles:

Policy	Behavior
off	Station only; never start the fallback AP
fallback	Start the AP immediately when no profiles exist, or after one failed profile cycle; disable it after station connectivity has remained stable for 30 seconds
always	Keep AP and station active concurrently

Fresh defaults enable Wi-Fi with fallback policy, AP channel 6, a MAC-suffixed SSID such as ESP32-SAK-A1B2C3, and a randomly generated 16-character password. The initial random credential is saved to NVS automatically when possible so it remains stable across reboot. Retrieve it deliberately from the physical UART0 administration console with wifi ap show-secret.

The wifi command provides:

wifi status
wifi profiles
wifi start|stop|reconnect
wifi profile set <slot> <priority> <mixed|wpa3> <ssid>
wifi profile secret <slot>
wifi profile enable|disable|delete <slot>
wifi ap policy <off|fallback|always>
wifi ap ssid <ssid>
wifi ap channel <1..11>
wifi ap secret|show-secret
wifi save|load|defaults|reset
wifi counters|clear-counters
wifi ping <host> [count]
wifi nslookup <host>
wifi traceroute <host> [max-hops]

The network diagnostics are also registered as root aliases, so ping, nslookup, and traceroute are equivalent to their wifi-prefixed forms. ping accepts 1–20 probes and supports IPv4 or IPv6. nslookup prints unique numeric IPv4/IPv6 results. traceroute is currently IPv4-only, sends one ICMP Echo probe per hop, accepts 1–30 hops, and uses a one-second timeout per hop; routers that suppress ICMP replies appear as *.

Ordinary status and profile output never displays passwords. wifi profile secret and wifi ap secret read through a dedicated no-echo UART0 prompt, keeping credentials out of the command line and its history. SSIDs containing spaces can be quoted. Profile and AP edits apply to the working RAM configuration and restart Wi-Fi asynchronously if it is running; use wifi save explicitly to persist them. wifi start and wifi stop also change the working enabled-at-boot setting, which becomes persistent only after wifi save.

A typical station setup is:

wifi profile set 0 10 mixed "your SSID"
wifi profile secret 0
wifi profile enable 0
wifi save
wifi reconnect
wifi status

The fallback AP uses Espressif's default 192.168.4.1/24 network for now. AP clients receive addresses through its DHCP server but are not routed to the station network. ESP32-S3 has one 2.4 GHz radio, so in AP+STA mode the AP follows the connected station's channel. Station connection attempts and scans can temporarily increase AP latency, and clients can briefly reconnect when the channel moves.

Wi-Fi credentials currently reside as plaintext in the application-owned wifi_app/config NVS blob. Selecting WIFI_STORAGE_RAM prevents the ESP-IDF driver from creating a second persistent credential copy, but it does not encrypt the application's blob. The reserved nvs_key partition alone does not enable encryption. NVS encryption, secure boot, flash encryption, and core-dump credential exposure require a deliberate later security phase.

Wi-Fi validation

1. Boot with no station profiles. wifi status should report ap-only, and the generated SSID should be visible from another device.
2. Use wifi ap show-secret, join the AP, confirm a 192.168.4.x lease, and run wifi ping 192.168.4.1. The authenticated HTTPS page should be reachable at https://192.168.4.1/.
3. Configure and enable a WPA2/WPA3 station profile using the example above. wifi status should progress through connecting, waiting-ip, and online and display the acquired address, channel, RSSI, and negotiated authentication.
4. Reboot and verify profile and AP credential persistence.
5. Configure two profiles with different priorities, make the first unavailable, and verify failover to the second after its timeout.
6. Make all profiles unavailable and verify fallback AP startup plus increasing retry delays in wifi status/wifi counters.
7. Test wifi ap policy always while online and confirm both interfaces remain available; expect the AP channel to follow the station.
8. Test wifi stop, wifi start, and wifi reconnect while confirming UART0 and native USB serial operation remain unaffected.
9. If available, test a WPA3-only profile and a wrong password, then inspect the disconnect reason and counters.

Phase 5A authenticated HTTPS foundation

One ESP-IDF HTTPS server listens on TCP port 443 across whichever AP and station interfaces are active. There is no plaintext port 80 listener. This slice intentionally exposes only:

GET /
GET /api/status

Both endpoints require HTTP Basic authentication over TLS. / is a small self-contained status landing page; /api/status returns JSON containing uptime plus non-secret Wi-Fi, serial-service, broker, native-USB, and HTTPS state/counters. No WebSocket, xterm.js terminal, or web broker client exists in Phase 5A.

On first boot, the device generates and persists:

• username admin and a random 24-character Base64URL-safe password;
• an ECDSA P-256 private key;
• a device-specific self-signed SHA-256 certificate valid from 2025-01-01 through 2049-12-31;
• certificate SANs for 192.168.4.1 and a MAC-suffixed name such as esp32-sak-a1b2c3.local.

The certificate and credentials remain stable across ordinary reboot and OTA-slot changes until explicitly rotated. The .local name is included for future hostname discovery, but this phase does not yet advertise mDNS; use the AP address or the station address reported by wifi status.

The physical UART0 administration console provides:

web
web help
web status
web start|stop
web counters|clear-counters
web credentials show
web credentials rotate --force
web certificate info
web certificate rotate --force
web reset --force

web and web help print the same usage summary. web credentials show is the intended first-boot credential-retrieval path. Rotation and reset operations write NVS immediately rather than creating RAM-only secrets. Destructive operations require a literal --force; web reset --force is also the recovery path for an incompatible or damaged web_sec/material blob and starts HTTPS with the recovered material. Invalid stored material is never overwritten automatically, and HTTPS failure never disables UART0, native USB, the serial core, or Wi-Fi recovery.

The self-signed certificate is not trusted by browsers or host tools by default. After retrieving the password, validate from a host connected to the fallback AP with:

curl -k -u 'admin:YOUR_24_CHARACTER_PASSWORD' https://192.168.4.1/
curl -k -u 'admin:YOUR_24_CHARACTER_PASSWORD' https://192.168.4.1/api/status

A request without credentials should return 401 Unauthorized and a WWW-Authenticate challenge:

curl -k -i https://192.168.4.1/api/status

Inspect and compare the live certificate with web certificate info:

openssl s_client -connect 192.168.4.1:443 -servername esp32-sak-device.local </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates -fingerprint -sha256

Replace the example SNI name with the DNS SAN printed by web certificate info. SNI is not required for this single-certificate server, but supplying the device name makes the test representative of future hostname use. Repeat the fingerprint check after reboot to confirm persistence, then optionally test each explicit rotation command and verify that only the requested material changes.

HTTPS is memory-bounded to two simultaneous client sockets; ESP-IDF documents approximately 40 KiB per TLS socket. Basic authentication is acceptable here only because plaintext HTTP is disabled. It is an initial administration mechanism, not the final authorization design.

Current security limitation: the web password and ECDSA private key are stored as plaintext in the application-owned web_sec/material NVS blob, just as Wi-Fi credentials are currently plaintext in wifi_app/config. The reserved nvs_key partition does not activate NVS encryption. ESP-IDF 5.5 also keeps an internal heap copy of the active TLS private key and does not guarantee zeroization when that allocation is freed. Do not treat the current firmware as resistant to physical flash or RAM extraction; NVS encryption, flash encryption, secure boot, protected OTA, secret-aware core-dump handling, and framework-level key zeroization belong to the later hardening phase.

Phase 0 diagnostics

The top-level status command retains quick MAX3243 signal-state inspection. Potentially disruptive hardware-characterization operations are grouped below debug so the primary help page stays concise:

status
debug transceiver <enable|disable>
debug drivers <tx 0|1> <dtr 0|1> <rts 0|1>
debug loopback-a
debug loopback-b
debug valid-test
debug uart-loopback <baud> [8N1|8E1|8O1|8N2|7E1|7O1] [bytes]
debug uart-suite
debug cts-flow-test
debug rts-flow-test

Run debug without a subcommand for its usage summary. debug uart-loopback defaults to 8N1 and 256 bytes. Its accepted payload range is 1–512 bytes. debug uart-suite covers 300 through 250000 baud and all supported frame formats. debug cts-flow-test verifies transmit gating and exact resumption, while debug rts-flow-test uses UART2 as an internal traffic generator to verify automatic receive backpressure. Follow the command-specific loopback wiring in wiring.md before invoking any test.

A mutex-protected port lease prevents diagnostics, UART1 service startup, and future clients from reconfiguring the same GPIOs concurrently. If a UART driver cannot be removed during cleanup, the firmware keeps the MAX3243 shut down and marks the port faulted until reboot rather than exposing an ambiguous hardware state.

The onboard RGB LED reports the most recent test-harness state:

Color	Meaning
Blue	Idle; waiting for a command
Yellow/orange	Test running
Green	Last test passed
Red	Last test failed

This hardware profile uses the onboard RGB LED on GPIO48. Official ESP32-S3-DevKitC-1 v1.1 boards commonly use GPIO38 instead, and compatible boards or clones may vary. A different board revision requires an adjusted board pin profile before running this firmware.

License

This project is licensed under the GNU General Public License version 3 only (GPL-3.0-only). This is compatible with using the GPLv3 releases of wolfSSL and wolfSSH later. Third-party components remain subject to their respective licenses.