Garbled long pastes to CircuitPython: one raw-paste window at a time (#50)

[bdbarnett]

Fixes #50.

What you'd see: a long exec or run to a CircuitPython board over serial sometimes arrived garbled. So did serial put with the drive off. It always went wrong the same way: the paste kept its length, but one 64-byte block held the next block's bytes, and that next block came through again straight after it. Put another way, one USB packet was lost and its successor arrived twice. A syntax error, or a string with a patch of wrong characters, is how that shows up.

Cause: a race in CircuitPython's USB stack on its ESP32 ports, where TinyUSB runs in a FreeRTOS task of its own.

• lib/tinyusb/src/device/usbd.c clears the OUT endpoint's busy flag and only then calls cdcd_xfer_cb, which copies the packet out of the endpoint buffer.
• If the VM task reads the CDC FIFO in that gap, tud_cdc_n_read → _prep_out_transaction re-arms the endpoint on the same buffer.
• mpremote's raw-paste writer keeps two 128-byte windows in flight, so the host's next packet is already waiting, and it lands on the packet that hasn't been copied yet.

This isn't mpftp's chunking. The host respects the flow control. The firmware loses the data after the flow control has let it through.

Fix: mpftp/rawpaste.py adds a serial transport whose raw-paste writer sends one window at a time. It waits for the board's \x01 before each next window, which the board sends only once it has read everything sent. So nothing arrives while a packet is still in the endpoint buffer. The sidecar uses the paced writer only when the board runs CircuitPython. MicroPython keeps mpremote's writer.

What I proved on the T-Embed (CircuitPython 10.3.0 on COM25; identified by boot_out.txt's UID on the CIRCUITPY drive; no soft reset, no erase, no file written):

path	writer	32,000-char pastes intact
mpftp run --follow	main	22 of 30
mpftp run --follow	this branch (same 30 inputs)	30 of 30
mpremote directly	stock	89 of 120 (two runs of 60, in both orders)
mpremote directly	paced	120 of 120

Each paste is a random base64 string. The board prints its length and CRC-32, and the host compares them with its own.

Speed: paced ran at 43-44 KB/s. Stock ran at 55 KB/s on one run and 6.5 KB/s on the other.

The unit tests drive a fake board that does raw-paste as pyexec.c does. They check that the paced writer never has more than one window unread, and that mpremote's own writer, against the same fake, has more than one (the control). They also check the stop-early \x04 path, and that the sidecar paces CircuitPython only.

Not done, and your call: the firmware race itself is still there for any other host that runs ahead of the board, mpremote included. It would be an upstream TinyUSB/CircuitPython report. I haven't filed or drafted one.