feat: Stream request bodies from files, iterables, and responses

[vdusek]

Summary

set_record, the run_input of start / call / metamorph, and HttpClient.call(data=...) accept an io.IOBase stream such as an open file, an iterable of bytes or str chunks, or a streamed HttpResponse. The client uploads it in chunks without holding it in memory, so one Actor's OUTPUT record can become another Actor's input. The async client also accepts async iterables and aiofiles-style readers.

Streaming is experimental. The docs say so, and the first streamed body in a process emits a UserWarning.

Behavior

• An io.IOBase source is read in 64 KiB chunks. Any other object with a read method is read whole and sent as one chunk.
• Only a seekable io.IOBase source is retried, rewound before each attempt. Other sources get a single attempt.
• Streamed bodies are never compressed. A file-like set_record value used to be compressed, so uploading from a file handle now sends more bytes; file.read() keeps the old behavior.
• If the API answers 408 because the body didn't arrive within about 5 minutes, the client logs a warning suggesting a temporary file.
• send_request of a custom transport receives bytes | Iterator[bytes] | None (async: AsyncIterator[bytes]). It only sees an iterator when a caller passes a streamable value, so existing transports keep working.

Docs and tests

A "Streaming uploads" section on the streaming concept page, plus updates to the compression and HTTP client pages, the custom transport examples, and the README. Unit tests cover StreamedRequestBody, the retry pipeline, and set_record over Impit and HTTPX2; two integration tests cover a 3 MiB upload and a record piped between stores.

Issues

• Closes Support streaming request bodies to avoid buffering large uploads in memory #972

✍️ Drafted by Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 96.08696% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.30%. Comparing base (88770f9) to head (ffad3a4).
⚠️ Report is 4 commits behind head on master.

Files with missing lines	Patch %	Lines
src/apify_client/types.py	0.00%	7 Missing ⚠️
src/apify_client/http_clients/_streamed_body.py	98.83%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1060      +/-   ##
==========================================
+ Coverage   95.22%   95.30%   +0.08%     
==========================================
  Files          59       60       +1     
  Lines        5548     5753     +205     
==========================================
+ Hits         5283     5483     +200     
- Misses        265      270       +5     

Flag	Coverage Δ
integration	91.48% <70.86%> (-0.36%)	⬇️
unit	87.74% <96.08%> (+0.36%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Pijukatel]

I ran some tests with long-lasting streams. Currently, the API will return 408 (treated as not retryable) after around 300s-360s of streaming duration. There is no setting in the client to prevent this as far as I know.

This fails the upload and, in some cases, it can leave the source stream open.

Should there be an update to the API code?

This is not an edge case, as streaming makes a lot of sense for large objects or some incoming data source - so for it to last long is not that unusual.

The least we can do is to document this limit and clearly show it in the logs when it happens, but this limit makes the feature far less useful.