-
-
Notifications
You must be signed in to change notification settings - Fork 2.1k
Fix IOBasePayload reading entire files into memory instead of chunking #11139
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #11139 +/- ##
==========================================
+ Coverage 98.28% 98.85% +0.57%
==========================================
Files 132 131 -1
Lines 42908 42712 -196
Branches 2357 2307 -50
==========================================
+ Hits 42173 42225 +52
+ Misses 560 337 -223
+ Partials 175 150 -25
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
CodSpeed Performance ReportMerging #11139 will not alter performanceComparing Summary
|
Backport to 3.12: 💚 backport PR created✅ Backport PR branch: Backported as #11141 🤖 @patchback |
Backport to 3.13: 💚 backport PR created✅ Backport PR branch: Backported as #11142 🤖 @patchback |
What do these changes do?
This PR fixes a memory consumption issue where
IOBasePayload
andTextIOPayload
would read entire files into memory when streaming large files. The bug was introduced in #10915 and caused the payload classes to readremaining_content_length
bytes at once instead of chunking the reads.The fix ensures that both payload classes never read more than
READ_SIZE
(64KB) at a time, regardless of the file size or content length. This prevents out-of-memory errors when uploading large files using file-like objects with the aiohttp client.Are there changes in behavior for the user?
No behavioral changes for users. The fix is transparent and only affects memory usage during file streaming. Files are still uploaded correctly, but now they're read in 64KB chunks instead of potentially reading gigabytes into memory at once.
Is it a substantial burden for the maintainers to support this?
No, this is a minimal fix that restores the original intended behavior. The changes are:
READ_SIZE
as the first parameter tomin()
in_read_and_available_len()
for bothIOBasePayload
andTextIOPayload
_read()
method is called with appropriate chunk sizesThe fix is straightforward and maintains backward compatibility while fixing a critical memory issue.
Related issue number
Fixes #11138
Checklist
CONTRIBUTORS.txt
CHANGES/
foldername it
<issue_or_pr_num>.<type>.rst
(e.g.588.bugfix.rst
)if you don't have an issue number, change it to the pull request
number after creating the PR
.bugfix
: A bug fix for something the maintainers deemed animproper undesired behavior that got corrected to match
pre-agreed expectations.
.feature
: A new behavior, public APIs. That sort of stuff..deprecation
: A declaration of future API removals and breakingchanges in behavior.
.breaking
: When something public is removed in a breaking way.Could be deprecated in an earlier release.
.doc
: Notable updates to the documentation structure or buildprocess.
.packaging
: Notes for downstreams about unobvious side effectsand tooling. Changes in the test invocation considerations and
runtime assumptions.
.contrib
: Stuff that affects the contributor experience. e.g.Running tests, building the docs, setting up the development
environment.
.misc
: Changes that are hard to assign to any of the abovecategories.
Make sure to use full sentences with correct case and punctuation,
for example:
Use the past tense or the present tense a non-imperative mood,
referring to what's changed compared to the last released version
of this project.