Skip to content

# Fix: Respect Content-Length when Content-Encoding is present #1653

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 11 commits into
base: master
Choose a base branch
from

Conversation

ali90h
Copy link

@ali90h ali90h commented Aug 7, 2025

Fix: Respect Content-Length when Content-Encoding is present

Problem Summary

HTTPie's --download flag incorrectly reports "Incomplete download" errors when receiving responses with Content-Encoding: gzip (or other encodings), even when the download is actually complete.

Example error:

http: error: Incomplete download: size=5084527; downloaded=42846965

This occurs because HTTPie was comparing the Content-Length header (which represents compressed size) against the decompressed content size.

Root Cause

The issue violates RFC 9110 § 8.6, which states:

"The Content-Length field indicates the number of octets in the body of the message after any content codings have been applied."

When Content-Encoding: gzip is present:

  • Content-Length: 5084527 = size of compressed data
  • Downloaded content: 42,846,965 bytes = size after decompression
  • HTTPie was incorrectly comparing these mismatched values

Fix Overview

  • Modified download validation logic to respect encoded vs. decoded sizes
  • When Content-Encoding is present, validate against compressed payload size
  • Without Content-Encoding, behavior remains unchanged
  • Added proper tracking of raw bytes received vs. decompressed content

Key Changes

  1. Size Validation: Compare Content-Length against raw bytes received when content encoding is present
  2. Progress Reporting: Show progress based on encoded bytes for encoded responses
  3. Encoding Detection: Check for all standard encodings (gzip, deflate, br, compress)

Testing

  • Added comprehensive regression test with local HTTP server serving gzip content
  • Test verifies download completes without false "Incomplete download" errors
  • Validates downloaded gzip file integrity
  • Ensures non-encoded downloads continue working as before

Backward Compatibility

  • ✅ Fully backward compatible
  • ✅ No breaking changes to API or behavior
  • ✅ Existing functionality unchanged for non-encoded responses
  • ✅ Progress display and validation work correctly in both cases

Related Issues

Fixes #1642

Acknowledgments

Thanks to @kohnalex for the detailed bug report and reproduction steps, which made this fix straightforward to implement and test.

@ali90h
Copy link
Author

ali90h commented Aug 7, 2025

@httpie/maintainers This PR fixes the Content-Length validation bug (#1642) that was causing false "Incomplete download" errors with gzipped responses.

Would appreciate a review when you have time. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

🐛 Bug Report: http --download misinterprets Content-Length when Content-Encoding: gzip is set
1 participant