Skip to content

Gzip: compatibility with recent zlib-ng 2.2.x versions. #403

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jan 9, 2025

Conversation

pluknet
Copy link
Contributor

@pluknet pluknet commented Dec 23, 2024

It now uses 5/4 times more memory for the pending buffer.

Further, a single allocation is used, with up to 128 additional bytes for alignment. Though, this fits into "8 * (64 + sizeof(void*))" alignment adjustment used for previous zlib-ng versions. The comment was updated to reflect this.

Closes #263.

Copy link
Contributor

@arut arut left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are two changes in zlib-ng we are addressing here.

First, zlib-ng/zlib-ng@a3fb271 which increases pending buffer size by a factor of 5/4 under certain circumstances. I agree with the change to increase pending buffer size and with the relevant part of the commit log.

Second, zlib-ng/zlib-ng@5c7a066 which changes allocation model to make only one big allocation instead of multiple small ones as before. Here, according to my calculations (see https://github.com/zlib-ng/zlib-ng/blob/develop/deflate.c#L202), the overall buffer size should increase. Particularly, 4 * (64 + sizeof(void *)) should be changed to 7 * (64 + sizeof(void *)):

  • 64 per allocation x5
  • 64 for total size padding
  • 64 for window padding (non-s390)
  • 7 * sizeof(void *) for the deflate_allocs structure

The original 4 * (64 + sizeof(void *)) is no longer needed since ZALLOC is no longer used.

I did not get the alignment adjustment explanation of why the 4->7 (you mentioned 8) change can be ignored.

@pluknet
Copy link
Contributor Author

pluknet commented Dec 30, 2024

Previously that was unconditional increase in memory, now this is roundup (see related macros). Subranges are large enough in size (cannot prove that atm b/c afk), so the roundup should be nop.

@pluknet
Copy link
Contributor Author

pluknet commented Jan 6, 2025

@arut

Now with numbers as mentioned.

As stated in 8747191, previously used alloc_aligned() wrapper gave unconditional extra memory space allocation, see ZALLOC and alloc_aligned().
Now it uses alloc_deflate() with sub-allocations roundup such as in PAD_16 and PAD_64, and the total size aligned to PAD_64(curr_size + (WINDOW_PAD_SIZE - 1)).
So there is no issue here in practice for particular allocations used in nginx.

In particular:

  • default wbits=15 and memlevel=8:
2025/01/06 17:29:38 [debug] 35065#0: *2 malloc: 0000000102ED4800:352560
window is 65536 bytes, offset 0, padded 0
prev is 65536 bytes, offset 65536, padded 0
head is 131072 bytes, offset 131072, padded 0
pending is 81920 bytes, offset 262144, padded 0
state is 6144 bytes, offset 344064, padded 0
alloc is 56 bytes, offset 350208, padded 0
2025/01/06 17:46:03 [debug] 35334#0: *2 gzip alloc: n:1 s:350336 a:350336 p:0000000102BF8800
Buffer alloc is 350336 bytes, offset 0, padded 8
  • smallest allowed wbits=9 and memlevel=1 (both directives undocumented), and wbits=13 capping removed e.g. by gzip_comp_leve=2;.
2025/01/06 17:46:41 [debug] 35344#0: *2 malloc: 000000010301C800:142256
window is 1024 bytes, offset 0, padded 0
prev is 1024 bytes, offset 1024, padded 0
head is 131072 bytes, offset 2048, padded 0
pending is 640 bytes, offset 133120, padded 0
state is 6144 bytes, offset 133760, padded 0
alloc is 56 bytes, offset 139904, padded 0
2025/01/06 17:46:41 [debug] 35344#0: *2 gzip alloc: n:1 s:140032 a:140032 p:000000010301C800
Buffer alloc is 140032 bytes, offset 0, padded 8

In both cases, this gives the last sub-allocation padded to 16 bytes alignment, and the total_size allocated with further extra WINDOW_PAD_SIZE - 1 padded to 64 alignment, which gives 8+64=72 extra bytes in total. The last padding is where "up to 128 additional bytes for alignment" comes from in the commit message, as with curr_size perfectly unaligned in the worst theoretical scenario.

Hope this helps.

It now uses 5/4 times more memory for the pending buffer.

Further, a single allocation is now used, which takes additional 56 bytes
for deflate_allocs in 64-bit mode aligned to 16, to store sub-allocation
pointers, and the total allocation size now padded up to 128 bytes, which
takes theoretically 200 additional bytes in total.  This fits though into
"4 * (64 + sizeof(void*))" additional space for ZALLOC used in zlib-ng
2.1.x versions.  The comment was updated to reflect this.
@pluknet
Copy link
Contributor Author

pluknet commented Jan 8, 2025

Second, zlib-ng/zlib-ng@5c7a066 which changes allocation model to make only one big allocation instead of multiple small ones as before. Here, according to my calculations (see https://github.com/zlib-ng/zlib-ng/blob/develop/deflate.c#L202), the overall buffer size should increase. Particularly, 4 * (64 + sizeof(void *)) should be changed to 7 * (64 + sizeof(void *)):

* 64 per allocation x5

* 64 for total size padding

* 64 for window padding (non-s390)

* `7 * sizeof(void *)` for the `deflate_allocs` structure

The original 4 * (64 + sizeof(void *)) is no longer needed since ZALLOC is no longer used.

I agree with calculation, they make sense in the worst theoretical case.
In practice, (almost) all sub-allocations have a good fixed size or a power of 2 multiplier, which does not require padding.
An exception might be deflate_state: its size isn't fixed and may require additional space to align the subsequent deflate_allocs (although in my tests deflate_state has a well chosen size 6144, or 3 * 2^11).

This changes to:
* 64+64 for total size padding
* 7 * sizeof(void *) for the deflate_allocs structure
* 16 for the deflate_allocs structure alignment

I did not get the alignment adjustment explanation of why the 4->7 (you mentioned 8) change can be ignored.

That was a typo, thx, should be read as "4". Fixed in the commit message.

Copy link
Contributor

@arut arut left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks ok.

@arut arut merged commit 57d54fd into nginx:master Jan 9, 2025
1 check passed
@pluknet pluknet deleted the zlib branch January 17, 2025 00:45
drupol pushed a commit to NixOS/nixpkgs that referenced this pull request May 6, 2025
Nginx version 1.27.4 and above is compatible with zlib-ng versions 2.2.x:
nginx/nginx#403
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

"gzip filter failed to use preallocated memory" with current zlib-ng
2 participants