http driver compression options #4137

therandomstring · 2022-09-16T13:08:25Z

Added compression ability to http destination driver using partially built-in libcurl-capabilities (for accepting compressed headers) and own compression algorithm for outward compression using zlib.
Currently supported compression types are gzip and deflate.

resolves #3923

kira-syslogng · 2022-09-16T13:08:28Z

This user does not have permission to start the build. Can one of the admins verify this patch and start the build?
(admin: you have the next options (make sure you checked the code):
"ok to test" to accept this pull request (and further changes) for testing
"test this please" for a one time test run
"add to whitelist" add author of a Pull Request to whitelist (globally, be careful, it means this user can trigger kira for any PR)
do nothing -> CI won't start)

kira-syslogng · 2022-09-16T13:08:28Z

This user does not have permission to start the build. Can one of the admins verify this patch and start the build?
(admin: you have the next options (make sure you checked the code):
"ok to test" to accept this pull request (and further changes) for testing
"test this please" for a one time test run
"add to whitelist" add author of a Pull Request to whitelist (globally, be careful, it means this user can trigger kira for any PR)
do nothing -> CI won't start)

github-actions · 2022-09-16T13:08:38Z

No news file has been detected. Please write one, if applicable.

MrAnno · 2022-09-16T13:11:41Z

@kira-syslogng add to whitelist

MrAnno · 2022-09-16T13:12:51Z

@kira-syslogng do stresstest

kira-syslogng · 2022-09-16T13:39:47Z

Kira-stress-test: Build SUCCESS

bazsi · 2022-09-16T16:43:31Z

Wow. @ryanfaircloth I think you were interested in something like this.

Signed-off-by: Bálint Horváth <bal.horv.98@gmail.com>

alltilla

This is a really nice feature, good job!

Unfortunately, I did not have the time to finish my review, but I will get back to it later.
In the meantime I think it is worth to share some of my findings, so you have the possibility to start working on them.

modules/http/http.c

modules/http/http-curl-compression.h

modules/http/http.c

modules/http/http-curl-compression.c

modules/http/Makefile.am

news/feature-4137.md

modules/http/http-curl-compression.c

modules/http/http-curl-compression.h

kira-syslogng · 2022-10-03T14:11:57Z

Build FAILURE

kira-syslogng · 2022-10-05T12:15:13Z

Build FAILURE

Signed-off-by: Bálint Horváth <bal.horv.98@gmail.com>

modules/http/compression.c

modules/http/http.c

modules/http/tests/test_compression.c

modules/http/http-worker.c

modules/http/tests/test_http-compression.c

alltilla · 2023-07-31T12:17:33Z

Thanks for the changes, this looks good!

Signed-off-by: Bálint Horváth <bal.horv.98@gmail.com>

…compression' Signed-off-by: Bálint Horváth <bal.horv.98@gmail.com>

Signed-off-by: Bálint Horváth <bal.horv.98@gmail.com>

Zlib handles gzip and deflate the same, the difference is only a magic constant. (16 needs to be added to WBITS to indicate gzip compression to Zlib. See: https://www.zlib.net/manual.html chapter VIII.) Compressed data can be 10% larger than uncompressed (see Zlib `compress2()`) + 22 bytes are added for header/footer. Signed-off-by: Bálint Horváth <bal.horv.98@gmail.com>

…lemented compression types Signed-off-by: Bálint Horváth <bal.horv.98@gmail.com>

Signed-off-by: Bálint Horváth <bal.horv.98@gmail.com>

The specific test function is needed, because the compressed string is likely to contain null characters. Signed-off-by: Bálint Horváth <bal.horv.98@gmail.com>

alltilla

Amazing feature, good job!

From syslog-ng/syslog-ng#4137

MrAnno self-requested a review September 17, 2022 12:21

therandomstring added a commit to therandomstring/syslog-ng that referenced this pull request Sep 21, 2022

news: Added entry for syslog-ng#4137

d7812db

Signed-off-by: Bálint Horváth <bal.horv.98@gmail.com>

therandomstring force-pushed the http-curl-gzip branch from bad6ef9 to d7812db Compare September 21, 2022 12:20

alltilla reviewed Sep 29, 2022

View reviewed changes

alltilla reviewed Oct 1, 2022

View reviewed changes

therandomstring added a commit to therandomstring/syslog-ng that referenced this pull request Oct 5, 2022

news: Added entry for syslog-ng#4137

dbc4607

Signed-off-by: Bálint Horváth <bal.horv.98@gmail.com>

therandomstring force-pushed the http-curl-gzip branch from 3587300 to 166a8c2 Compare October 5, 2022 12:31

therandomstring added a commit to therandomstring/syslog-ng that referenced this pull request Oct 5, 2022

news: Added entry for syslog-ng#4137

818e21d

Signed-off-by: Bálint Horváth <bal.horv.98@gmail.com>

therandomstring force-pushed the http-curl-gzip branch from 7747609 to 818e21d Compare October 5, 2022 16:32

therandomstring added a commit to therandomstring/syslog-ng that referenced this pull request Oct 5, 2022

news: Added entry for syslog-ng#4137

237dde7

Signed-off-by: Bálint Horváth <bal.horv.98@gmail.com>

therandomstring force-pushed the http-curl-gzip branch from 818e21d to 237dde7 Compare October 5, 2022 16:34

therandomstring added a commit to therandomstring/syslog-ng that referenced this pull request Oct 5, 2022

news: Added entry for syslog-ng#4137

2e7997a

Signed-off-by: Bálint Horváth <bal.horv.98@gmail.com>

therandomstring force-pushed the http-curl-gzip branch from 237dde7 to 2e7997a Compare October 5, 2022 16:41

therandomstring added a commit to therandomstring/syslog-ng that referenced this pull request Oct 5, 2022

news: Added entry for syslog-ng#4137

1b743e6

Signed-off-by: Bálint Horváth <bal.horv.98@gmail.com>

therandomstring force-pushed the http-curl-gzip branch from 2e7997a to 1b743e6 Compare October 5, 2022 16:53

therandomstring added a commit to therandomstring/syslog-ng that referenced this pull request Oct 5, 2022

news: Added entry for syslog-ng#4137

06d033c

Signed-off-by: Bálint Horváth <bal.horv.98@gmail.com>

therandomstring added a commit to therandomstring/syslog-ng that referenced this pull request Oct 5, 2022

fixup! news: Added entry for syslog-ng#4137

ba5d83d

Signed-off-by: Bálint Horváth <bal.horv.98@gmail.com>

therandomstring force-pushed the http-curl-gzip branch from 1b743e6 to ba5d83d Compare October 5, 2022 17:12

therandomstring added a commit to therandomstring/syslog-ng that referenced this pull request Oct 6, 2022

news: Added entry for syslog-ng#4137

5b8c6e8

Signed-off-by: Bálint Horváth <bal.horv.98@gmail.com>

therandomstring added a commit to therandomstring/syslog-ng that referenced this pull request Oct 6, 2022

fixup! news: Added entry for syslog-ng#4137

a5fc2c6

Signed-off-by: Bálint Horváth <bal.horv.98@gmail.com>

therandomstring force-pushed the http-curl-gzip branch from ba5d83d to a5fc2c6 Compare October 6, 2022 11:51

MrAnno removed their request for review February 17, 2023 13:51

MrAnno marked this pull request as draft April 25, 2023 23:10

therandomstring added a commit to therandomstring/syslog-ng that referenced this pull request Jul 24, 2023

news: Added entry for syslog-ng#4137

76c47ac

Signed-off-by: Bálint Horváth <bal.horv.98@gmail.com>

therandomstring force-pushed the http-curl-gzip branch from 776f3c6 to d18fe5b Compare July 24, 2023 14:54

therandomstring added a commit to therandomstring/syslog-ng that referenced this pull request Jul 25, 2023

news: Added entry for syslog-ng#4137

18213c0

Signed-off-by: Bálint Horváth <bal.horv.98@gmail.com>

therandomstring force-pushed the http-curl-gzip branch from 274af59 to 192b1d7 Compare July 25, 2023 07:00

therandomstring changed the title ~~WIP: http driver compression options~~ http driver compression options Jul 25, 2023

therandomstring marked this pull request as ready for review July 25, 2023 07:41

alltilla reviewed Jul 31, 2023

View reviewed changes

Bálint Horváth added 2 commits August 1, 2023 09:23

http: Added compression files and skeleton

c39c794

Signed-off-by: Bálint Horváth <bal.horv.98@gmail.com>

http: Added variable to header 'accepted encoding' and 'http message …

0eeb91b

…compression' Signed-off-by: Bálint Horváth <bal.horv.98@gmail.com>

therandomstring added a commit to therandomstring/syslog-ng that referenced this pull request Aug 1, 2023

news: Added entry for syslog-ng#4137

3068e1f

Signed-off-by: Bálint Horváth <bal.horv.98@gmail.com>

therandomstring force-pushed the http-curl-gzip branch from 192b1d7 to 726936e Compare August 1, 2023 11:37

Bálint Horváth and others added 10 commits August 1, 2023 14:44

http: Added setters for content and header encoding

b486be0

Signed-off-by: Bálint Horváth <bal.horv.98@gmail.com>

http: Added compression skeleton

a7d0fc6

Signed-off-by: Bálint Horváth <bal.horv.98@gmail.com>

http: Introduced unified error handling scheme to work with yet unimp…

07b14be

…lemented compression types Signed-off-by: Bálint Horváth <bal.horv.98@gmail.com>

http: Added compression ability to workers

ce035c0

Signed-off-by: Bálint Horváth <bal.horv.98@gmail.com>

http: Introduced already existing compression capabilities to grammar

b8deacf

Signed-off-by: Bálint Horváth <bal.horv.98@gmail.com>

http: Added automake/autoconf support

28c1bf5

Signed-off-by: Bálint Horváth <bal.horv.98@gmail.com>

http: Fixed copyright testing

b393e14

Signed-off-by: Bálint Horváth <bal.horv.98@gmail.com>

news: Added entry for syslog-ng#4137

f8beac0

Signed-off-by: Bálint Horváth <bal.horv.98@gmail.com>

http: Added basic tests for compression.

b77018b

The specific test function is needed, because the compressed string is likely to contain null characters. Signed-off-by: Bálint Horváth <bal.horv.98@gmail.com>

therandomstring force-pushed the http-curl-gzip branch from d93601a to b77018b Compare August 1, 2023 12:47

alltilla approved these changes Aug 18, 2023

View reviewed changes

alltilla merged commit 1d69d32 into syslog-ng:master Aug 18, 2023

yaleman mentioned this pull request Aug 18, 2023

feat - http compression splunk/splunk-connect-for-syslog#2144

Closed

czanik mentioned this pull request Aug 23, 2023

latest git snapshot fails on SLES 12 #4603

Closed

fekete-robert pushed a commit to axoflow/axosyslog-core-docs that referenced this pull request Sep 30, 2023

[4.4][http] Documents accept-encoding and content-compression

f9d8438

From syslog-ng/syslog-ng#4137

http driver compression options #4137

http driver compression options #4137

Uh oh!

Conversation

therandomstring commented Sep 16, 2022

Uh oh!

kira-syslogng commented Sep 16, 2022

Uh oh!

kira-syslogng commented Sep 16, 2022

Uh oh!

github-actions bot commented Sep 16, 2022

Uh oh!

MrAnno commented Sep 16, 2022

Uh oh!

MrAnno commented Sep 16, 2022

Uh oh!

kira-syslogng commented Sep 16, 2022

Uh oh!

bazsi commented Sep 16, 2022

Uh oh!

alltilla left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kira-syslogng commented Oct 3, 2022

Uh oh!

kira-syslogng commented Oct 5, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alltilla commented Jul 31, 2023

Uh oh!

alltilla left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!