Skip to content

Add $TRANSPORT and $IP_PROTO macros #4673

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 17 commits into from
Nov 23, 2023
Merged

Add $TRANSPORT and $IP_PROTO macros #4673

merged 17 commits into from
Nov 23, 2023

Conversation

bazsi
Copy link
Collaborator

@bazsi bazsi commented Oct 15, 2023 •
edited
Loading

This branch implements two new name-value pairs populated by syslog-ng to make it easier to create drill-downs into the message streams via metrics.

$TRANSPORT:

This branch implements $TRANSPORT, a new key-value pair, automatically populated by syslog-ng to indicate the kind of transport that was used to receive the message.

The $TRANSPORT is a higher level protocol, at least compared to $PROTO (contains the L3 protocol id from /etc/protocols) or $IP_PROTO

$IP_PROTO:
$IP_PROTO contains 4 for ipv4 and 6 for ipv6.

These are the current values from the afsocket module:

  • rfc3164+tls
  • rfc3164+tcp
  • rfc3164+udp
  • rfc3164+proxied-tls
  • rfc3164+<custom logproto like altp>
  • rfc5426: new style syslog over udp
  • rfc5425: new style syslog over tls
  • rfc6587: new style syslog over tcp
  • rfc5424+<custom logproto like altp>: new style syslog over a logproto plugin
  • unix-stream
  • unix-dgram

I have also added support for this to:

  • otel source (uses "otlp")
  • mqtt source (uses "mqtt")
  • hypr source (uses "hypr-api")

The branch also contains a couple of "other" patches:

  • a small refactor to define "predefined" name-value pairs that are not static in NVTable
  • a fix for an uninitialized value in transport-aux-data
  • the $PROTO value for transport(tls) connections is now "6"(tcp)
  • v4-mapped addresses in IPv6 sources should be translated to simple IPv4 addresses, instead of their more complex v4-mapped syntax (e.g. 192.168.1.1 instead of ::ffff:192.168.1.1)
  • inline documentation for the explicit source side batching feature in Python

Some of these could be extracted, others we are depending on with the new features.

@bazsi bazsi force-pushed the add-transport-value branch from 54ef805 to a10cc3b Compare October 15, 2023 18:43
@kira-syslogng
Copy link
Contributor

Build FAILURE

@bazsi bazsi force-pushed the add-transport-value branch from a10cc3b to 361aee1 Compare October 16, 2023 08:33
@kira-syslogng
Copy link
Contributor

Build FAILURE

@bazsi bazsi mentioned this pull request Oct 16, 2023
Closed
@bazsi bazsi force-pushed the add-transport-value branch from 361aee1 to fbe2ce3 Compare October 16, 2023 12:12
@bazsi
Copy link
Collaborator Author

bazsi commented Oct 16, 2023

I dropped the patch that causes MacOS builds to build and added it to #4674

@bazsi bazsi force-pushed the add-transport-value branch from fbe2ce3 to 67b5083 Compare October 16, 2023 12:23
@kira-syslogng
Copy link
Contributor

Build FAILURE

@bazsi bazsi force-pushed the add-transport-value branch from 67b5083 to 3bf5184 Compare October 19, 2023 08:05
@kira-syslogng
Copy link
Contributor

Build FAILURE

bazsi added 16 commits October 20, 2023 10:06
@bazsi
logmsg: add "predefined" handles
47b4cfd 
Add the possibility of adding a set of LM_V_XXXX handles that are allocated
at startup but not "static" in the sense that they are stored in the
NVTable as normal dynamic name-value pairs.

This mechanism replaces LOG_MSG_GET_VALUE_HANDLE_STATIC() macro and
is easier to read and avoids races.

Signed-off-by: Balazs Scheidler <balazs.scheidler@axoflow.com>
@bazsi
affile: use LM_V_FILE_NAME instead of dynamically allocating the NVHa…
3577b4c 
…ndle

Signed-off-by: Balazs Scheidler <balazs.scheidler@axoflow.com>
@bazsi
transport: initialize proto member to 0
22c7e0a 
Signed-off-by: Balazs Scheidler <balazs.scheidler@axoflow.com>
@bazsi
transport: fix whitespace issues in transport-tls.c
5e41fe1 
Signed-off-by: Balazs Scheidler <balazs.scheidler@axoflow.com>
@bazsi
transport: report TLS encrypted traffic as TCP in $PROTO
be0b47c 
Signed-off-by: Balazs Scheidler <balazs.scheidler@axoflow.com>
@bazsi
gsockaddr: add g_sockaddr_inet6_is_v4_mapped() function
951fad6 
Signed-off-by: Balazs Scheidler <balazs.scheidler@axoflow.com>
@bazsi
template: add $IP_PROTOCOL macro
63f7d75 
Signed-off-by: Balazs Scheidler <balazs.scheidler@axoflow.com>
@bazsi
afsocket: added transport_name support to TransportMapper
b925be1 
Signed-off-by: Balazs Scheidler <balazs.scheidler@axoflow.com>
@bazsi
afsocket: populate $TRANSPORT using TransportMapper->transport_name
4341fb2 
Signed-off-by: Balazs Scheidler <balazs.scheidler@axoflow.com>
@bazsi
logthrsource: add support for setting $TRANSPORT
2e28a70 
Signed-off-by: Balazs Scheidler <balazs.scheidler@axoflow.com>
@bazsi
mqtt: initialize transport_name to "mqtt"
c6e51c3 
Signed-off-by: Balazs Scheidler <balazs.scheidler@axoflow.com>
@bazsi
otel: set $TRANSPORT to "otlp"
d9133e2 
Signed-off-by: Balazs Scheidler <balazs.scheidler@axoflow.com>
@bazsi
python-modules: document close_batch() and auto_close_batches
0ff73a8 
Signed-off-by: Balazs Scheidler <balazs.scheidler@axoflow.com>
@bazsi
python-modules: added set_transport_name() method to LogFetcher & Log…
1833da1 
…Source

Signed-off-by: Balazs Scheidler <balazs.scheidler@axoflow.com>
@bazsi
python-modules/hypr: set $TRANSPORT to hypr-api
960ed0b 
Signed-off-by: Balazs Scheidler <balazs.scheidler@axoflow.com>
@bazsi
news: added news entries
274807c 
Signed-off-by: Balazs Scheidler <balazs.scheidler@axoflow.com>
@bazsi bazsi force-pushed the add-transport-value branch from 3bf5184 to 274807c Compare October 20, 2023 08:07
@bazsi
Copy link
Collaborator Author

bazsi commented Oct 20, 2023

Found the reason for the @kira-syslogng breakage. Let's hope it works out this time.

@bazsi
gsockaddr: V4-mapped addresses should be formatted as an IPv4 address
196eb7b 
V4-mapped address space is a compatibility feature of dual-stack TCP/IP
stacks, where IPv4 connections can be established to IPv6 capable sockets.

When this happens, the client IP address would be a V4-mapped address,
as determined by the IN6_IS_ADDR_V4MAPPED() function, which
is equivalent to an "::FFFF:<ipv4>" address (see RFC3493)

This change would trickle into how we fill $HOST if we didn't find a
hostname there, previously these would have the form of "::FFFF:ipv4",
with this one they would become "ipv4" just as if we received them using
an ipv4 specific source.

Signed-off-by: Balazs Scheidler <balazs.scheidler@axoflow.com>
@kira-syslogng
Copy link
Contributor

Build FAILURE

@bazsi
Copy link
Collaborator Author

bazsi commented Oct 30, 2023

@kira-syslogng retest this please;

@MrAnno MrAnno merged commit c4c127a into syslog-ng:master Nov 23, 2023
@eldarnash eldarnash mentioned this pull request Nov 25, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
1 more reviewer

@MrAnno MrAnno MrAnno approved these changes

Reviewers whose approvals may not affect merge requirements
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@bazsi @kira-syslogng @MrAnno