C++ modules: remove direct libstdc++ linkage #4933

MrAnno · 2024-05-03T14:11:23Z

libtool should follow transitive dependencies and link against the appropriate C++ standard lib (libc++ or libstdc++) automatically.

Fixes #4932, but let's wait for @alltilla's review because something doesn't seem right here.

bazsi · 2024-05-03T14:19:57Z

I think this was originally needed because sometimes we link c++ objects into an otherwise c program using gcc and not g++

MrAnno · 2024-05-03T14:23:04Z

Those C++ objects are managed by libtool. In my understanding, libtool should handle transitive dependencies automatically in case the C++ library is static (noinst_LTLIBRARIES).

MrAnno · 2024-05-03T14:29:56Z

Hm, it may not be the case, and my patch may work only because we link against gRPC/protobuf libraries, which pull the standard lib transitively.

I guess the proper fix would be detecting which standard c++ library we should use, but I can't come up with an easy implementation (checking clang++ vs. g++ is not a good one)

bazsi · 2024-05-04T05:05:00Z

Yeah, sometimes g++ is using libc++ and I guess clang can also use libstdc++. this is more of a platform decision. A simple and limited fix could be to just use libc++ on FreeBSD and libstdc++ on everything else.

But I can see your PR compiled, so maybe we don't need any of those?

czanik · 2024-05-06T09:42:05Z

It compiled on FreeBSD, however there seem to be problems running the result:

root@fb132:~/freebsd/syslog-ng4-devel # syslog-ng -V
syslog-ng 4 (4.7.1.167.g3d81f41)
Config version: 4.2
Installer-Version: 4.7.1.167.g3d81f41
Revision: 
Compile-Date: May  6 2024 13:25:36
Module-Directory: /usr/local/lib/syslog-ng
Module-Path: /usr/local/lib/syslog-ng
Include-Path: /usr/local/share/syslog-ng/include
Available-Modules: afamqp,http,geoip2-plugin,loki,otel,bigqueryError opening plugin module; module='cloud_auth', error='/usr/local/lib/syslog-ng/libcloud_auth.so: Undefined symbol "_ZTISt13runtime_error"'
Error opening plugin module; module='mod-java', error='Shared object "libjvm.so" not found, required by "libmod-java.so"'
Error opening plugin module; module='examples', error='/usr/local/lib/syslog-ng/libexamples.so: Undefined symbol "_ZTISt12length_error"'
,json-plugin,kafka,afmongodb,mod-python,redis,riemann,afsmtp,afsnmp,afsql,add-contextual-data,affile,afprog,afsocket,afstomp,afuser,appmodel,azure-auth-header,basicfuncs,cef,confgen,cryptofuncs,csvparser,correlation,disk-buffer,graphite,hook-commands,kvformat,linux-kmsg-format,stardate,map-value-pairs,metrics-probe,pseudofile,regexp-parser,secure-logging,syslogformat,system-source,tags-parser,tfgetent,timestamp,xml,rate-limit-filter
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-IPv6: on
Enable-Spoof-Source: on
Enable-TCP-Wrapper: on
Enable-Linux-Caps: off
Enable-Systemd: off

cloud_auth and libexamples has undefined symbols.

czanik · 2024-05-06T09:59:47Z

And the same problem on openSUSE Leap 15.5, which is unlike FreeBSD, is GCC-based:

czplaptop:~ # syslog-ng -V
syslog-ng 4 (4.7.1.167.g3d81f41)
Config version: 4.2
Installer-Version: 4.7.1.167.g3d81f41
Revision: 
Module-Directory: /usr/lib64/syslog-ng
Module-Path: /usr/lib64/syslog-ng
Include-Path: /usr/share/syslog-ng/include
Error opening plugin module; module='examples', error='/usr/lib64/syslog-ng/libexamples.so: undefined symbol: __gxx_personality_v0'
Error opening plugin module; module='cloud_auth', error='/usr/lib64/syslog-ng/libcloud_auth.so: undefined symbol: _ZTVN10__cxxabiv117__class_type_infoE'
Available-Modules: add-contextual-data,affile,afprog,afsocket,afstomp,afuser,appmodel,azure-auth-header,basicfuncs,cef,confgen,cryptofuncs,csvparser,disk-buffer,graphite,hook-commands,json-plugin,kvformat,linux-kmsg-format,map-value-pairs,pseudofile,regexp-parser,sdjournal,secure-logging,stardate,syslogformat,system-source,tags-parser,tfgetent,timestamp,xml,correlation,metrics-probe,pacctformat,rate-limit-filter,mod-python
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-IPv6: on
Enable-Spoof-Source: on
Enable-TCP-Wrapper: on
Enable-Linux-Caps: on
Enable-Systemd: on

czanik · 2024-05-06T10:02:52Z

Yeah, sometimes g++ is using libc++ and I guess clang can also use libstdc++. this is more of a platform decision. A simple and limited fix could be to just use libc++ on FreeBSD and libstdc++ on everything else.

But I can see your PR compiled, so maybe we don't need any of those?

It's not Linux vs. FreeBSD. It's GCC vs. CLANG. And CLANG is also used on MacOS and a number of Linux distros (Gentoo AFAIR from an earlier report).

This patch seems to work for OpenTelemetry, but not for examples or cloud_auth.

MrAnno · 2024-05-06T10:40:36Z

Yeah, we need to explicitly link against the correct C++ standard lib, because libtool won't track that for us. The "correct" way of doing that is not the easiest because even with gcc vs clang, you can override the default libc++ with a flag.

czanik · 2024-05-06T10:54:03Z

Did a new compile on FreeBSD, this time after editing the source with sed:

for i in `find . -type f` ; do gsed -i 's/stdc++/c++/g' $i ; done

The examples module seems to be OK, but cloud_auth is still problematic:

root@fb132:~/freebsd/syslog-ng4-devel # syslog-ng -V
syslog-ng 4 (4.7.1.166.g0f51c01)
Config version: 4.2
Installer-Version: 4.7.1
Revision: 
Compile-Date: May  6 2024 14:21:13
Module-Directory: /usr/local/lib/syslog-ng
Module-Path: /usr/local/lib/syslog-ng
Include-Path: /usr/local/share/syslog-ng/include
Available-Modules: afamqp,http,geoip2-plugin,loki,otel,bigqueryError opening plugin module; module='cloud_auth', error='/usr/local/lib/syslog-ng/libcloud_auth.so: Undefined symbol "_ZTISt13runtime_error"'
Error opening plugin module; module='mod-java', error='Shared object "libjvm.so" not found, required by "libmod-java.so"'
,json-plugin,kafka,afmongodb,mod-python,redis,riemann,afsmtp,afsnmp,afsql,add-contextual-data,affile,afprog,afsocket,afstomp,afuser,appmodel,azure-auth-header,basicfuncs,cef,confgen,cryptofuncs,csvparser,correlation,disk-buffer,examples,graphite,hook-commands,kvformat,linux-kmsg-format,stardate,map-value-pairs,metrics-probe,pseudofile,regexp-parser,secure-logging,syslogformat,system-source,tags-parser,tfgetent,timestamp,xml,rate-limit-filter
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-IPv6: on
Enable-Spoof-Source: on
Enable-TCP-Wrapper: on
Enable-Linux-Caps: off
Enable-Systemd: off

alltilla · 2024-05-06T12:32:00Z

FTR I have just tried with link_all_deplibs, and it did not solve the issue, the stdc++ dep is not transited.

_LT_TAGVAR([link_all_deplibs],)=yes
_LT_TAGVAR([link_all_deplibs],[CXX])=yes

https://ftp.gnu.org/old-gnu/Manuals/libtool-1.4.2/html_node/libtool_70.html

$ ldd install/lib/syslog-ng/libexamples.so | grep stdc | wc -l
0

MrAnno · 2024-05-06T13:24:45Z

@alltilla Thanks for trying :)

I'll try to solve it "perfectly" in an 1-hour time box. If I can't find a clean way, I'll just go with the gcc-clang check.

MrAnno · 2024-05-07T09:54:51Z

I've found the most portable way of fixing this, I'll update the PR soon.

MrAnno · 2024-05-07T12:03:10Z

@czanik Can you give it another go, please? :)

czanik · 2024-05-08T06:01:37Z

Thanks @MrAnno. It seems that the GCC side is now OK (tested on openSUSE Leap 15.5, Tumbleweed needs a 3GB download before I can test). The FreeBSD / CLANG side also improved, only cloud_auth remained problematic, but otel, loki, etc load fine:

root@fb132:~/freebsd/syslog-ng4-devel # syslog-ng -V
syslog-ng 4 (4.7.1.196.gb9ec945)
Config version: 4.2
Installer-Version: 4.7.1.196.gb9ec945
Revision: 
Compile-Date: May  8 2024 09:44:27
Module-Directory: /usr/local/lib/syslog-ng
Module-Path: /usr/local/lib/syslog-ng
Include-Path: /usr/local/share/syslog-ng/include
Available-Modules: afamqp,http,geoip2-plugin,loki,otel,bigqueryError opening plugin module; module='cloud_auth', error='/usr/local/lib/syslog-ng/libcloud_auth.so: Undefined symbol "_ZTISt13runtime_error"'
Error opening plugin module; module='mod-java', error='Shared object "libjvm.so" not found, required by "libmod-java.so"'
,json-plugin,kafka,afmongodb,mod-python,redis,riemann,afsmtp,afsnmp,afsql,add-contextual-data,affile,afprog,afsocket,afstomp,afuser,appmodel,azure-auth-header,basicfuncs,cef,confgen,cryptofuncs,csvparser,correlation,disk-buffer,examples,graphite,hook-commands,kvformat,linux-kmsg-format,stardate,map-value-pairs,metrics-probe,pseudofile,regexp-parser,secure-logging,syslogformat,system-source,tags-parser,tfgetent,timestamp,xml,rate-limit-filter
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-IPv6: on
Enable-Spoof-Source: on
Enable-TCP-Wrapper: on
Enable-Linux-Caps: off
Enable-Systemd: off

czanik · 2024-05-08T08:15:08Z

Tumbleweed is also OK, which uses all the latest versions from GCC and other dependencies:

tumbleweed:~ # syslog-ng -V
syslog-ng 4 (4.7.1.196.gb9ec945)
Config version: 4.2
Installer-Version: 4.7.1.196.gb9ec945
Revision: 
Module-Directory: /usr/lib64/syslog-ng
Module-Path: /usr/lib64/syslog-ng
Include-Path: /usr/share/syslog-ng/include
Available-Modules: tags-parser,tfgetent,timestamp,xml,mod-python,otel,http,pacctformat,cloud_auth,add-contextual-data,affile,afprog,afsocket,afstomp,afuser,appmodel,azure-auth-header,basicfuncs,cef,confgen,correlation,cryptofuncs,csvparser,disk-buffer,examples,graphite,hook-commands,json-plugin,kvformat,linux-kmsg-format,map-value-pairs,metrics-probe,pseudofile,rate-limit-filter,regexp-parser,sdjournal,secure-logging,stardate,syslogformat,system-source
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-IPv6: on
Enable-Spoof-Source: on
Enable-TCP-Wrapper: on
Enable-Linux-Caps: on
Enable-Systemd: on

So, it seems that the only problem point left is cloud_auth with CLANG / FreeBSD.

We should link against the appropriate C++ standard lib (libc++ or libstdc++), but libtool does not follow such dependencies through static libraries. In order not to hard-code the C++ standard lib, the nodist_EXTRA_*_SOURCES trick can be used to force using the C++ linker: https://www.gnu.org/software/automake/manual/html_node/Libtool-Convenience-Libraries.html CMake does this by default in case it detects a static C++ library dependency. Signed-off-by: László Várady <laszlo.varady@anno.io>

Signed-off-by: László Várady <laszlo.varady@anno.io>

kira-syslogng · 2024-05-08T10:50:48Z

Build FAILURE

MrAnno · 2024-05-08T11:02:55Z

@kira-syslogng retest this please

MrAnno · 2024-05-08T12:10:16Z

@kira-syslogng retest this please

MrAnno mentioned this pull request May 3, 2024

clang c++ support #4932

Closed

MrAnno requested a review from alltilla May 3, 2024 14:16

MrAnno force-pushed the remove-direct-stdc++-linking branch from e7da594 to 6948e75 Compare May 7, 2024 11:59

MrAnno force-pushed the remove-direct-stdc++-linking branch from 6948e75 to 163c894 Compare May 8, 2024 10:22

MrAnno changed the title ~~grpc: remove direct libstdc++ linkage~~ C++ modules: remove direct libstdc++ linkage May 8, 2024

news: add entry for syslog-ng#4933

b9d1d2b

Signed-off-by: László Várady <laszlo.varady@anno.io>

alltilla approved these changes May 8, 2024

View reviewed changes

MrAnno merged commit b1b5f00 into syslog-ng:master May 8, 2024

C++ modules: remove direct libstdc++ linkage #4933

C++ modules: remove direct libstdc++ linkage #4933

Uh oh!

Conversation

MrAnno commented May 3, 2024

Uh oh!

bazsi commented May 3, 2024

Uh oh!

MrAnno commented May 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MrAnno commented May 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bazsi commented May 4, 2024

Uh oh!

czanik commented May 6, 2024

Uh oh!

czanik commented May 6, 2024

Uh oh!

czanik commented May 6, 2024

Uh oh!

MrAnno commented May 6, 2024

Uh oh!

czanik commented May 6, 2024

Uh oh!

alltilla commented May 6, 2024

Uh oh!

MrAnno commented May 6, 2024

Uh oh!

MrAnno commented May 7, 2024

Uh oh!

MrAnno commented May 7, 2024

Uh oh!

czanik commented May 8, 2024

Uh oh!

czanik commented May 8, 2024

Uh oh!

kira-syslogng commented May 8, 2024

Uh oh!

MrAnno commented May 8, 2024

Uh oh!

MrAnno commented May 8, 2024

Uh oh!

Uh oh!

MrAnno commented May 3, 2024 •

edited

Loading

MrAnno commented May 3, 2024 •

edited

Loading