Skip to content

Conversation

dongcarl
Copy link
Contributor

@dongcarl dongcarl commented May 17, 2019

Secondary dependencies don't need to be shared. Comes at theuni's request.

To do: add documentation so future secondary dependencies aren't shared. DONE


Let's merge this first as #15844 is based on this PR.

@dongcarl dongcarl force-pushed the 2019-05-depends-static-if-secondary-dep branch from 740303c to a08982c Compare May 17, 2019 16:06
@dongcarl dongcarl marked this pull request as ready for review May 17, 2019 16:06
@practicalswift
Copy link
Contributor

Concept ACK

@DrahtBot
Copy link
Contributor

DrahtBot commented May 17, 2019

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #15844 (depends: Purge libtool archives by dongcarl)
  • #15277 ([Help Wanted] contrib: Enable building in Guix containers by dongcarl)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@dongcarl dongcarl force-pushed the 2019-05-depends-static-if-secondary-dep branch from a08982c to 256b204 Compare May 17, 2019 19:22
@fanquake
Copy link
Member

Concept ACK

@dongcarl dongcarl force-pushed the 2019-05-depends-static-if-secondary-dep branch from 256b204 to 27226e6 Compare May 22, 2019 20:05
@dongcarl
Copy link
Contributor Author

Added more detailed justifications to depends/packages.md.

@fanquake
Copy link
Member

Rebooted all the tests.

@fanquake
Copy link
Member

utACK 27226e6

@practicalswift
Copy link
Contributor

utACK 27226e6

1 similar comment
@jb55
Copy link
Contributor

jb55 commented May 24, 2019 via email

@bitcoin bitcoin deleted a comment from DrahtBot May 24, 2019
@dongcarl dongcarl force-pushed the 2019-05-depends-static-if-secondary-dep branch from 27226e6 to 1420928 Compare May 28, 2019 14:58
@dongcarl
Copy link
Contributor Author

dongcarl commented May 28, 2019

Update: squashed the two commits.

Let's merge this first as #15844 is based on this PR.

Secondary dependencies don't need to be shared.
@dongcarl dongcarl force-pushed the 2019-05-depends-static-if-secondary-dep branch from 1420928 to 3560d8c Compare May 29, 2019 15:04
@theuni
Copy link
Member

theuni commented May 29, 2019

Concept ACK, thanks for picking this up!

Reviewing.

@fanquake
Copy link
Member

diffoscope x86_64-pc-linux-gnu/lib x86_64-pc-linux-gnu-3560d8c/lib

libX11-xcb.so.1.0.0
libX11.la
libX11.so
libX11.so.6
libX11.so.6.3.0
libXau.a
libXau.la
+libXext.a
libXext.la
-libXext.so
-libXext.so.6
-libXext.so.6.4.0
libboost_chrono-mt.a
libboost_filesystem-mt.a
libboost_prg_exec_monitor-mt.a
libboost_system-mt.a
libboost_test_exec_monitor-mt.a
libboost_thread-mt.a
libboost_timer-mt.a
libboost_unit_test_framework-mt.a
libcrypto.a
libdb-4.8.a
libdb.a
libdb_cxx-4.8.a
libdb_cxx.a
+libdbus-1.a
libdbus-1.la
-libdbus-1.so
-libdbus-1.so.3
-libdbus-1.so.3.14.11
libevent.a
libevent.la
libevent_core.a
libevent_core.la
libevent_extra.a
libevent_extra.la
libevent_pthreads.a
libevent_pthreads.la
+libexpat.a
libexpat.la
-libexpat.so
-libexpat.so.1
-libexpat.so.1.6.8
libfontconfig.la
libfontconfig.so
libfontconfig.so.1
libfontconfig.so.1.9.2
libfreetype.la
libfreetype.so
libfreetype.so.6

@theuni
Copy link
Member

theuni commented May 29, 2019

A few things about this make me nervous, so I'm going to look in more depth.

  • Some of these static libs (libXext for example) obviously still work when compiled as non-pic, implying that they never end up linked into anything. In these cases where they're not used, we may be able to avoid the dependencies altogether. Maybe only headers are needed?
  • I'm nervous about duplicate symbols in static libs where versioning tricks might have otherwise been used for shared libs. Qt, for example, builds internal copies of some of our libs. I can't come up with any tangible problems though, so this point is very thin.
  • I would've expected these changes to require a little more hand-holding for pkg-config. For example, I would have expected this to be necessary.

@dongcarl
Copy link
Contributor Author

dongcarl commented May 31, 2019

In this PR we identify 4 secondary deps that should be built statically:

  • dbus
  • expat
  • libXext
  • xtrans

However, only expat required being built --with-pic for depends builds to succeed, which is rather odd. Therefore, the rest of the deps must not be linked in, or are headers-only.

Results of my investigation (please verify these as I might be completely wrong):

libXext

The only package that we declare as depending on libXext is qt, and it doesn't actually... Tested by removing the package definition and references to it, everything seems to compile fine.

xtrans

It seems that xtrans is just code+headers and not a library that needs to be linked in. We can verify this by looking at the build outputs.

Its configure is very confused as to why we supply it with --disable-{static,shared}. It is, however, required as a build-time dependency for libX11.

dbus

The only reference to dbus is in the configure options for qt: -dbus-runtime. Which indicates to qt to dynamically open libdbus-1 at runtime (with dlopen?). I believe this means that we don't actually use the dbus we build. (This one I'm the least sure about)

@theuni

laanwj added a commit that referenced this pull request Jun 5, 2019
8541cbe depends: libX*: --disable-malloc0returnsnull in conf (Carl Dong)
0e75263 depends: libXext: Bump to 1.3.3 to fix _XEatDataWords (Carl Dong)
683b7d7 depends: Purge libtool archives (Carl Dong)
1420928 depends: Build secondary deps statically. (Carl Dong)

Pull request description:

  ```
  We use pkg-config where we can, which generally replaces libtool at a
  higher level and does not have the same downsides as libtool. These
  archives sit in our depends tree with no purpose and pollute the final
  bitcoin build with massive overlinking.
  ```

  See [here](https://wiki.gentoo.org/wiki/Project:Quality_Assurance/Handling_Libtool_Archives) for an explanation of the various problems libtool archives can cause.

  Unrelated in every way except in spirit: `-D__LIBTOOL_IS_A_FOOL__`!!

  -----

  This PR is based on #16041, and therefore should be merged after #16041.

ACKs for commit 8541cb:

Tree-SHA512: 76030cf32361f0b1cfe14e3827a0cbec99994e7da00a56194ca40cf6cf7d87f78552f49d03d41ce9cf9b642992b90d993578ed1f0ad6bae15cd3f1c88dfaa4b0
@DrahtBot
Copy link
Contributor

DrahtBot commented Jun 5, 2019

Needs rebase

@fanquake
Copy link
Member

fanquake commented Jun 5, 2019

This has been merged as part of #15844. I've opened #16150 to keep track of followup depends changes.

@fanquake fanquake closed this Jun 5, 2019
@laanwj
Copy link
Member

laanwj commented Jun 5, 2019

Oops. Didn't realize this was a separate PR and reviewed this as part of #15844.

@maflcko maflcko reopened this Jun 5, 2019
@maflcko maflcko closed this Jun 5, 2019
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Jun 6, 2019
8541cbe depends: libX*: --disable-malloc0returnsnull in conf (Carl Dong)
0e75263 depends: libXext: Bump to 1.3.3 to fix _XEatDataWords (Carl Dong)
683b7d7 depends: Purge libtool archives (Carl Dong)
1420928 depends: Build secondary deps statically. (Carl Dong)

Pull request description:

  ```
  We use pkg-config where we can, which generally replaces libtool at a
  higher level and does not have the same downsides as libtool. These
  archives sit in our depends tree with no purpose and pollute the final
  bitcoin build with massive overlinking.
  ```

  See [here](https://wiki.gentoo.org/wiki/Project:Quality_Assurance/Handling_Libtool_Archives) for an explanation of the various problems libtool archives can cause.

  Unrelated in every way except in spirit: `-D__LIBTOOL_IS_A_FOOL__`!!

  -----

  This PR is based on bitcoin#16041, and therefore should be merged after bitcoin#16041.

ACKs for commit 8541cb:

Tree-SHA512: 76030cf32361f0b1cfe14e3827a0cbec99994e7da00a56194ca40cf6cf7d87f78552f49d03d41ce9cf9b642992b90d993578ed1f0ad6bae15cd3f1c88dfaa4b0
deadalnix pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this pull request Apr 1, 2020
Summary:
```
  We use pkg-config where we can, which generally replaces libtool at a
  higher level and does not have the same downsides as libtool. These
  archives sit in our depends tree with no purpose and pollute the final
  bitcoin build with massive overlinking.

See here for an explanation of the various problems libtool archives can
cause:
https://wiki.gentoo.org/wiki/Project:Quality_Assurance/Handling_Libtool_Archives
```

Backport of core [[bitcoin/bitcoin#15844 | PR15844]].

This does actually more that described, as it includes [[bitcoin/bitcoin#16041 | PR16041]] (it was
merged as 1 PR instead of 2).

Test Plan: Run the Gitian builds.

Reviewers: #bitcoin_abc, deadalnix

Reviewed By: #bitcoin_abc, deadalnix

Differential Revision: https://reviews.bitcoinabc.org/D5621
ftrader pushed a commit to bitcoin-cash-node/bitcoin-cash-node that referenced this pull request Aug 13, 2021
Summary:
```
  We use pkg-config where we can, which generally replaces libtool at a
  higher level and does not have the same downsides as libtool. These
  archives sit in our depends tree with no purpose and pollute the final
  bitcoin build with massive overlinking.

See here for an explanation of the various problems libtool archives can
cause:
https://wiki.gentoo.org/wiki/Project:Quality_Assurance/Handling_Libtool_Archives
```

Backport of core [[bitcoin/bitcoin#15844 | PR15844]].

This does actually more that described, as it includes [[bitcoin/bitcoin#16041 | PR16041]] (it was
merged as 1 PR instead of 2).

Test Plan: Run the Gitian builds.

Reviewers: #bitcoin_abc, deadalnix

Reviewed By: #bitcoin_abc, deadalnix

Differential Revision: https://reviews.bitcoinabc.org/D5621
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Dec 16, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants