Codecov Report

Merging #9218 (a9a5cca) into master (90ce084) will decrease coverage by 10.97%.
Report is 4 commits behind head on master.
The diff coverage is n/a.

❗ Current head a9a5cca differs from pull request most recent head 2977f9f. Consider uploading reports for the commit 2977f9f to get more accurate results

@@             Coverage Diff             @@
##           master    #9218       +/-   ##
===========================================
- Coverage   70.18%   59.22%   -10.97%     
===========================================
  Files         218      209        -9     
  Lines       47742    45292     -2450     
  Branches    11373     9388     -1985     
===========================================
- Hits        33508    26824     -6684     
- Misses      11748    16218     +4470     
+ Partials     2486     2250      -236     

see 427 files with indirect coverage changes

@mikezackles Ok, it's starting to get useful result now:

glib_dep = dependency('glib-2.0', static: true)
static_library('foo', 'foo.c', dependencies: glib_dep.as_link_whole())

This creates libfoo.a that contains glib from system. Let me know if it works for you, especially on OSX.

@nirbheek You could be interested by this too, I remember you suggested this kind of solution a loooooong time ago.

I'm getting ValueError: embedded null byte from with output.open('wb') as ofile on OS X. Maybe a null byte in the filename? Might be able to dig some more later.

Can you attach the .a file to help debug that? Probably will have to add it into a unit test anyway.

Sorry I was trying to do just that, but this OS X machine is graying out the file for upload for some reason.

Seems it's a github thing? I emailed it to you.

This seems like a lot of work to avoid just creating an extracted objects object from a bunch of .o files, or am i missing something?

This seems like a lot of work to avoid just creating an extracted objects object from a bunch of .o files, or am i missing something?

This is for the case where the static library comes from the system, you don't have .o files until you manually extract the .a, unless there is something I'm missing?

On second thought, I think it's better to extract at configure time. The cons is it means reconfiguring when system deps change which should be pretty rare. But doing it at build time means using a Python wrapper for every link which has significant performance impact for each rebuild. Also from an implementation pov it is easier to extract at configure time because it is backend agnostic. I think keeping implementation self contained is important too for a niche use case, we cannot justify the maintenance burden otherwise.

Also extracting archives will explode the number of args passed to the linker, that will easily exceed the limit on Windows. Doing it at configure time means the backend generator can switch to using a rsp file AFAIK.

As an alternative thing, maybe we could consider adding a top level function like merge_static_libs('newname', lib1, lib2). No other build system does static library merging so maybe it would be better to be explicit about it?

@jpakkane Hmm, you could also want to link_whole an external library into a shared_library(), that's also a case we don't currently support, do we? But in that case I don't think we need to extract the library, the compiler can do it for us, right? I think building on top of existing as_link_whole() API would be more elegant, but your solution is easier and more self contained indeed.

@mikezackles I updated the PR to fix the OSX archive case, included the .a you sent me in the unit test.

you could also want to link_whole an external library into a shared_library(), that's also a case we don't currently support, do we?

I don't know if it's intentional, but I think this does work currently even without using as_link_whole as I am able to ship shared libraries that link against my deps statically. (I haven't looked into why.) Otherwise this would be a much bigger problem for me.

@mikezackles I updated the PR to fix the OSX archive case, included the .a you sent me in the unit test.

Awesome, thank you!

I am able to ship shared libraries that link against my deps statically

I think in that case the shared library does not expose the full ABI of its static dependencies. The linker only picks what it needs from static deps and drop the rest.

does not expose the full ABI

Ah I see, that would make sense. I see the distinction now. (The dynamic libs I ship do not attempt to do this.)

On second thought, I think it's better to extract at configure time. The cons is it means reconfiguring when system deps change which should be pretty rare. But doing it at build time means using a Python wrapper for every link which has significant performance impact for each rebuild. Also from an implementation pov it is easier to extract at configure time because it is backend agnostic. I think keeping implementation self contained is important too for a niche use case, we cannot justify the maintenance burden otherwise.

Instead of doing it at configure time, why not make a new target that does the job? It will only run once per build.

Instead of doing it at configure time, why not make a new target that does the job? It will only run once per build.

If I don't extract the archive at configure time then I don't know the list of object files to pass to ar command when generating the link target. The way I used in a previous version of this patch is to wrap ar command completely with a meson script (see old commit 0a186ab).

If I don't extract the archive at configure time then I don't know the list of object files to pass to ar command when generating the link target

Why not list the contents of the static archive with ar t path/to/libfoo.a and lib /list drive:\path\to\foo.lib? In my testing that's 10x faster than extracting the static library.

Why not list the contents of the static archive with ar t path/to/libfoo.a and lib /list drive:\path\to\foo.lib?

Does it deal with duplicate names? Even if we only list the content at configure it still means we need to reconfigure when the lib changes, so that does not make much difference to just extract directly at configure time. Since we need an ar parser anyway to extract, better use the same code to list the content too, pretty sure if I add an option to not extract it's going to be faster than using extrnal command for that.

Does it deal with duplicate names? Even if we only list the content at configure it still means we need to reconfigure when the lib changes, so that does not make much difference to just extract directly at configure time.

A 100MB static library takes 200ms to extract for me and 20ms to list contents; that's going to add up very quickly I think. Semantically as well, I think it's wrong to do I/O-heavy tasks in setup.

Since we need an ar parser anyway to extract, better use the same code to list the content too, pretty sure if I add an option to not extract it's going to be faster than using extrnal command for that.

I agree that your parser is probably the best option for listing files in the archive, and not just extracting. Cross-platform support will probably be easier.

I think it's wrong to do I/O-heavy tasks in setup.

Ok, that's actually a good point, you're right.

Just for completeness (I don't think this is worth spending time on), I'll mention that we can also eliminate reconfigure when the static library changes by doing the same thing we do for avoiding relinks to shared libraries: have another target that checks whether the list of objects inside the static library is the same, and if it is, then we don't need to reconfigure.

@nirbheek I revisited this PR, it now does as you suggested: list object files contained in a static lib at configure but only extract them at build time with a custom target.

Hi!

Just wondering, would it be feasible to
submit Allow as_link_whole() on external dependencies
separately from an archive extractor?
For my problem that would be enough,
and maybe it can come easier than the
whole thing together?

I think the problem is the other way around. We need the archive extractor first, because implementing as_link_whole() depends on having been able to extract the archive.

as_link_whole() is based on using the individual *.o files.

That being said, it would be great to land this PR...

Then maybe I don't need as_link_whole()?
I definitely don't need an archive extractor.
All I need, is this:

libdjs = dependency('dj64static', static: true)
elf = custom_target(TARGET + '.elf',
  input: [ some_inputs, libdjs],
  command: [ just call linker -static here, and it deals with static libs on its own],
...

So what do I need for this?
Since I can call the linker directly, archive
extractor is not what I really need.
Maybe it would be better to answer in
#12709 if this appears unrelated to this PR.

Hi,

I also came across this functionlity where external dependency needs to be added with "as_link_whole()". What is the recommened method for doing it?