cc-wrapper: generating compile_commands.json using cc-wrapper #192694

danielbarter · 2022-09-24T04:30:54Z

Description of changes

Modifying the cc-wrapper so that it can be used to generate compile_commands.json files. Here is the corresponding package:
https://github.com/danielbarter/mini_compile_commands

Inserts a post-wrapper-hook which the user can provide to insert a hook into the wrapper.

mini-compile-commands provides the function mini-compile-commands.wrap which can be used to add compile command generating functionality to a standard environment:

with (import <nixpkgs> {});
let llvm = llvmPackages_latest;
in (mkShell.override {stdenv = ( mini-compile-commands.wrap llvm.stdenv );}) {
   buildInputs = [ cmake gtest ];
}

Things done

nixos-discourse · 2022-09-24T16:40:36Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/using-nix-infrastructure-to-reliably-generate-compile-commands-json/21961/1

danielbarter · 2022-09-26T18:58:39Z

still tinkering a little with the mini compile commands code. Should have it settled by later today

pkgs/development/tools/mini-compile-commands/default.nix

nixos-discourse · 2022-10-05T17:42:15Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-ready-for-review/3032/1238

nixos-discourse · 2022-10-09T15:31:06Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/using-nix-infrastructure-to-reliably-generate-compile-commands-json/21961/14

fricklerhandwerk · 2022-10-09T19:46:54Z

After at first being confused what this is supposed to be about and then reading the linked Discourse thread, this seems like a valuable addition.

Once someone approves technical fitness, please ping me for a review of the user-facing documentation. I can't promise to take a lot of time or be very quick, but I really would like this to be at least reasonably good and fear that otherwise it will be quite useless to the uninitiated.

Sorry if this sounds harsh - I really appreciate your contribution! But I can't even quickly make constructive comments without a few basic preconditions:

use one sentence per line in markdown, then one can make suggestions directly or ask questions about individual statements.
link every term you use that is defined in this manual to the respective definition, and link specialist technical terms to authoritative definitions. otherwise users (and reviewers like me, who don't know all the context but can judge if they understand what you try to convey, as a proxy for future users) will be lost immediately and won't be able to make use of your otherwise useful feature
I seriously don't understand the description of what the thing does, and the example doesn't help. Please run this against someone who knows software development but not the thing you're trying to describe, and answer all questions specific to the problem that the person raises in the manual section. This may be a lot of work. But just as you test your code, you have to test your documentation - before handing it in for review.

Usually I'd jump in as a test subject, but there's only so much I can do in a day, and this topic is too far away from my current priorities. And yet, I would veto merging this PR before documentation is in a state that is worthwhile to discuss in more detail.

danielbarter · 2022-10-10T16:01:03Z

Thanks for the feedback @fricklerhandwerk! very much appreciated. Will get to work on addressing the issues.

…i-compile-commands

danielbarter · 2022-10-10T16:40:37Z

OK, I have put each sentence in the docs on a new line, which hopefully will make discussing easier.

I totally agree that currently, I am not doing a fantastic job explaining. I think the main issue is that there are two orthogonal things happening here, and I am conflating them.

Adding post-wrapper-hook.sh into the compiler wrapper. This allows users to inject their own code into the wrapper.
mini-compile-commands, which is an example application of the post-wrapper-hook.sh. Generating correct compile_commands.json files is currently a little difficult on nixos because of the compiler wrappers. The standard tool people use, bear, struggles to distinguish between the wrapped and unwrapped compiler invocations. Fortunately, given a programmable wrapper, it is very easy to generate a correct compile_commands.json, hence this PR!

I 100% agree that nix documentation is one of its weak spots, and am committed to getting the documentation right in this case! I'll start accumulating commits until we have this resolved so that we don't lose comments. Will squish down at the end.

fzakaria · 2022-10-12T22:08:05Z

This will make CLion integration with Nix first class right?

danielbarter · 2022-10-12T22:53:11Z

@fzakaria: I haven't used CLion before, but based on my reading of https://www.jetbrains.com/help/clion/compilation-database.html, I would say yes

danielbarter · 2022-10-12T22:55:10Z

I was just digging around in the ld-wrapper and noticed this:

nixpkgs/pkgs/build-support/bintools-wrapper/ld-wrapper.sh

Line 259 in 21b14e9

source @out@/nix-support/post-link-hook

which is exactly what I am adding to the cc-wrapper. For the cc-wrapper it does need to go above the compiler invocation though, because of the execs. Also, i should probably change the name post-wrapper-hook to something like cc-wrapper-hook.

Does anyone know why the compilers are execd from cc-wrapper instead of just being called like for the ld-wrapper? I could imagine it is for performance reasons, but I am not sure. If we got rid of the execs, then we could move the hook after the compiler invocation and call it post-compile-hook which would be nice and consistent with what happens in ld-wrapper

nixos-discourse · 2022-10-19T22:09:32Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/using-nix-infrastructure-to-reliably-generate-compile-commands-json/21961/20

danielbarter force-pushed the mini-compile-commands branch 2 times, most recently from 6fc914a to 4ef9345 Compare September 26, 2022 18:10

ofborg bot requested review from edolstra, adevress, WilliButz, SuperSandro2000, globin, mmahut and np September 26, 2022 18:20

danielbarter marked this pull request as ready for review September 26, 2022 18:44

danielbarter requested a review from Ericson2314 as a code owner September 26, 2022 18:44

danielbarter changed the title ~~Generating compile_commands.json using nix infrastructure~~ Generating compile_commands.json using nix compiler wrappers Sep 26, 2022

danielbarter force-pushed the mini-compile-commands branch 3 times, most recently from 891f054 to de22708 Compare September 27, 2022 18:35

danielbarter changed the title ~~Generating compile_commands.json using nix compiler wrappers~~ cc-wrapper: generating compile_commands.json using cc-wrapper Sep 28, 2022

SuperSandro2000 reviewed Sep 28, 2022

View reviewed changes

pkgs/development/tools/mini-compile-commands/default.nix Outdated Show resolved Hide resolved

danielbarter force-pushed the mini-compile-commands branch 2 times, most recently from c569071 to 2306333 Compare October 4, 2022 04:15

ofborg bot requested a review from SuperSandro2000 October 4, 2022 05:46

danielbarter force-pushed the mini-compile-commands branch 2 times, most recently from 47bd5c5 to ae0886c Compare October 5, 2022 18:38

danielbarter force-pushed the mini-compile-commands branch from ae0886c to ee89e01 Compare October 9, 2022 17:30

danielbarter requested a review from fricklerhandwerk as a code owner October 9, 2022 17:30

github-actions bot added the 8.has: documentation This PR adds or changes documentation label Oct 9, 2022

cc-wrapper: added a post-wrapper hook and example of its use with min…

0b9c3e2

…i-compile-commands

danielbarter force-pushed the mini-compile-commands branch from 36dadb4 to 0b9c3e2 Compare October 10, 2022 16:25

danielbarter added 4 commits October 10, 2022 11:02

factoring the docs into two orthogonal sections

1481954

tweaked wording a little

c0e5d9b

typo

99bad26

typo

e318c4a

danielbarter added 3 commits October 19, 2022 13:59

switching to python3Minimal

eabb6cd

set allowedRequisites = null in wrapped env

192a47f

adding a more explicit example to the docs

b21d11d

danielbarter force-pushed the mini-compile-commands branch from d751537 to b21d11d Compare October 19, 2022 22:28

danielbarter mentioned this pull request Oct 26, 2022

cc-wrapper: adding a cc-wrapper-hook to the cc-wrapper #197937

Merged

13 tasks

danielbarter closed this Oct 27, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

cc-wrapper: generating compile_commands.json using cc-wrapper #192694

cc-wrapper: generating compile_commands.json using cc-wrapper #192694

Uh oh!

danielbarter commented Sep 24, 2022 •

edited

Loading

Uh oh!

nixos-discourse commented Sep 24, 2022

Uh oh!

danielbarter commented Sep 26, 2022

Uh oh!

Uh oh!

nixos-discourse commented Oct 5, 2022

Uh oh!

nixos-discourse commented Oct 9, 2022

Uh oh!

fricklerhandwerk commented Oct 9, 2022

Uh oh!

danielbarter commented Oct 10, 2022 •

edited

Loading

Uh oh!

danielbarter commented Oct 10, 2022 •

edited

Loading

Uh oh!

fzakaria commented Oct 12, 2022

Uh oh!

danielbarter commented Oct 12, 2022 •

edited

Loading

Uh oh!

danielbarter commented Oct 12, 2022 •

edited

Loading

Uh oh!

nixos-discourse commented Oct 19, 2022

Uh oh!

Uh oh!

Uh oh!

cc-wrapper: generating compile_commands.json using cc-wrapper #192694

cc-wrapper: generating compile_commands.json using cc-wrapper #192694

Uh oh!

Conversation

danielbarter commented Sep 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of changes

Things done

Uh oh!

nixos-discourse commented Sep 24, 2022

Uh oh!

danielbarter commented Sep 26, 2022

Uh oh!

Uh oh!

nixos-discourse commented Oct 5, 2022

Uh oh!

nixos-discourse commented Oct 9, 2022

Uh oh!

fricklerhandwerk commented Oct 9, 2022

Uh oh!

danielbarter commented Oct 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

danielbarter commented Oct 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fzakaria commented Oct 12, 2022

Uh oh!

danielbarter commented Oct 12, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

danielbarter commented Oct 12, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nixos-discourse commented Oct 19, 2022

Uh oh!

Uh oh!

danielbarter commented Sep 24, 2022 •

edited

Loading

danielbarter commented Oct 10, 2022 •

edited

Loading

danielbarter commented Oct 10, 2022 •

edited

Loading

danielbarter commented Oct 12, 2022 •

edited

Loading

danielbarter commented Oct 12, 2022 •

edited

Loading