BTW, at some point one starts to wonder if using a generic template library makes more sense, but i'd rather not add more dependencies if it can be avoided.

Generally I like the idea of separating the templates from the tool, but I think it was easier to read without all the escaping that is needed now.
Also it doesn't really make sense to add this functionality to a tool specific to constfs, does it?
If there is shared/generic functionality for both use cases it seems better suited to move that code out to a generic tool and then use it within the specific helper tools (i.e. mkconstfs2, luawhatever etc.).

Thinking about lua, why not embed the lua modules as all other files (e.g. within constfs, but not restricted to it) and then simply use those files? Why is this different handling needed in the first place?

Generally I like the idea of separating the templates from the tool, but I think it was easier to read without all the escaping that is needed now.

The escaping is necessary if one wants to preserve whitespaces. The other formats supported in python's standards library (JSON, CSV, etc) need even more escaping. Hacking my own format is in my opinion not a good idea. The last option is to separate each chunk in its individual file.

Also it doesn't really make sense to add this functionality to a tool specific to constfs, does it?

I guess now it's not specific to constfs anymore, it is more of a mutant xxd command. How about renaming it xxd-plus.

then use it within the specific helper tools

Is there any advantage on writing super-thin wrappers vs just calling the tool with the required command when one needs to?

Thinking about lua, why not embed the lua modules as all other files (e.g. within constfs, but not restricted to it)

I don't need a file, just a text buffer and a table listing all the buffers. I don't want the code to depend on VFS+consfs, and I don't want module loading to have to go through fs functions and loads of layers when its not needed.

Is there any advantage on writing super-thin wrappers vs just calling the tool with the required command when one needs to?

To clarify my point, in a makefile I prefer to have

generic-script.py -t lua-template.ini -o some-option etc etc etc

rather than

lua-specific-wrapper.sh etc etc etc

And then having to go to lua-specific-wrapper.sh to see what it does.

Hacking my own format is in my opinion not a good idea.

ack

Is there any advantage on writing super-thin wrappers vs just calling the tool with the required command >when one needs to?

Not really, but at some (not purely objective) point, separating should be preferred. Nobody want's almighty tool42.py with an overly complicated parameter handling but I also don't think that's the case here.

How about renaming it xxd-plus

Right direction, but not really self-explaining. Though, I can't really come up with a better name atm.

I don't need a file, just a text buffer and a table listing all the buffers. I don't want the code to depend on >VFS+consfs, and I don't want module loading to have to go through fs functions and loads of layers when its >not needed.

agree

I was looking at #10308. It embeds a SSL certificate in the application, as a C array. I think it could benefit from the template-enabled mkconstfs2.

Can we reactivate this PR? What do I have to do?

I think just proper naming and file location for the script should do.

CI failed because of unrelated error so I re-triggered CI-build

I don't understand this. The old mkconstfs could take a folder and just recursively add all files in that folder and create a constfs with it.

I understand that that's a little unflexible (though it would probably totally alright for our use case to create a tree on the fly in /tmp, just copying every file to it's intended position).

Why not just extend this so mkconstfs can accept a list of files and maybe target positions within the generated constfs? I mean, "[, <target position relative to constfs root]" tuples.

What on earth is the use case for implementing templating in this? "arrays of embedded lua modules", what's that?

What on earth is the use case for implementing templating in this?

citing from the contribution description

reusing the script for other purposes other than doing constfs filesystems

I think it in general it can be argued that the script is not strictly necessary to have in the riot tree, but it makes using constfs much easier and doesn't really hurt.
Because some of the functionality can be used for other use cases - e.g. generating plain arrays for certificate files - it IMO makes sense to factor out this part and also use it there.

I think it in general it can be argued that the script is not strictly necessary to have in the riot tree, but it makes using constfs much easier and doesn't really hurt.

How? I think it is actually important for tools in the RIOT tree to more than "don't really hurt".

Because some of the functionality can be used for other use cases - e.g. generating plain arrays for certificate files - it IMO makes sense to factor out this part and also use it there.

What's wrong with xxd for generating plain arrays from files?

Guys, I'm not picking on this PR, I just skim it and I really don't know what you are using this for other than a super high level description. Usually I do know how and for what tools in RIOT are used, so I'm just asking for a full use-case.

How?

By providing a simple interface to make files available to riot code without having to deal with details on how to format what.

I think it is actually important for tools in the RIOT tree to more than "don't really hurt".

True. For me the reasoning was that there are different situations requiring to embed file contents directly to the code which may be formatted differently -> so factoring out the template (while keeping it simple) sounds valuable.

What's wrong with xxd for generating plain arrays from files?

Nothing, I agree. Actually that's a bad example - I just looked up how the arrays are formatted in the referenced PR and I'd prefer using just xxd for that.

I don't have a strong opinion on this, maybe @jcarrano can elaborate on the lua use-case.

What's wrong with xxd for generating plain arrays from files?

1. One has to install yet-another-tool (that may not even be in your OS' official repos)
2. To change the output one has to sed the result afterwards.
3. It can possibly output partial files (the lazy-sponge can improve that though - see dist/tools: add lazysponge tool #9634)
4. Possible variable name collisions.
5. Only one C-file per file.

The old mkconstfs could take a folder and just recursively add all files in that folder and create a constfs with it.

That is make-unfriendly. By having an explicit list of files rebuilds can be handled better.

"arrays of embedded lua modules"

In the case of Lua, I don't want to have to pull in VFS and constf just to be able to load a script when the user types require("whatever")

Why not just extend this so mkconstfs can accept a list of files and maybe target positions within the generated constfs?

Once I implemented that, and fixed command line argument parsing, and made it so that the output is not constfs specific, and fixed the name mangling (mkconstfs can generate colliding and even invalid names) there would not have been much left of the original file. Because of that and because I don't know who else may be using that script, I made a new one.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you want me to ignore this issue, please mark it with the "State: don't stale" label. Thank you for your contributions.