Feature Request

What challenge are you facing?

Right now, pipeline.yml files are getting to big and also informations for e.g. one job is really spread over the whole file. Editing a job and finding it is pretty painful, even considering i use the visual-code concourse plugin with the outliner+search.

I miss a grouping of jobs and their resources into a file, which is then combined into e.g. a pipeline file

A Modest Proposal

I would propose, that we allow splitting the pipeline.yml into several files with a static structure for better oversight, understanding and a better editing experience.

A typical folder of that would look like

- mypipeline.pl.yml
- mypipeline/jobs/myjob1.job.yml
- mypipeline/jobs/myjob2.job.yml
- mypipeline/resource/resource1.resource.yml
- mypipeline/resource/resource2.resource.yml

Groups and resource-types should stay in the mypipeline.pl.yml.

The integration should go into fly directly, so when passing a pl.yml to fly, it compile it using the template engine.

Milestone 1: single-file

Every resource and every job must go in exactly one file, it is not allowed to have 2 resources in one *resource.yml file, same for the jobs.

Milestone 2: allow "bundles"

E.g. by using a template_engine: bundle in the mypipeline.pl.yml people can use the combining method. In this case, one can put several jobs and several resources into one file. Most probably the files will get an additional file ending myjob1.bundle.job.yml

Open Questions:

a) Should we rather separate by folder name:mypipeline/jobs/.. or by file suffix: myjob1.job.yml?
I am for the latter, while allowing people to create any subfolders they want - we scan recursively. This also allows for better IDE support

b