Feature Request

What challenge are you facing?

There is a common use case where something needs to be "published", "finalized", "tagged" or similar, after several jobs have run and proven that a change didn't break anything.

Similar to the Flightschool, I'm using the following example to build, deploy, integration-test, and publish my-product to demonstrate the problem:

jobs:
- name: build-rc
  plan:
  - get: my-product-develop
    trigger: true
  - put: my-product-rc

- name: deploy-rc
  plan:
  - get: my-product-rc
    trigger: true
  - get: ci
  - put: deployment

- name: run-integration-tests
  plan:
  - get: my-product-rc
    passed: [ deploy-rc ]
    trigger: true

- name: publish-release
  plan:
  - get: my-product-rc
    passed: [ run-integration-tests ]
    trigger: true
  - get: my-product-develop
    passed: [run-integration-tests]
  - put: my-product-master

This is example does not work, because the lines

  - get: my-product-develop
    passed: [run-integration-tests]

require the my-product-develop resource to be consumed by run-integration-tests. However, run-integration-tests doesn't do anything with my-product-develop. The only reason to consume it there is to satisfy the publish-release job. It's worse, because after adding it to run-integration-tests, it also needs to be added to deploy-rc. In real pipelines, the problem is often worse, because the number of jobs consuming resources to satisfy later jobs can easily go up to 5 or more.

All of this can of course be solved by adding the necessary resources to the jobs. While this is a bit annoying, it seems to me there are 2 deeper issues:

1. It is violating the principle that a job only defines its dependencies (= in-resources) and its outputs (= out-resources), and Concourse takes care of plugging that together to a pipeline. Now, instead, job definitions need to take into account all properties of other jobs in the pipeline, even jobs that come at a later stage. In the example above e.g. the deploy-rc job knows there is a later job using my-product-develop.
2. Looking at a job (in the source) it's now not obvious anymore why it consumes certain resources and one cannot rely that it's somehow involved to create the product for the out-resources.

A Modest Proposal

In the same way that I manually add these resources to the jobs in a mechanical way, maybe Concourse could infer that automatically. To help it with that, I could add all the jobs in question to the passed block like so:

- name: publish-release
  plan:
  - get: my-product-rc
    passed: [ run-integration-tests ]
    trigger: true
  - get: my-product-develop
    passed:
    - build-rc
    - deploy-rc
    - run-integration-tests
  - put: my-product-master

That information should be enough for an algorithm to add the resources automatically. Of course this requires that the first job in this "sub-pipeline" (here build-rc) must already consume the resource explicitly. If that is not the case fly would return an error.