Skip to content

New resource interface (+ add versioning) #534

@vito

Description

@vito

Proposed changes

  1. Rename in and out scripts to get and put. These are confusingly named something different because a long time ago jobs just had _in_puts and _out_puts.
  2. Run ./check, ./get, and ./put, and ./info (see next point) rather than /opt/resource/X. Having them in /opt is a bit Linux-specific and assumes containers have their own chroot, which is not the case on Darwin or Windows.
  3. Add a info script which prints the resource's API version, e.g. {"version":"2.0"}. This will start at 2.0. If /info does not exist we'll execute today's resource interface behavior.
  4. Change get to not emit the version or metadata. This was from when in didn't know what it was fetching until it was done. It should now be an error if it can't fetch the requested version. We'll also move all metadata collection to check (see point 8).
  5. Add a delete action, which looks like put (can be given files and params to determine what to delete) but returns the set of versions that it deleted. This is to support bulk deletes, e.g. to garbage collect intermediate artifacts after a final build is shipped.
  6. Remove the source and destination arguments passed to get and put - just make it the working directory.
  7. Give check an official scratch space, which is the current working directory. No real semantics change here, just consistency with get and put, and potential for future behavior like reusing the scratch space but not reusing check containers for extended periods. Note: we've introduced /scratch recently so this change may just mean making that the work-dir.
  8. Move all metadata emitting to check, so that it's always present. The original thought was that metadata collection may be expensive, but so far we haven't seen that to be the case.
  9. Change put to emit an array of versions, rather than just one, and without metadata. Technically the git resource may push many commits, so this is necessary to track them all as outputs of a build. This could also support batch creation. Notably, the versions emitted by put are ordered chronologically, so that the dependent get knows to fetch the latest version. We would save them internally, along with an association with the build, and rely on check to determine the final ordering and metadata, being careful to not leave gaps in the version history (i.e. other commits pushed slightly before the put registered the build's).
  10. Change put to write to a specified file, rather than stdout, so that we don't have to be attached to process its response. This is one of the few ways a build can error after the ATC reattaches (unexpected end of JSON). With it written to a file, we can just try to read the file when we re-attach after seeing that the process exited. This also frees up stdout/stderr for normal logging, which has been an occasional pitfall during resource development/debugging.
  11. Remove the distinction between source and params; resources will receive a single config. The distinction will remain in the pipeline. This makes it easier to implement a resource without planning ahead for interesting dynamic vs. static usage patterns, and will get more powerful with Dynamic build plan generation #684.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions