In order to implement outputs in Duffle, they need to be produced by cnab-go. The tricky part of this is to get the output files from the drivers. In the Kubernetes driver, the Invocation Image is run as a Job. We've come up with a few options for how the driver might get the output files from /cnab/app/outputs in the container back to the (k8s client) driver code.

The simplest option would be to use a persistent volume. The driver could mount the volume at /cnab/app/outputs in the invocation image and then retrieve the contents after the image has run. However, we don't think the driver should assume or require that persistent volumes are available on all clusters.

The rest of these options assume that an emptyDir volume will be mounted at /cnab/app/outputs in the invocation container and that there will be a second "sidecar" container in the same pod that runs the invocation image to do some extra work in retrieving the output files. The containers need to be in the same pod so that they can share the emptyDir volume.

The sidecar container is given a service account token with permission to write secrets. It uses this token to write a secret containing the paths and contents of all files written by the invocation image (when that container finishes running).
Challenges: This requires setting up a service account with the desired permissions; it's not clear that the Duffle driver expects to have those permissions already or that it would be better to ask users to setup this service account in advance. The sidecar container will also need to figure out when the other container in the pod has finished, so that it can write the final state of the files to the secret.
The sidecar container could be used to just keep the pod running, so that the driver could use the API equivalent of kubectl exec to grab the output files from the volume.
Challenges: The invocation image is run in a Job; the driver waits for the Job to finish. For the Job to finish, the Pod needs to exit, which means the containers need to exit. This sidecar would prevent that -- meaning a significant restructuring of how the driver waits for invocation image completion would be required.
The sidecar container could use its log to output the files from the output volume. This would require choosing some format that can be written to the stdout stream yet is easily parsed into a number of arbitrary binary files.
Challenges: Picking a good format, complicates log stream logic in driver, seems ugly, could hit log size limit edge cases.
The sidecar container is a file server. The driver requests each expected output file over https from the sidecar when the invocation image finishes.
Challenges: The sidecar does not exit when the invocation image does, which ruins the waiting for the Job to exit logic. There is no guarantee that the driver can connect to the pod directly (it appears to only talk to the API server so far), so connecting and/or securing the connection could be an issue.

All of these approaches have drawbacks, and there are likely some (of both) that we have not thought of.

cc @youreddy to complete the we

Implement output handling #747

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions