Describe the issue
While investigating potential options how to support partial pipeline execution, I've noticed that nodes that are implemented using generic components are not cached by Kubeflow Pipelines. This is an unintentional side effect of a unique environment variable value that we are passing to the container. In a nutshell this variable contains a unique run id, which is a constant value across all nodes in the same pipeline.

Caching can have significant benefits with respect to resource usage and performance. Imagine, if you will, a notebook that downloads a data set archive, extracts it, and performs some processing. If the archive content doesn't change at the source, downloading and processing it again is completely unnecessary if the produced outputs are identical during multiple runs.

I've created this issue to discus how we should deal with caching in general:

• Should the problematic variable be removed or somehow be encapsulated to hide its unique value from KFP? If we did, caching would work as expected assuming no other constraints are violated that would result in a cache miss.
• Should we provide users the ability to disable caching for any generic or custom node, if this is supported by the runtime environment? In certain scenarios forcing a re-run of a component might be desirable. Going back to the earlier download/extract example, if the archive content had changed since it was first downloaded and nothing else has changed, KFP would not re-run the node. As a result any downstream node would operate on the old output, which doesn't reflect the current state of the data set archive.