/kind feature

Describe the solution you'd like
As part of #1763, and #1745, lets us this issue to discuss how to expose the logs for a Trial.

Looking at the docs there are many different types of workers for a Trial https://www.kubeflow.org/docs/components/katib/trial-template/#custom-resource. The K8s clients can allow the backend to fetch logs from a Pod. With this , the main question I have is how can the backend find the Pod for a specific Trial worker type?

• Will there always be an annotation/label that reflects the Trial that the Pod belongs to?
• Is this the case for all worker types?
• What could we show when we have an Argo Workflow?

I suggest that we handle each worker type separately. We can start with K8s Jobs and/or TFJobs and start adding more later on.

@d-gol @andreyvelich @johnugeorge @gaocegege

Love this feature? Give it a 👍 We prioritize the features with the most 👍