/kind feature

Describe the solution you'd like
[A clear and concise description of what you want to happen.]

I'd like to run katib-experiment with istio-sidecar injection.
I guess the reason why katib induce user to run katib-experiment with istio-sidecar injection "false" was something like this:

• katib cannot work in namespace with istio-injection enabled #955
• Support startup dependencies between containers on the same Pod kubernetes/kubernetes#65502
• Better support for sidecar containers in batch jobs istio/istio#6324
• https://medium.com/redbox-techblog/handling-istio-sidecars-in-kubernetes-jobs-c392661c4af7

Thus, the following problem is maybe the main issue.
A) katib's metrics-logger-and-collector sidecar container should know when the istio-proxy sidecar container started
B) metrics-logger-and-collector sidecar container should kill when the main container finished well or not.

By the way I think changing initial-logic and end-logic in metrics-collector image, katib could handle this.

Maybe something like this;

1. Handling issue B above from metrics-collector (

   katib/cmd/metricscollector/v1beta1/file-metricscollector/main.go

   Line 339 in eea6ada

   func main() {

   )

func killIstioProxySidecar() {
	klog.Infof("Let's kill istio-proxy sidecar")

	// curl -fsI -X POST http://localhost:15020/quitquitquit
	resp, err := http.Post("http://localhost:15020/quitquitquit", "text/plain", bytes.NewBufferString(""))
	if err != nil {
		// TODO 
		klog.Infof("there is no istio-proxy sidecar")
	}

	defer resp.Body.Close()
	klog.Infof("istio-proxy sidecar killed")
}

func main() {
	defer killIstioProxySidecar()

	flag.Var(&stopRules, "stop-rule", "The list of early stopping stop rules")
	flag.Parse()
	klog.Infof("Trial Name: %s", *trialName)
.....

2. Also, pass the process from istio-proxy sidecar nearby here

		// Ignore istio-sidecar processes
                // TODO this must be more sophisticated to catch istio-proxy's process
		if strings.Contains(cmdline, "envoy") || strings.Contains(cmdline, "sidecar") {
			fmt.Printf("Ignore istio-sidecar container's processes")
			continue
		}

3. Handling issue A above.
   I have no idea how to implement this now, but the primary container's command should be changed automatically by katib experiment/trial's controller as follows:

• Before

- command:
  - <YOUR_COMMAND>;

• After

- command:
  - /bin/sh
  - -c
  - |
    until curl -fsI http://localhost:15021/healthz/ready; do echo \"Waiting for Sidecar...\"; sleep 3; done;
    echo \"Sidecar available. Running the command...\";
    <YOUR_COMMAND>

• I guess there is the way since katib-controller already automatically adds echo completed > /var/log/katib/$$$$.pid at the end of primary-container's command.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

• I'm afraid this feature could be generalized except "Job" kind.
  • But for "Job", this might be useful who wants to work with istio-sidecar