Is your feature request related to a problem? Please describe.
I run cert-manager with runAsNonRoot: true and readOnlyRootFilesystem: true in my security context for all of the cert-manager workloads. For the most part, this works fine. However, one of my deployments is particularly overloaded, and I'm seeing the webhook crash because it apparently attempts to log some sort of message which apparently involves creating a file of some sort under /tmp.

I1108 09:36:55.243809       1 request.go:601] Waited for 1.190057546s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/apps.open-cluster-management.io/v1?timeout=32s

log: exiting because of error: log: cannot create log: open /tmp/ctl.cert-manager-startupapicheck-542zc.1000.log.INFO.20221108-093655.1: read-only file system

I have no idea what it's trying to log. I also have no idea why it needs to log it to a file in /tmp instead of to stdout/stderr. (I also have no real interest in INFO level log messages.)

We've mitigated this temporarily by running the webhook in an insecure fashion.

Describe the solution you'd like

If we were able to mount an emptyDir to back the /tmp directory, then the application could still be run securely and the logger could continue doing whatever-it-is in the /tmp directory.

This need not be exposed in any way for configuration via the values.yaml and can just be added to the templates themselves something like this:

volumes:
  - name: tmp
    {{- if semverCompare ">=1.21.0-0" $.Capabilities.KubeVersion.GitVersion }}
    emptyDir:
      medium: Memory
    {{- else }}
    emptyDir: {}
    {{- end }}

volumeMounts:
  - name: tmp
    mountPath: /tmp

Describe alternatives you've considered

The temporary workaround we've identified is to require cert-manager to be deployed insecurely. This can be done by requiring one of either of these configurations:

• readOnlyRootFilesystem: false
• runAsNonRoot: false

Unfortunately my organization does not allow us to run insecurely in production; I'm sure other organizations have similar policies.

Other solutions which we are not able to test would be:

• logger should not write to file
• allow configuration of minimum log level

Additional context

I've only observed this with the webhook, but possibly it's an issue with the other workloads as well.

This is my webhook security context configuration:

  webhook:
    containerSecurityContext:
      capabilities:
        drop: [ALL]
      runAsUser: 10001
      runAsGroup: 20001
      runAsNonRoot: true
      readOnlyRootFilesystem: true
      allowPrivilegeEscalation: false
      seccompProfile:
        type: RuntimeDefault

This basically reflects the minimum required security context required by my organization.

Environment details (remove if not applicable):

• Kubernetes version: N/A
• Cloud-provider/provisioner: N/A
• cert-manager version: 1.9.1
• Install method: helm

(k8s version and cloud-provider/provisioner are likely not relevant; but I observed this on a very overloaded Openshift cluster running k8s 1.23.5. I also see no indication that the behavior is any different on 1.10.0)

/kind feature