Memory Usage Spike when checking if refreshtokens.dex.coreos.com has been created #4019
Description
Preflight Checklist
- I agree to follow the Code of Conduct that this project adheres to.
- I have searched the issue tracker for an issue that matches the one I want to file, without success.
- I am not looking for support or already pursued the available support channels without success.
Version
2.39.1
Storage Type
etcd
Installation Type
Custom Helm chart, Official container image
Expected Behavior
The RAM usage of dex shouldn't increase very much during CRD installation and should preferably stay below 64 MB.
Actual Behavior
The RAM usage of dex increases above 64MB during CRD installation, which causes it to be OOM killed in our current configuration.
Steps To Reproduce
- Set a memory limit of 64MB on the dex pod
- Cause dex to create 13531 refreshtokens
- Restart dex
Additional Information
I'm pretty sure this is caused by dex attempting to list all of the resources when checking if CRDs exist.
Audit logs showed no attempts to create/update the CRD before Dex crashes.
I am using a kustomized version of kubeflow's vendored dex.
A workaround is to increase/remove the memory limit.
Configuration
issuer: ...
storage:
type: kubernetes
config:
inCluster: true
web:
http: 0.0.0.0:5556
logger:
level: "debug"
format: text
oauth2:
skipApprovalScreen: true
passwordConnector: local # for deploying from mlflow
enablePasswordDB: true
staticPasswords:
- email: ...
hash: ...
username: ...
- email: ...
hash: ...
username: ...
- email: ...
hash: ...
username: ...
- email: ...
hash: ...
username: ...
- email: ...
hash: ...
username: ...
staticClients:
# https://github.com/dexidp/dex/pull/1664
- idEnv: OIDC_CLIENT_ID
redirectURIs:
- "/oauth2/callback"
name: "Dex Login Application"
secretEnv: OIDC_CLIENT_SECRET
- idEnv: OAUTH2_PROXY_CLIENT_ID
redirectURIs:
- /oauth2/callback
name: oauth2-proxy
secretEnv: OAUTH2_PROXY_CLIENT_SECRET
- id: kserve-client
name: kserve-client
secret: ...
- idEnv: INFERENCE_CLIENT_ID
name: inference-client
secretEnv: INFERENCE_CLIENT_SECRET
# This is a client that can be used to programmatically authenticate to Dex with a non-private secret
- id: ...
secretEnv: ...
name: "..."
public: true
connectors:
- type: google
id: google
name: Google
config:
# Connector config values starting with a "$" will read from the environment.
clientID: $GOOGLE_CLIENT_ID
clientSecret: $GOOGLE_CLIENT_SECRET
# Dex's issuer URL + "/callback"
redirectURI: ...
# Users must have an email with the following domains
hostedDomains:
- ...
- ...
- ...Logs
time="2025-03-03T17:19:06Z" level=info msg="Dex Version: v2.39.1, Go Version: go1.22.2, Go OS/ARCH: linux amd64"
time="2025-03-03T17:19:06Z" level=info msg="config using log level: debug"
time="2025-03-03T17:19:06Z" level=info msg="config issuer: [...]"
time="2025-03-03T17:19:06Z" level=info msg="kubernetes client apiVersion = dex.coreos.com/v1"
time="2025-03-03T17:19:06Z" level=info msg="creating custom Kubernetes resources"
time="2025-03-03T17:19:06Z" level=info msg="checking if custom resource authcodes.dex.coreos.com has already been created..."
time="2025-03-03T17:19:06Z" level=info msg="The custom resource authcodes.dex.coreos.com already available, skipping create"
time="2025-03-03T17:19:06Z" level=info msg="checking if custom resource authrequests.dex.coreos.com has already been created..."
time="2025-03-03T17:19:06Z" level=info msg="The custom resource authrequests.dex.coreos.com already available, skipping create"
time="2025-03-03T17:19:06Z" level=info msg="checking if custom resource oauth2clients.dex.coreos.com has already been created..."
time="2025-03-03T17:19:06Z" level=info msg="The custom resource oauth2clients.dex.coreos.com already available, skipping create"
time="2025-03-03T17:19:06Z" level=info msg="checking if custom resource signingkeies.dex.coreos.com has already been created..."
time="2025-03-03T17:19:06Z" level=info msg="The custom resource signingkeies.dex.coreos.com already available, skipping create"
time="2025-03-03T17:19:06Z" level=info msg="checking if custom resource refreshtokens.dex.coreos.com has already been created..."
stream closed EOF for auth/dex-6f757b7ffb-xc6zw (dex)