OOMkilled #2121
Description
Detailed Description of the Problem
Via a helm deployment of argocd I'm seeing haproxy maxing out cpu & memory:
argoproj/argo-helm#1958
That argocd helm deployment uses this helm chart for haproxy:
https://github.com/DandyDeveloper/charts/tree/master/charts/redis-ha
The result is containers running haproxy. I've tried image 2.6.4, 2.6.5, 2.7.0, and 2.8-dev7.
Expected Behavior
haproxy to work as expected
Steps to Reproduce the Behavior
See argoproj/argo-helm#1958 .
Do you have any idea what may have caused this?
Possibly the issue is the kubernetes cluster is running on top of Redhat 9?
Do you have an idea how to solve the issue?
Is haproxy known to have an issue when running on redhat 9?
Could it maybe be a permission issue, that the container needs to run as privileged or something? ... and when denied the privileged it causes haproxy to max out cpu & memory?
What is your configuration?
defaults REDIS
mode tcp
timeout connect 4s
timeout server 330s
timeout client 330s
timeout check 2s
listen health_check_http_url
bind [::]:8888 v4v6
mode http
monitor-uri /healthz
option dontlognull
# Check Sentinel and whether they are nominated master
backend check_if_redis_is_master_0
mode tcp
option tcp-check
tcp-check connect
tcp-check send PING\r\n
tcp-check expect string +PONG
tcp-check send SENTINEL\ get-master-addr-by-name\ argocd\r\n
tcp-check expect string 10.99.239.156
tcp-check send QUIT\r\n
server R0 argocd-redis-ha-announce-0:26379 check inter 1s
server R1 argocd-redis-ha-announce-1:26379 check inter 1s
server R2 argocd-redis-ha-announce-2:26379 check inter 1s
# Check Sentinel and whether they are nominated master
backend check_if_redis_is_master_1
mode tcp
option tcp-check
tcp-check connect
tcp-check send PING\r\n
tcp-check expect string +PONG
tcp-check send SENTINEL\ get-master-addr-by-name\ argocd\r\n
tcp-check expect string 10.97.61.112
tcp-check send QUIT\r\n
server R0 argocd-redis-ha-announce-0:26379 check inter 1s
server R1 argocd-redis-ha-announce-1:26379 check inter 1s
server R2 argocd-redis-ha-announce-2:26379 check inter 1s
# Check Sentinel and whether they are nominated master
backend check_if_redis_is_master_2
mode tcp
option tcp-check
tcp-check connect
tcp-check send PING\r\n
tcp-check expect string +PONG
tcp-check send SENTINEL\ get-master-addr-by-name\ argocd\r\n
tcp-check expect string 10.96.168.109
tcp-check send QUIT\r\n
server R0 argocd-redis-ha-announce-0:26379 check inter 1s
server R1 argocd-redis-ha-announce-1:26379 check inter 1s
server R2 argocd-redis-ha-announce-2:26379 check inter 1s
# decide redis backend to use
#master
frontend ft_redis_master
bind [::]:6379 v4v6
use_backend bk_redis_master
# Check all redis servers to see if they think they are master
backend bk_redis_master
mode tcp
option tcp-check
tcp-check connect
tcp-check send PING\r\n
tcp-check expect string +PONG
tcp-check send info\ replication\r\n
tcp-check expect string role:master
tcp-check send QUIT\r\n
tcp-check expect string +OK
use-server R0 if { srv_is_up(R0) } { nbsrv(check_if_redis_is_master_0) ge 2 }
server R0 argocd-redis-ha-announce-0:6379 check inter 1s fall 1 rise 1
use-server R1 if { srv_is_up(R1) } { nbsrv(check_if_redis_is_master_1) ge 2 }
server R1 argocd-redis-ha-announce-1:6379 check inter 1s fall 1 rise 1
use-server R2 if { srv_is_up(R2) } { nbsrv(check_if_redis_is_master_2) ge 2 }
server R2 argocd-redis-ha-announce-2:6379 check inter 1s fall 1 rise 1
frontend stats
mode http
bind [::]:9101 v4v6
http-request use-service prometheus-exporter if { path /metrics }
stats enable
stats uri /stats
stats refresh 10sOutput of haproxy -vv
HAProxy version 2.6.4-2a2078c 2022/08/22 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2027.
Known bugs: http://www.haproxy.org/bugs/bugs-2.6.4.html
Running on: Linux 5.14.0-162.23.1.el9_1.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 23 20:08:28 EDT 2023 x86_64
Build options :
TARGET = linux-glibc
CPU = generic
CC = cc
CFLAGS = -O2 -g -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_PROMEX=1
DEBUG = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS
Feature list : +EPOLL -KQUEUE +NETFILTER -PCRE -PCRE_JIT +PCRE2 +PCRE2_JIT +POLL +THREAD +BACKTRACE -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -ENGINE +GETADDRINFO +OPENSSL +LUA +ACCEPT4 -CLOSEFROM -ZLIB +SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER +PRCTL -PROCCTL +THREAD_DUMP -EVPORTS -OT -QUIC +PROMEX -MEMORY_PROFILING
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_THREADS=64, default=4).
Built with OpenSSL version : OpenSSL 1.1.1n 15 Mar 2022
Running on OpenSSL version : OpenSSL 1.1.1n 15 Mar 2022
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.3
Built with the Prometheus exporter as a service
Built with network namespace support.
Support for malloc_trim() is enabled.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.36 2020-12-04
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 10.2.1 20210110
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL_RISK|NO_UPG
fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
<default> : mode=HTTP side=FE|BE mux=H1 flags=HTX
h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG
<default> : mode=TCP side=FE|BE mux=PASS flags=
none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG
Available services : prometheus-exporter
Available filters :
[CACHE] cache
[COMP] compression
[FCGI] fcgi-app
[SPOE] spoe
[TRACE] trace
### Last Outputs and Backtraces
_No response_
### Additional Information
_No response_