MQTT Connector/Bridge looses connection and does not reconnect #15538
Description
What happened?
I've set up a MQTT connector to an external MQTT server and republish some topics locally. Periodically, emqx looses the connection to the external MQTT broker and does not reconnect and it is even showing up in the dashboard as "connected".
Once you disable and re-enable the connector (or update some config), everything works again.
What did you expect to happen?
EMQX does reconnect to the external mqtt broker
How can we reproduce it (as minimally and precisely as possible)?
create a EMQX docker and create a connector. If this connection looses connection, it may not reconnect correctly (see my attached logs)
Anything else we need to know?
This happens with SSL more often (up to multiple times a day), without SSL far less (maybe every 1-2 weeks)
(Anyways, thanks for the great software!)
EMQX version
Note: I'm using the docker image
5.10.0 (Enterprise)
$ ./bin/emqx_ctl broker
Node 'emqx@emqx.prox' not responding to pings.OS version
# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
$ uname -a
Linux redacted-hostname 5.19.17-1-pve #1 SMP PREEMPT_DYNAMIC PVE 5.19.17-1 (Mon, 14 Nov 2022 20:25:12 x86_64 GNU/LinuxLog files
using ssl
2025-06-26T05:21:44.398256+00:00 [info] msg: socket_closed_when_connected
2025-06-26T05:21:51.570260+00:00 [info] msg: socket_closed_before_connack
2025-06-26T05:21:51.570364+00:00 [warning] msg: ingress_client_connect_failed, name: <<"connector:mqtt:TN">>, reason: {ssl_closed,{sslsocket,{gen_tcp,#Port<0.31349>,tls_connection,undefined},[<0.15675566.0>,<0.15675565.0>]}}
2025-06-26T05:21:51.570690+00:00 [debug] ecpool_worker received unexpected exit:{shutdown,ssl_closed} from <0.15675562.0>. Supervisees: []
using no ssl
2025-07-21T16:06:30.999101+00:00 [debug] msg: ingress_client_starting, options: [{ecpool_worker_id,1},{client_opts,#{keepalive => 15,ssl => false,hosts => [{"HOST.DOMAIN.TLD",1883}],connect_timeout => 20,password => <<"******">>,username => <<"REDACTED">>,ssl_opts => [{depth,10},{hibernate_after,5000},{verify,verify_none},{ciphers,[]},{log_level,notice},{versions,['tlsv1.3','tlsv1.2','tlsv1.1',tlsv1]},{middlebox_comp_mode,true},{secure_renegotiate,true},{reuse_sessions,true}],retry_interval => 15000,clientid => {<<"clid_">>,<<"TTN9b4e7d58">>},proto_ver => v4,clean_start => true,max_inflight => 32,bridge_mode => false,force_ping => true,topic_to_handler_index => #Ref<0.2043034392.753532931.81109>}},{available_clientids,[<<"wsclid_TTN9b4e7d58:1">>]},{name,<<"connector:mqtt:TN">>}]
2025-07-21T16:06:31.064448+00:00 [debug] msg: send_data, packet: CONNECT(Q0, R0, D0, ClientId=clid_TTN9b4e7d58:1, ProtoName=MQTT, ProtoVsn=4, CleanStart=true, KeepAlive=15, Username=REDACTED, Password=******), socket: #Port<0.392115>, payload_encode: text
sadly, this is at debug level and trace logs will be really hard to get due to the massiv size and random occurence, but if you need them, I can try it.