gNMI - TLS handshake failure on Ciena devices #16476
Description
Relevant telegraf.conf
[[inputs.gnmi]]
interval = "5m"
alias = "ciena-gnmi"
addresses = [ "XX.XX.XX.XX:6702" ]
username = "XXXXXXXXXXXXXX"
password = "XXXXXXXXXXXXXX"
encoding = "proto"
redial = "10s"
tls_enable = true
insecure_skip_verify = true
tls_ca = "/etc/telegraf/ciena-ca.cert.pem"
tls_cert = "/etc/telegraf/ciena-client.cert.pem"
tls_key = "/etc/telegraf/ciena-client.key.pem"
name_override = "saos10xgnmi"
updates_only = true
fieldpass = ["path","source","name", "in_crc_error_pkts", "in_discards", "in_errors", "in_octets", "out_errors", "out_octets"]
tagexclude = ["path","name"]
[[inputs.gnmi.subscription]]
name = "ifcounters"
origin = "Ciena"
path = "/oc-if:interfaces/oc-if:interface/oc-if:state/oc-if:counters"
subscription_mode = "sample"
sample_interval = "30s"Logs from Telegraf
Broken starting with the 1.29.2 release (through latest):
---------------------------------------------------------
telegraf | 2025-02-04T14:40:45Z I! Loading config: /etc/telegraf/telegraf.conf
telegraf | 2025-02-04T14:40:45Z I! Loading config: /etc/telegraf/telegraf.d/ciena.conf
telegraf | 2025-02-04T14:40:45Z W! DeprecationWarning: Option "fieldpass" of plugin "inputs.gnmi" deprecated since version 1.29.0 and will be removed in 2.0.0: use 'fieldinclude' instead
telegraf | 2025-02-04T14:40:45Z I! Starting Telegraf 1.29.2 brought to you by InfluxData the makers of InfluxDB
telegraf | 2025-02-04T14:40:45Z I! Available plugins: 241 inputs, 9 aggregators, 30 processors, 24 parsers, 60 outputs, 6 secret-stores
telegraf | 2025-02-04T14:40:45Z I! Loaded inputs: gnmi
telegraf | 2025-02-04T14:40:45Z I! Loaded aggregators:
telegraf | 2025-02-04T14:40:45Z I! Loaded processors: converter rename strings
telegraf | 2025-02-04T14:40:45Z I! Loaded secretstores:
telegraf | 2025-02-04T14:40:45Z W! Outputs are not used in testing mode!
telegraf | 2025-02-04T14:40:45Z I! Tags enabled: host=10.5.200.224
telegraf | 2025-02-04T14:40:45Z D! [agent] Initializing plugins
telegraf | 2025-02-04T14:40:45Z D! [inputs.gnmi::ciena-gnmi] Internal alias mapping: map[oc-if:/interfaces/oc-if:interface/oc-if:state/oc-if:counters:ifcounters]
telegraf | 2025-02-04T14:40:45Z D! [agent] Starting service inputs
telegraf | 2025-02-04T14:40:45Z E! [inputs.gnmi::ciena-gnmi] Error in plugin: failed to setup subscription: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: remote error: tls: handshake failure"
telegraf | 2025-02-04T14:40:55Z E! [inputs.gnmi::ciena-gnmi] Error in plugin: failed to setup subscription: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: remote error: tls: handshake failure"
Gracefully stopping... (press Ctrl+C again to force)
[+] Stopping 1/0
✔ Container telegraf Stopped
Exact same config works fine up to 1.29.1 release:
--------------------------------------------------
telegraf | 2025-02-04T14:41:15Z I! Loading config: /etc/telegraf/telegraf.conf
telegraf | 2025-02-04T14:41:15Z I! Loading config: /etc/telegraf/telegraf.d/ciena.conf
telegraf | 2025-02-04T14:41:15Z W! DeprecationWarning: Option "fieldpass" of plugin "inputs.gnmi" deprecated since version 1.29.0 and will be removed in 2.0.0: use 'fieldinclude' instead
telegraf | 2025-02-04T14:41:15Z I! Starting Telegraf 1.29.1 brought to you by InfluxData the makers of InfluxDB
telegraf | 2025-02-04T14:41:15Z I! Available plugins: 241 inputs, 9 aggregators, 30 processors, 24 parsers, 60 outputs, 6 secret-stores
telegraf | 2025-02-04T14:41:15Z I! Loaded inputs: gnmi
telegraf | 2025-02-04T14:41:15Z I! Loaded aggregators:
telegraf | 2025-02-04T14:41:15Z I! Loaded processors: converter rename strings
telegraf | 2025-02-04T14:41:15Z I! Loaded secretstores:
telegraf | 2025-02-04T14:41:15Z W! Outputs are not used in testing mode!
telegraf | 2025-02-04T14:41:15Z I! Tags enabled: host=10.5.200.224
telegraf | 2025-02-04T14:41:15Z D! [agent] Initializing plugins
telegraf | 2025-02-04T14:41:15Z D! [inputs.gnmi::ciena-gnmi] Internal alias mapping: map[oc-if:/interfaces/oc-if:interface/oc-if:state/oc-if:counters:ifcounters]
telegraf | 2025-02-04T14:41:15Z D! [agent] Starting service inputs
telegraf | 2025-02-04T14:41:15Z D! [inputs.gnmi::ciena-gnmi] Connection to gNMI device 10.255.32.14:6702 established
telegraf | > interface,agent_host=10.255.32.14,host=10.5.200.224,ifIndex=1 ifHCInOctets=94533575503546i,ifHCOutOctets=873939187818389i,ifInCrcErrors=0i,ifInDiscards=236i,ifInErrors=0i,ifOutErrors=0i 1738680087844000000
telegraf | > interface,agent_host=10.255.32.14,host=10.5.200.224,ifIndex=2 ifHCInOctets=94569510275635i,ifHCOutOctets=874902982673682i,ifInCrcErrors=0i,ifInDiscards=0i,ifInErrors=0i,ifOutErrors=0i 1738680087844000000
. . . (output trimmed fo clarity)
Gracefully stopping... (press Ctrl+C again to force)
[+] Stopping 1/0
✔ Container telegraf Stopped
System info
Telegraf 1.29.2+, Docker 4.37.2, MacOS 15.2 (m4-Max)
Docker
services:
telegraf:
image: telegraf:1.29.2-alpine
container_name: telegraf
restart: no
command: telegraf --debug --test-wait 45
volumes:
- /etc/snmp:/etc/snmp:ro
- ./mibs:/usr/share/snmp/mibs:rw
- ./telegraf/etc:/etc/telegraf:rw
ports:
- '8125:8125'
logging:
options:
max-size: "1m"
max-file: "5"
Steps to reproduce
- Start with Telegraf 1.29.2
- Subscribe to a Ciena device
- TLS Authentication fails (certs are valid / vendor supplied, don't expire until 2050)
- Revert to Telegraf 1.29.1, connection works fine
Expected behavior
TLS handshake is expected to still work with known good config from 1.29.1 to subsequent versions
Actual behavior
TLS handshake breaks starting in 1.29.2
Additional info
Confirmed to be working with my Cisco IOS-XR devices, so this problem is unique to Ciena.