bgpd from version 8.4 to 9.1 (current stable) memory leak when using default-originate route map (set large-community / set community) #14828
Description
bgpd# show version
FRRouting 9.0.1 (gazgw) on Linux(6.1.0-12-amd64).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
configured with:
'--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-option-checking' '--disable-silent-rules' '--libdir=${prefix}/lib/x86_64-linux-gnu' '--libexecdir=${prefix}/lib/x86_64-linux-gnu' '--disable-maintainer-mode' '--localstatedir=/var/run/frr' '--sbindir=/usr/lib/frr' '--sysconfdir=/etc/frr' '--with-vtysh-pager=/usr/bin/pager' '--libdir=/usr/lib/x86_64-linux-gnu/frr' '--with-moduledir=/usr/lib/x86_64-linux-gnu/frr/modules' '--disable-dependency-tracking' '--enable-rpki' '--disable-scripting' '--enable-pim6d' '--with-libpam' '--enable-doc' '--enable-doc-html' '--enable-snmp' '--enable-fpm' '--disable-protobuf' '--disable-zeromq' '--enable-ospfapi' '--enable-bgp-vnc' '--enable-multipath=256' '--enable-user=frr' '--enable-group=frr' '--enable-vty-group=frrvty' '--enable-configfile-mask=0640' '--enable-logfile-mask=0640' 'build_alias=x86_64-linux-gnu' 'PYTHON=python3'
system top reports:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15022 frr 20 0 6820200 6.4g 5640 S 0.3 20.4 1:31.66 bgpd
1668 frr 20 0 636332 186880 2040 S 0.0 0.6 59:08.28 zebra
bgpd -> show memory
System allocator statistics: Total heap allocated: > 2GB Holding block headers: 10252 KiB Used small blocks: 0 bytes Used ordinary blocks: > 2GB Free small blocks: 3712 bytes Free ordinary blocks: 5282 KiB Ordinary blocks: 1404 Small blocks: 74 Holding blocks: 3 (see system documentation for 'mallinfo' for meaning) --- qmem libfrr --- Type : Current# Size Total Max# MaxBytes Buffer : 6 24 144 6 144 Buffer data : 1 variable 4120 8 32960 Host config : 8 variable 304 8 304 Completion item : 0 variable 0 1 24 Command Tokens : 12944 72 932672 12966 934272 Command Token Text : 9273 variable 323864 9294 330096 Command Token Help : 9273 variable 223544 9294 224048 Command Argument : 2 variable 48 24 576 Command Argument Name : 2248 variable 53984 2269 54488 Command Match Stack : 0 variable 0 125 3000 Lexer token (temporary) : 0 variable 0 2 64 Access List : 1 56 56 1 56 Access List Str : 1 14 24 1 24 Access Filter : 2 112 240 2 240 RCU thread : 2 128 272 2 272 FRR POSIX Thread : 4 variable 320 4 320 POSIX sync primitives : 4 variable 192 4 192 Graph : 41 8 984 42 1008 Graph Node : 15224 32 611040 15226 611120 Hash : 703 variable 34376 703 34376 Hash Bucket : 355646 32 14231408 360167 14412216 Hash Index : 352 variable 5645048 353 7746280 Interface : 14 272 3920 14 3920 Connected : 25 48 1416 25 1416 Link List : 159 40 6408 163 6576 Link Node : 680 24 16512 807 19608 Temporary memory : 217 variable 11656 229 12168 Bitfield memory : 2 variable 10256 2 10256 Nexthop : 237 152 36200 237 36200 Northbound Node : 258 1192 307536 258 307536 Northbound Configuration : 2 24 48 2 48 Northbound Configuration Entry: 372 1032 384144 372 384144 Prefix List : 10 88 880 10 880 Prefix List Str : 10 variable 240 10 240 Prefix List Entry : 44 136 6048 44 6048 Prefix List Trie Table : 28 4096 114912 28 114912 Prefix : 25 56 1480 25 1480 Privilege information : 3 variable 136 3 136 Ring buffer : 62 variable 3048992 62 3048992 Route map : 22 120 2640 22 2640 Route map name : 171 variable 4104 172 4128 Route map index : 92 152 14464 92 14464 Route map rule : 201 40 8088 201 8088 Route map rule str : 192 variable 4944 192 4944 Route map compiled : 267 variable 6408 267 6408 Route map dependency : 42 24 1008 42 1008 Route map dependency data : 75 16 1800 75 1800 Skip List : 2 56 112 2 112 Skip Node : 2 160 336 2 336 Skiplist Counters : 2 68 144 2 144 Socket union : 44 112 5280 44 5280 Stream : 49 variable 2602472 33800 7476528 Stream FIFO : 62 64 4464 62 4464 Route table : 144 56 8128 144 8128 Route node : 56 120 6880 56 6880 Thread : 81 160 13640 110 18544 Thread master : 12 variable 50352 12 50352 Thread Poll Info : 6 8192 49200 6 49200 Thread stats : 29 96 3032 29 3032 Typed-hash bucket : 53 variable 6304328 53 6304328 Typed-heap array : 1 576 584 1 584 Vector : 30532 24 733952 30537 734072 Vector index : 30532 variable 963376 30537 963880 VRF : 1 216 216 1 216 VRF bit-map : 3 8 72 3 72 VTY : 5 variable 61256 5 61256 VTY server : 2 32 96 2 96 VTY output buffer : 0 1038 0 1 1048 VTY history : 9 variable 216 9 216 Work queue : 3 144 456 3 456 Work queue item : 0 24 0 7 168 Work queue name string : 3 variable 72 3 72 YANG module : 5 48 296 5 296 YANG data structure : 0 1032 0 1 1048 Zclient : 2 3144 6288 2 6288 Redistribution instance IDs : 6 2 144 6 144 log thread-local buffer : 2 24608 49232 2 49232 --- qmem logging subsystem --- Type : Current# Size Total Max# MaxBytes log file target : 1 88 88 1 88 log file name : 1 22 24 1 24 syslog target : 1 56 56 1 56 --- qmem bgpd --- Type : Current# Size Total Max# MaxBytes Peer KeepAlive Timer : 22 24 528 22 528 BGP Peer pthread Conditional : 1 48 56 1 56 BGP Peer pthread Mutex : 1 40 40 1 40 Mac Hash Entry : 7 16 168 7 168 Mac Hash Entry Intf String : 14 variable 336 15 360 BGP instance : 25 variable 10760 25 10760 BGP listen socket details : 2 144 304 2 304 BGP peer : 53 20864 1106216 53 1106216 BGP peer hostname : 70 variable 1920 71 1960 Peer group : 4 64 288 4 288 BGP Peer group hostname : 4 9 96 4 96 Peer description : 26 variable 624 26 624 BGP peer af : 26 80 2288 26 2288 BGP update group : 22 104 2288 22 2288 BGP update subgroup : 22 240 5456 22 5456 BGP packet : 22 56 1232 32 1792 BGP attribute : 273809 320 89809800 273825 89815048 BGP aspath : 78026 40 3121184 78068 3122864 BGP aspath seg : 78057 24 1874040 78100 1875296 BGP aspath segment data : 78057 variable 2014600 78100 2015104 BGP aspath str : 78027 variable 5007672 78070 5010080 BGP table : 87 56 4872 87 4872 BGP node : 607055 192 121431832 607169 121454504 BGP route : 558869 136 76066488 558971 76080232 BGP ancillary route info : 1 432 440 1 440 BGP connected : 16 4 384 16 384 BGP static : 7 152 1064 7 1064 BGP adv attr : 0 24 0 12594 302368 BGP adv : 0 64 0 56940 4099760 BGP synchronise : 22 48 1232 22 1232 BGP adj in : 572533 48 34468296 572635 34478776 BGP adj out : 514729 96 53844360 514849 53857560 BGP multipath info : 679 48 38024 679 38024 BGP AS list : 1 40 40 1 40 BGP AS filter : 2 48 112 2 112 BGP AS filter str : 2 variable 48 2 48 community : 532 40 21312 534 21408 community val : 532 variable 18624 534 18672 community str : 520 variable 4264000 521 4272216 extcommunity : 53 40 2136 54 2192 extcommunity val : 53 variable 1352 54 1392 extcommunity str : 53 variable 7720 53 7720 community-list : 35 56 1976 35 1976 community-list name : 35 variable 840 35 840 community-list entry : 35 56 2120 35 2120 community-list config : 32 variable 768 32 768 community-list handler : 1 120 136 1 136 BGP Process queue : 0 32 0 7 280 BGP transit attr : 5 24 120 5 120 BGP transit val : 5 25 200 5 200 BGP nexthop : 237 184 43608 237 43608 BGP regexp : 34 64 2448 34 2448 BGP own address : 16 64 1152 16 1152 BGP Filter Information : 118 variable 2896 120 2944 Large Community : 79424644 40 3177027200 79424645 3177027240 Large Community display string: 149 variable 9128 149 9128 Large Community value : 79424644 variable 1906212544 79424645 1906212568 BGP EVPN MH Information : 1 56 72 1 72 Software Version : 2 16 48 2 48 BGP Martian Addr Intf String : 16 variable 384 16 384 BGP PBR Context : 1 32 40 1 40 BGP interface context : 14 4 336 14 336 BGP EVPN instance information : 1 56 56 1 56 --- qmem rfapi --- Type : Current# Size Total Max# MaxBytes NVE Configuration : 1 2984 2984 1 2984 RFAPI Generic : 1 296 296 1 296 RFAPI Import Table : 1 208 216 1 216
~ 10 minutes later
system top reports:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15022 frr 20 0 11.2g 11.0g 5640 S 0.3 35.4 2:29.07 bgpd
1668 frr 20 0 636332 186880 2040 S 0.0 0.6 59:09.59 zebra
show memory
System allocator statistics: Total heap allocated: > 2GB Holding block headers: 10252 KiB Used small blocks: 0 bytes Used ordinary blocks: > 2GB Free small blocks: 4544 bytes Free ordinary blocks: 5370 KiB Ordinary blocks: 1401 Small blocks: 81 Holding blocks: 3 (see system documentation for 'mallinfo' for meaning) --- qmem libfrr --- Type : Current# Size Total Max# MaxBytes Buffer : 6 24 160 6 160 Buffer data : 1 variable 4120 8 32960 Host config : 8 variable 304 8 304 Completion item : 0 variable 0 1 24 Command Tokens : 12944 72 932672 12966 934272 Command Token Text : 9273 variable 323864 9294 330096 Command Token Help : 9273 variable 223544 9294 224048 Command Argument : 2 variable 48 24 576 Command Argument Name : 2248 variable 53984 2269 54488 Command Match Stack : 0 variable 0 125 3000 Lexer token (temporary) : 0 variable 0 2 64 Access List : 1 56 56 1 56 Access List Str : 1 14 24 1 24 Access Filter : 2 112 240 2 240 RCU thread : 2 128 272 2 272 FRR POSIX Thread : 4 variable 320 4 320 POSIX sync primitives : 4 variable 192 4 192 Graph : 41 8 984 42 1008 Graph Node : 15224 32 611040 15226 611120 Hash : 703 variable 34376 703 34376 Hash Bucket : 355834 32 14239008 360167 14412216 Hash Index : 352 variable 5645048 353 7746280 Interface : 14 272 3920 14 3920 Connected : 25 48 1416 25 1416 Link List : 159 40 6408 163 6576 Link Node : 680 24 16512 807 19608 Temporary memory : 217 variable 11656 229 12168 Bitfield memory : 2 variable 10256 2 10256 Nexthop : 237 152 36200 237 36200 Northbound Node : 258 1192 307536 258 307536 Northbound Configuration : 2 24 48 2 48 Northbound Configuration Entry: 372 1032 384144 372 384144 Prefix List : 10 88 880 10 880 Prefix List Str : 10 variable 240 10 240 Prefix List Entry : 44 136 6048 44 6048 Prefix List Trie Table : 28 4096 114912 28 114912 Prefix : 25 56 1480 25 1480 Privilege information : 3 variable 136 3 136 Ring buffer : 62 variable 3048992 62 3048992 Route map : 22 120 2640 22 2640 Route map name : 171 variable 4104 172 4128 Route map index : 92 152 14464 92 14464 Route map rule : 201 40 8088 201 8088 Route map rule str : 192 variable 4944 192 4944 Route map compiled : 267 variable 6408 267 6408 Route map dependency : 42 24 1008 42 1008 Route map dependency data : 75 16 1800 75 1800 Skip List : 2 56 112 2 112 Skip Node : 2 160 336 2 336 Skiplist Counters : 2 68 144 2 144 Socket union : 44 112 5280 44 5280 Stream : 49 variable 2602472 33800 7476528 Stream FIFO : 62 64 4464 62 4464 Route table : 144 56 8128 144 8128 Route node : 56 120 6880 56 6880 Thread : 79 160 13288 110 18544 Thread master : 12 variable 50352 12 50352 Thread Poll Info : 6 8192 49200 6 49200 Thread stats : 29 96 3032 29 3032 Typed-hash bucket : 53 variable 6304328 53 6304328 Typed-heap array : 1 576 584 1 584 Vector : 30532 24 733952 30537 734072 Vector index : 30532 variable 963376 30537 963880 VRF : 1 216 216 1 216 VRF bit-map : 3 8 72 3 72 VTY : 5 variable 61256 5 61256 VTY server : 2 32 96 2 96 VTY output buffer : 0 1038 0 1 1048 VTY history : 1 variable 24 10 240 Work queue : 3 144 456 3 456 Work queue item : 0 24 0 7 168 Work queue name string : 3 variable 72 3 72 YANG module : 5 48 296 5 296 YANG data structure : 0 1032 0 1 1048 Zclient : 2 3144 6288 2 6288 Redistribution instance IDs : 6 2 144 6 144 log thread-local buffer : 2 24608 49232 2 49232 --- qmem logging subsystem --- Type : Current# Size Total Max# MaxBytes log file target : 1 88 88 1 88 log file name : 1 22 24 1 24 syslog target : 1 56 56 1 56 --- qmem bgpd --- Type : Current# Size Total Max# MaxBytes Peer KeepAlive Timer : 22 24 528 22 528 BGP Peer pthread Conditional : 1 48 56 1 56 BGP Peer pthread Mutex : 1 40 40 1 40 Mac Hash Entry : 7 16 168 7 168 Mac Hash Entry Intf String : 14 variable 336 15 360 BGP instance : 25 variable 10760 25 10760 BGP listen socket details : 2 144 304 2 304 BGP peer : 53 20864 1106216 53 1106216 BGP peer hostname : 70 variable 1920 71 1960 Peer group : 4 64 288 4 288 BGP Peer group hostname : 4 9 96 4 96 Peer description : 26 variable 624 26 624 BGP peer af : 26 80 2288 26 2288 BGP update group : 22 104 2288 22 2288 BGP update subgroup : 22 240 5456 22 5456 BGP packet : 22 56 1232 32 1792 BGP attribute : 273963 320 89860312 274031 89882616 BGP aspath : 78060 40 3122544 78090 3123744 BGP aspath seg : 78091 24 1874856 78121 1875576 BGP aspath segment data : 78091 variable 2015480 78121 2016552 BGP aspath str : 78061 variable 5009240 78091 5012424 BGP table : 87 56 4872 87 4872 BGP node : 607105 192 121441880 607185 121457896 BGP route : 558907 136 76071688 558986 76082432 BGP ancillary route info : 1 432 440 1 440 BGP connected : 16 4 384 16 384 BGP static : 7 152 1064 7 1064 BGP adv attr : 0 24 0 12594 302368 BGP adv : 0 64 0 56940 4099760 BGP synchronise : 22 48 1232 22 1232 BGP adj in : 572571 48 34469784 572650 34478776 BGP adj out : 514723 96 53843848 514921 53864328 BGP multipath info : 679 48 38024 679 38024 BGP AS list : 1 40 40 1 40 BGP AS filter : 2 48 112 2 112 BGP AS filter str : 2 variable 48 2 48 community : 532 40 21312 534 21408 community val : 532 variable 18624 534 18672 community str : 520 variable 4264000 521 4272216 extcommunity : 53 40 2136 54 2192 extcommunity val : 53 variable 1352 54 1392 extcommunity str : 53 variable 7720 53 7720 community-list : 35 56 1976 35 1976 community-list name : 35 variable 840 35 840 community-list entry : 35 56 2120 35 2120 community-list config : 32 variable 768 32 768 community-list handler : 1 120 136 1 136 BGP Process queue : 0 32 0 7 280 BGP transit attr : 5 24 120 5 120 BGP transit val : 5 25 200 5 200 BGP nexthop : 237 184 43608 237 43608 BGP regexp : 34 64 2448 34 2448 BGP own address : 16 64 1152 16 1152 BGP Filter Information : 118 variable 2896 120 2944 Large Community : 143792317 40 5751753192 143792318 5751753232 Large Community display string: 149 variable 9128 149 9128 Large Community value : 143792317 variable 3451044872 143792318 3451044896 BGP EVPN MH Information : 1 56 72 1 72 Software Version : 2 16 48 2 48 BGP Martian Addr Intf String : 16 variable 384 16 384 BGP PBR Context : 1 32 40 1 40 BGP interface context : 14 4 336 14 336 BGP EVPN instance information : 1 56 56 1 56 --- qmem rfapi --- Type : Current# Size Total Max# MaxBytes NVE Configuration : 1 2984 2984 1 2984 RFAPI Generic : 1 296 296 1 296 RFAPI Import Table : 1 208 216 1 216
IPv4 Unicast Summary (VRF default):
BGP router identifier XXX.XXX.XXX.XXX, local AS number XXXXXX vrf-id 0
BGP table version 128817
RIB entries 235456, using 43 MiB of memory
Peers 13, using 265 KiB of memory
Peer groups 4, using 256 bytes of memory
IPv6 Unicast Summary (VRF default):
BGP router identifier XXX.XXX.XXX.XXX, local AS number XXXXXX vrf-id 0
BGP table version 254868
RIB entries 371565, using 68 MiB of memory
Peers 13, using 265 KiB of memory
Peer groups 4, using 256 bytes of memory
The OS is linux debian bookworm (all is upgrade to current stable). FRR is installed from package ( repository: https://deb.frrouting.org/frr)
The bgpd system memory usage keep growing until it not consume all the available memory (more than 25G after a hour or two) and kernel's oom-killer just kill the process. From what I see largest memory usage is from large community structures, in the config bgpd do set and match on every peer and peer-group (in the route maps).
I tried to debug with valgrind but it is hard, because that router is in production environment.