zebra: fix stale NHG in kernel #18899
Conversation
43aa593 to
e3024da
Compare
|
Adding another commit to address stale NHG during zebra shutdown. |
|
Is this related to #18891 ??? |
a3f1aae to
7fc84dc
Compare
|
ci:rerun |
|
There is one failure in the CI test. That looks to be failing in other PRs also. It is not related to this code change. |
|
@Mergifyio backport dev/10.4 stable/10.3 stable/10.2 |
✅ Backports have been created
|
yes. this call stack looks to be relevant. let me fix it. |
7fc84dc to
4ceb1bf
Compare
There is an issue with using hash_iterate improperly. The function hash_iterate stores hbnext = hb->next; before calling the callback function . Also, in the zebra_nhg_sweep_stale_entry function, the callback is not just deleting the current bucket - it's causing a cascade of deletions that can free other buckets in the chain as well including ones that the iterator hasn't reached yet. This leads to use-after-free when the iterator tries to access the freed memory. |
|
There is one test failure and that does not look to be relevant to this change. Same failure is seen in other PRs as well. |
|
ci:rerun |
|
@Mergifyio rebase |
❌ Unable to rebase: user
|
|
ci:rerun |
f68fa49 to
3e5c0da
Compare
|
ci:rerun |
|
@ashred-lnx |
LTGM |
|
ci:rerun |
|
The test failures are unrelated to this changes. |
|
ci:rerun |
|
https://github.com/Mergifyio rebase |
I see this issue during below events sequencing 1. zebra creates an nhe and sets 'initial delay' flag for the nexthop received along with kernel/connected route and this routes is a v6 route. 2. Later zebra receives intf_address event for the interface that belongs to the same nhe created above. but this is v4 event. Then zebra iterates through the nhe set linked to this interface and eventually it will end up installing this nhe in kernel So, we install the NHG in kernel for connected/kernel routes and that looks to be deviating from the expected behaviour. All this happens when we receive interface event, we attempt a reinstall for all the NHGs associated with that intf. But if the 'initial delay' is already set for an NHG, we can skip that. Fixing the same. Signed-off-by: Krishnasamy <krishnasamyr@nvidia.com>
During FRR restart nexthop-group entries are not getting cleaned up in below scenario. 1. Let's say an NHG refcnt is getting decremented and it becomes zero. we add a timer for this NHG before deleting it in zebra/kernel. so this NHG will be intact in kernel until the timer expires. 2. Now, the timer is running and frr is getting restarted. All the NHGs are getting cleaned up in kernel but the one that has timer running is still installed in the kernel. Check if any NHG has timer running during zebra shutdown and remove from kernel. Signed-off-by: Krishnasamy <krishnasamyr@nvidia.com>
✅ Branch has been successfully rebased |
3e5c0da to
0743cca
Compare
|
All the comments are addressed and the tests are passing |
zebra: fix stale NHG in kernel (backport #18899)
zebra: fix stale NHG in kernel (backport #18899)
* bgpd: correct no form commands (backport FRRouting#18911) * bgpd: fix to show exist/non-exist-map in 'show run' properly FRRouting#18853 * redhat: make FRR RPM build to work on RedHat 10 (backport FRRouting#18920) * build: check for libunwind.h, not unwind.h (backport FRRouting#18912) * bgpd: use AS4B format for BGP loc-rib messages. (backport FRRouting#18936) * bgpd: fix for the validity and the presence of prefixes in the BGP VPN table. (backport FRRouting#17370) * bgpd: Force adj-rib-out updates if MRAI is kicked in (backport FRRouting#18959) * zebra: Provide SID value when sending SRv6 SID release notify message (backport FRRouting#18971) * bgpd: Fix crash when fetching statistics for bgp instance (backport FRRouting#19003) * nhrpd: fix crash when accessing invalid memory zone (backport FRRouting#18994) * zebra: Initialize RB tree for router tables (backport FRRouting#19049) * zebra: fix null pointer dereference in zebra_evpn_sync_neigh_del (backport FRRouting#19054) * zebra: fix stale NHG in kernel (backport FRRouting#18899) * bgpd: Fix incorrect stripping of transitive extended communities (backport FRRouting#19065) * lib: Fix no on-match goto NUM command (backport FRRouting#19108) * bgpd: Fix extended community check for IP non-transitive type (backport FRRouting#19097) * bgpd: Fix DEREF_OF_NULL.EX.COND in bgp_updgrp_packet (backport FRRouting#19126) * lib: revert addition of vtysh_flush() call in vty_out() (backport FRRouting#19109) * bgpd: Extract link bandwidth value from extcommunity before using for WCMP (backport FRRouting#19165) * Use ipv4 class E addresses (240.0.0.0/4) as connected routes by default (backport FRRouting#18095) * bfdd: Set bfd.LocalDiag when transitioning to AdminDown (backport FRRouting#18592) * zebra: clean up a json object leak (backport FRRouting#19192) * bgpd: Do not try to reuse freed route-maps (backport FRRouting#19191) * lib: fix routemap crash (backport FRRouting#19127) * bgpd: initialize local variable (backport FRRouting#19233) * ospfd: Use after free cleanup of lsa (backport FRRouting#19224) * vtysh: copy config from file should actually apply (backport FRRouting#19242) * bgpd : Fix compilation error in bgpd module: Update TP_ARGS for bgp (backport FRRouting#19266) * bgpd: Ensure addpath does not withdraw selected route in some situations (backport FRRouting#19210) * lib, zebra: mark singleton nexthops inactive/active on link state changes for wecmp (backport FRRouting#18947) * eigrp: validate hello packets and tlvs better (backport FRRouting#19251) * bgpd: [GR] fixed selectionDeferralTimer to display select_defer_time val FRRouting#19283 Signed-off-by: Donatas Abraitis <donatas@opensourcerouting.org>
Fixing stale NHG issue in kernel.
Issue1:
So, we install the NHG in kernel for connected/kernel routes and that looks to be deviating from the expected behaviour. All this happens when we receive interface event, we attempt a reinstall for all the NHGs associated with that intf. But if the 'initial delay' is already set for an NHG, we can skip that.
Fixing the same.
Issue2:
During FRR restart nexthop-group entries are not getting cleaned up in
below scenario.
add a timer for this NHG before deleting it in zebra/kernel.
so this NHG will be intact in kernel until the timer expires.
NHGs are getting cleaned up in kernel but the one that has timer
running is still installed in the kernel.
Check if any NHG has timer running during zebra shutdown and remove from
kernel.