provider: duplicated CIDs sent to provide queue #901
Description
part of ipshipyard/roadmaps#6
The provide queue is getting many duplicate CIDs. It turns out that normal ipfs add and mfs are both adding blocks to the blockservice, and causing duplicated CIDs to be provided.
This may be tolerable since the CIDs are deduplicated after being read from the queue, here, but it does mean the queue has to hold double the CIDs that are actually provided.
It cases where things are added in such a way that follows both call paths, it may be worth passing some parameter that suppresses providing in one of the paths.
Call stack from normal ipfs-add path:
github.com/ipfs/boxo/provider.(*reprovider).Provide(0x14001585080?, {0x10535d940?, 0x14001589e90?}, {{0x14000635ef0?, 0x1?}}, 0x1?)
github.com/ipfs/boxo@v0.29.2-0.20250326002258-acb85b96a907/provider/reprovider.go:464 +0x2c
github.com/ipfs/boxo/exchange/providing.(*Exchange).NotifyNewBlocks(0x14000d4e6a0, {0x10535d940, 0x14001589e90}, {0x140015ba5a0, 0x1, 0x1?})
github.com/ipfs/boxo@v0.29.2-0.20250326002258-acb85b96a907/exchange/providing/providing.go:41 +0xbc
github.com/ipfs/boxo/blockservice.(*blockService).AddBlocks(0x1400158cc40, {0x10535d978, 0x14000b67b80}, {0x140015ba5a0, 0x1, 0x1})
github.com/ipfs/boxo@v0.29.2-0.20250326002258-acb85b96a907/blockservice/blockservice.go:223 +0x3ac
github.com/ipfs/boxo/ipld/merkledag.(*dagService).AddMany(0x14000b3f5a8, {0x10535d978, 0x14000b67b80}, {0x140015ba580, 0x1, 0x10327cdd4?})
github.com/ipfs/boxo@v0.29.2-0.20250326002258-acb85b96a907/ipld/merkledag/merkledag.go:71 +0xe4
github.com/ipfs/go-ipld-format.(*Batch).asyncCommit.func1(...)
github.com/ipfs/go-ipld-format@v0.6.0/batch.go:100
created by github.com/ipfs/go-ipld-format.(*Batch).asyncCommit in goroutine 380
github.com/ipfs/go-ipld-format@v0.6.0/batch.go:98 +0x194
(edited)
Call stack from mfs path:
github.com/ipfs/boxo/provider.(*reprovider).Provide(0x140014ce480?, {0x106c3d940?, 0x14001685860?}, {{0x140016a0930?, 0x1?}}, 0x1?)
github.com/ipfs/boxo@v0.29.2-0.20250326002258-acb85b96a907/provider/reprovider.go:464 +0x2c
github.com/ipfs/boxo/exchange/providing.(*Exchange).NotifyNewBlocks(0x14000f77ca0, {0x106c3d940, 0x14001685860}, {0x140016a62c0, 0x1, 0x1?})
github.com/ipfs/boxo@v0.29.2-0.20250326002258-acb85b96a907/exchange/providing/providing.go:41 +0xbc
github.com/ipfs/boxo/blockservice.(*blockService).AddBlock(0x14001409140, {0x106c3d978, 0x14001421c20}, {0x14fcc9740, 0x140014ce480})
github.com/ipfs/boxo@v0.29.2-0.20250326002258-acb85b96a907/blockservice/blockservice.go:177 +0x250
github.com/ipfs-shipyard/nopfs/ipfs.(*BlockService).AddBlock(0x14001424b70, {0x106c3d978, 0x14001421c20}, {0x14fcc9740, 0x140014ce480})
github.com/ipfs-shipyard/nopfs/ipfs@v0.25.0/blockservice.go:78 +0x1a8
github.com/ipfs/boxo/ipld/merkledag.(*dagService).Add(0x14001424ba0?, {0x106c3d978?, 0x14001421c20?}, {0x106c53a10?, 0x140014ce480?})
github.com/ipfs/boxo@v0.29.2-0.20250326002258-acb85b96a907/ipld/merkledag/merkledag.go:63 +0x94
github.com/ipfs/boxo/mfs.(*Directory).getNode(0x140014ce4e0, 0xa8?)
github.com/ipfs/boxo@v0.29.2-0.20250326002258-acb85b96a907/mfs/dir.go:404 +0x110
github.com/ipfs/boxo/mfs.(*Directory).GetNode(...)
github.com/ipfs/boxo@v0.29.2-0.20250326002258-acb85b96a907/mfs/dir.go:387
github.com/ipfs/boxo/mfs.(*Root).Close(0x140014d8c50)
github.com/ipfs/boxo@v0.29.2-0.20250326002258-acb85b96a907/mfs/root.go:197 +0x34
github.com/ipfs/kubo/core/node.Files.func2({0x10675f880?, 0x106c27180?})
github.com/ipfs/kubo/core/node/core.go:208 +0x20