provider: clear provide queue when reprovide strategy changes #10863

gammazero · 2025-07-10T07:55:36Z

When the currently configured reprovide strategy does not match the previous strategy read from the datastore, then clear the provide queue and update the reprovide strategy that is stored in the datastore.

Testing was done manually.

Depends on ipfs/boxo#978

Closes #10829

When the currently configured reprovide strategy does not match the previous strategy read from the datastore, then clear the reprovide queue and update the reprovide strategy that is stored in the datastore. Depends on ipfs/boxo#978 Closes #10829

guillaumemichel

LGTM! We may also want to add a command to allow users to manually purge the provider queue.

docs/changelogs/v0.36.md

Co-authored-by: Guillaume Michel <guillaumemichel@users.noreply.github.com>

gammazero · 2025-07-10T08:48:42Z

@guillaumemichel

We may also want to add a command to allow users to manually purge the provider queue.

We could, but what is the use case? If the use case is very rare, then a user could always change the reprovide strategy, restart, then change it back and restart again.

guillaumemichel · 2025-07-10T09:01:27Z

@hsanjuan since you opened #10829, do you think having a command would be useful?

hsanjuan · 2025-07-10T14:00:16Z

@hsanjuan since you opened #10829, do you think having a command would be useful?

Yeah, I think it would be useful (if not too complicated). It can be useful for debugging purposes. "Is reproviding the cause of my high CPU/network usage?".. the question can be answered quickly by cleaning the queue and seeing if problem resolves at that moment. Might also be useful when you have pinned many things and then unpinned them, but you are nevertheless reproviding them. The "edit config and restart-workaround" will also eliminate other potential explanations for problems and it is not very obvious.

guillaumemichel · 2025-07-10T14:22:21Z

"Is reproviding the cause of my high CPU/network usage?"

Just to be clear, this PR is clearing the provide queue but not changing anything about reprovides.

The provide process is as follow:

kubo/boxo wants to provide a cid c
c is added to provide queue
when all the cids before c in the queue have been provided c is provided
periodically, every ReprovideInterval c is reprovided
a. All the cids returned by the KeyChanFunc are reprovided
b. There is no reprovide queue, but rather the reprovider is iterating on the keys to reprovide
c. The reprovide process currently cannot be cancelled

Hence, with a command, users would be able to know whether providing is the cause of high CPU/network usage.

Might also be useful when you have pinned many things and then unpinned them, but you are nevertheless ~~reproviding~~ providing them.

In this case it would help, no keys would be provided after the provide queue is cleared (for better or worse). But in any case, unpinned cids would NOT have been reprovided.

gammazero · 2025-07-10T17:09:00Z

@guillaumemichel @hsanjuan I will provide a command/subcommand to clear the provide queue, but I will do this is a separate PR. The command will be provide with a clear subcommand, i.e. ipfs provide clear. This will give us a place to put future commands that affect the provide and reprovide operation. Dose this sound good?

guillaumemichel

ipfs provide clear

That's perfect!

The `provide clear` command clears all items from the provide queue and prints out the number of items removed from the queue. The `quiet` option tells the command not to print output.

gammazero · 2025-07-11T06:55:50Z

Add `provide clear` command to clear provide queue

The provide clear command clears all items from the provide queue and prints out the number of items removed from the queue. The quiet option tells the command not to print output.

lidel

small nits inline

core/commands/provide.go

docs/changelogs/v0.37.md

core/node/provider.go

lidel · 2025-07-11T14:01:03Z

core/commands/provide.go

+	Subcommands: map[string]*cmds.Command{
+		"clear": provideClearCmd,


💭 in the future we may want to deprecate ipfs routing provide and ipfs routing reprovide have modern version at ipfs provide stat [cid] that returns state of both systems + optional stat per cid (when/why it was provided last time)

I can do that, but in a separate PR.

Issue #10865

We may want to deprecate ipfs stats provide and ipfs stats reprovide, but probably not ipfs routing provide <cid> since the command allows the user to advertise a provider record matching the cid to the Amino DHT.

@guillaumemichel Want to create an issue?

Co-authored-by: Marcin Rataj <lidel@lidel.org>

guillaumemichel

ipfs provide clear

That's perfect!

Now that I think again about it, it would be better to group the command (if possible) as a subcommand of ipfs routing provide rather than create the new ipfs provide just to clear the queue.

gammazero · 2025-07-14T22:59:23Z

Now that I think again about it, it would be better to group the command (if possible) as a subcommand of ipfs routing provide rather than create the new ipfs provide just to clear the queue.

The ipfs routing provide command already looks like this:

ipfs routing provide <key>...       - Announce to the network that you are providing given values.

This means that the clear subcommand would instead need to become a flag, or the keyword "clear" will need to be specially recognized as not a CID. I do not like making a --clear flag, because that completely changes the meaning of the existing provide command. Having a specially recognized keyword is just bad.

So, maybe better to have a different subcommand for ipfs routing. How about ipfs routing clear-keys? Other ideas welcome.

guillaumemichel · 2025-07-15T07:08:48Z

@gammazero how about a subcommand like in 6b7dacf?

I don't think that clear would be an accepted cid anyway, so we shouldn't have any collision here.

Feel free to revert the commit.

gammazero · 2025-07-15T14:12:37Z

I don't think that clear would be an accepted cid anyway, so we shouldn't have any collision here.

It is not collision that I worry about, it is this is not how any of the rest of the CLI works, and is therefore strange and unexpected. @lidel Do you have suggestion?

lidel · 2025-07-15T14:39:22Z

Triage notes:

this PR ~~stays as-is~~ adds ipfs provide clear cmd so we dont need to worry about command namespace conflicts + create namespace focused on providing
we cleanup/add other commands in new PRs - see Move ipfs routing provide and reprovide into ipfs provide namespace #10865

moving to new namespace to avoid conflicts, and also document other commands

lidel

Lgtm, automatic queue purging will likely help people who were stuck with all/flag.

Pushed non-functional changes:

moved clear command to new EXPERIMENTAL namespace (ipfs provide)
updated changelog/docs + wrote basic ipfs provide --help that will help users to discover existing provider-related commands

@gammazero @guillaumemichel If my changes look ok, feel free to merge.

let's create ipfs provide and see how it goes

gammazero requested a review from a team as a code owner July 10, 2025 07:55

gammazero requested review from lidel and guillaumemichel July 10, 2025 08:11

guillaumemichel reviewed Jul 10, 2025

View reviewed changes

docs/changelogs/v0.36.md Outdated Show resolved Hide resolved

guillaumemichel changed the title ~~provider: clear reprovide queue when reprovide strategy changes~~ provider: clear provide queue when reprovide strategy changes Jul 10, 2025

gammazero and others added 2 commits July 10, 2025 01:35

Update docs/changelogs/v0.36.md

143b5a9

Co-authored-by: Guillaume Michel <guillaumemichel@users.noreply.github.com>

update log message

e218197

update boxo

8f87573

Move change log to v0.37.md

4f5248a

gammazero force-pushed the clear-reprovide-queue branch from 5c5293d to 4f5248a Compare July 10, 2025 17:29

guillaumemichel approved these changes Jul 10, 2025

View reviewed changes

Add provide clear command to clear provide queue

625b669

The `provide clear` command clears all items from the provide queue and prints out the number of items removed from the queue. The `quiet` option tells the command not to print output.

gammazero requested a review from guillaumemichel July 11, 2025 07:14

lidel reviewed Jul 11, 2025

View reviewed changes

gammazero and others added 7 commits July 11, 2025 12:43

Update core/commands/provide.go

1409258

Co-authored-by: Marcin Rataj <lidel@lidel.org>

Update docs/changelogs/v0.37.md

6036273

Co-authored-by: Marcin Rataj <lidel@lidel.org>

Update core/node/provider.go

052a309

Co-authored-by: Marcin Rataj <lidel@lidel.org>

Update docs/changelogs/v0.37.md

2c8fe74

Co-authored-by: Marcin Rataj <lidel@lidel.org>

Merge branch 'master' into clear-reprovide-queue

064ee2f

Update core/node/provider.go

15c9440

Co-authored-by: Marcin Rataj <lidel@lidel.org>

Merge branch 'master' into clear-reprovide-queue

cc2e0df

guillaumemichel mentioned this pull request Jul 14, 2025

provider: keystore libp2p/go-libp2p-kad-dht#1096

Merged

guillaumemichel previously requested changes Jul 14, 2025

View reviewed changes

Merge branch 'master' into clear-reprovide-queue

b567286

guillaumemichel force-pushed the clear-reprovide-queue branch from a76d0a9 to 5eb2fda Compare July 15, 2025 07:17

ipfs routing provide clear

6b7dacf

guillaumemichel force-pushed the clear-reprovide-queue branch from 5eb2fda to 6b7dacf Compare July 15, 2025 07:26

This was referenced Jul 15, 2025

Move ipfs stats provide and reprovide into ipfs provide stat #10869

Closed

Provider interface libp2p/go-libp2p-kad-dht#1093

Closed

gammazero added the need/triage Needs initial labeling and prioritization label Jul 15, 2025

lidel mentioned this pull request Jul 15, 2025

Release 0.37 #10867

Closed

51 tasks

lidel mentioned this pull request Jul 15, 2025

Move ipfs routing provide and reprovide into ipfs provide namespace #10865

Open

gammazero and others added 3 commits July 15, 2025 08:29

Merge branch 'master' into clear-reprovide-queue

4db1ccd

refactor(cmds): ipfs provide clear

c4b49d8

moving to new namespace to avoid conflicts, and also document other commands

docs: clarify Reprovider.Strategy

b1cc294

lidel requested a review from guillaumemichel July 15, 2025 18:50

lidel approved these changes Jul 15, 2025

View reviewed changes

lidel and others added 2 commits July 15, 2025 20:55

chore: remove undesired md link

c822368

spelling in changelog

83fe56c

gammazero merged commit a22efea into master Jul 16, 2025
16 of 17 checks passed

gammazero deleted the clear-reprovide-queue branch July 16, 2025 08:28

		Subcommands: map[string]*cmds.Command{
		"clear": provideClearCmd,

Uh oh!

provider: clear provide queue when reprovide strategy changes #10863

provider: clear provide queue when reprovide strategy changes #10863

Uh oh!

Conversation

gammazero commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

guillaumemichel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gammazero commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

guillaumemichel commented Jul 10, 2025

Uh oh!

hsanjuan commented Jul 10, 2025

Uh oh!

guillaumemichel commented Jul 10, 2025

Uh oh!

gammazero commented Jul 10, 2025

Uh oh!

guillaumemichel left a comment

Choose a reason for hiding this comment

Uh oh!

gammazero commented Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Add provide clear command to clear provide queue

Uh oh!

lidel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lidel Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gammazero Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

guillaumemichel Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

gammazero Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

guillaumemichel left a comment

Choose a reason for hiding this comment

Uh oh!

gammazero commented Jul 14, 2025

Uh oh!

guillaumemichel commented Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gammazero commented Jul 15, 2025

Uh oh!

lidel commented Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lidel left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

gammazero commented Jul 10, 2025 •

edited

Loading

gammazero commented Jul 10, 2025 •

edited

Loading

gammazero commented Jul 11, 2025 •

edited

Loading

Add `provide clear` command to clear provide queue

lidel Jul 11, 2025 •

edited

Loading

gammazero Jul 11, 2025 •

edited

Loading

gammazero Jul 14, 2025 •

edited

Loading

guillaumemichel commented Jul 15, 2025 •

edited

Loading

lidel commented Jul 15, 2025 •

edited

Loading

lidel left a comment •

edited

Loading