-
-
Notifications
You must be signed in to change notification settings - Fork 3.1k
Description
This is about
/api/v0/pubsub/*
, and not IPNS-over-Pubsub.
Kubo's PubSub RPC API (/api/v0/pubsub/*
) provides access to a somewhat reliable multicast protocol.
However it has many issues:
Lack of a real contract on the reliability of messages
The messages reliability contract is more or less:
- It's mostly reliable, without retransmission but using reliable transports and a flood like action.
(AKA whatevergo-libp2p-pubsub
implements) - Duplicate messages shouldn't be shared too much across the mesh and should die off after some time.
(Attempt to avoid storms)
First point, go-libp2p-pubsub
has a bunch of options and callbacks you can configure to change the behaviour of the mesh (view Examples of correct usage of go-libp2p-pubsub
section below).
The options are not currently configurable, it's not just a question of exposing them, the really impactfull options are the one locked behind callback like validators.
Two potential solutions are to have clients maintain a websocket, grpc, ... stream open with Kubo, then when Kubo receives a callback from go-libp2p-pubsub
it can forward the arguments over the API stream and wait for the response. Or add something like WASM to run custom application code inside Kubo, then you will be able to configure WASM blobs which implements the validator you want.
This is much harder than just throwing a WASM interpreter and writing a few hundred SLOCs of glue code, because most validators you would want write would need to access and store some application related state. (for example in a CRDT application, do not relay messages that advertise a HEAD that is lower than the currently known HEAD).
Second point, our current implementation of message deduplication use a bounded cache to find duplicates, if the mesh gets wider than the cache size, you can reach an exponential broadcast storm like event: #9665, sadly linking to point one, even tho the fix is supposed to be transparent and implement a visibly similar message deduping logic except it does not have a bounded size this make our interop tests very flaky and thus might break various stuff in the ecosystem.
Confusing Architecture
I had more than I have fingers discussions with various peoples who complain that Kubo pubsub does not work, they never receive messages.
Almost always the issue is that they are running ipfs http clients in the browser, open two browser tabs, and then try to receive messages from the other tab.
This does not work because Kubo does not think of the two clients as two clients, from Kubo's point of view the http api is remote controlling the Kubo node. Thus the fact that the browsers are different tabs are different browsers instance, is not taken into a count, as far as Kubo can see, the messages are sent by the same node (itself) and it does not return you your own message because receiving messages you sent yourself is confusing.
This is a perfectly valid usecase, just not what the API was designed to do (you can implement this is to use js-libp2p in the browser then your browser node would use floodsub to a local Kubo node, with messages going through the libp2p swarm instead of the HTTP API).
Future of the API
Currently the pubsub API is not in a good place and correctly advertise this:
EXPERIMENTAL FEATURE It is not intended in its current state to be used in a production environment. To use, the daemon must be run with '--enable-pubsub-experiment'.
Our current team goals are to move away from the two ABIs (HTTP & Go) maintenance costs for people who want to build applications on top of IPFS by providing a consistent Go ABI story (go-libipfs
) and a comprehensive example collection on how to use this (go-libipfs/examples
).
Fixing the PubSub API require lots of work which does not allign with theses goals and thus to not justify allocating much time on this when we enginer time is at a premium.
go-libp2p-pubsub
's Go API is already competent and capable of satisfying the needs of consumers proven by the Production examples of correct usage of go-libp2p-pubsub
section below. go-libp2p-pubsub
will continue to stay part of libp2p given this really have very little to do with IPFS and can be used by any libp2p project (ETH2 for example).
Ways for creating a soft landing
To ease the pain of people currently using the PubSub HTTP API Kubo API, we could:
- create a new daemon binary that provide the same endpoints. That said, we wouldn't have plans to maintain it and it is most likely going to have the same issues as the current API. Someone else would need to pick it up to maintain it.
- As part of @ipfs/kubo-maintainer's example-driven development strategy, we could create missing examples on how to bootstrap a project using
go-libp2p-pubsub
if that is useful. (TBD where that example will live but could be something like full-example in libp2p/go-libp2p-pubub that is validated as part of CI)
Production examples of correct usage of go-libp2p-pubsub
For good example of how to use libp2p-pubsub's effectively see things like:
- Lotus's pubsub used in many layer of the Filecoin consensus to discover state and share state updates.
- Prysm's beacon chain which does something similar but for the ETH2 consensus layer.
## Tasks
- [x] @Jorropo Close https://github.com/ipfs/kubo/issues/6621 (pointing to the issue above)
- [X] @Jorropo Create a topic in discuss.ipfs.tech (https://discuss.ipfs.tech/t/help-kubo-maintainers-about-usecases-for-the-http-pubsub-api/16097). Include that we are planning to remove pubsub (link to this issue) and "Please comment below to share your usecase and why you have used this in Kubo."
- [x] @Jorropo For Kubo 0.19, PR for moving pubsub from experimental to deprecated (referencing this issue). This should have a changelog update. Hide/remove all the docs about this as well since we don't want anyone else putting weight on this. https://github.com/ipfs/kubo/pull/9718
- [ ] post 0.20 / IPFS Thing, create the migration path / soft landing so we can fully remove pubsub from Kubo.
- [ ] Remove pubsub from Kubo