Skip to content

Conversation

masih
Copy link
Member

@masih masih commented Apr 2, 2025

Introduce the first API for Lotus v2, focusing on ChainGetTipSet within the Chain group. Define TipSetSelector for advanced tipset retrieval options, and create a compact JSON-RPC call format.

Gracefully accommodate both EC and F3 finalized tipsets based on node configuration, where:

  • EC finalized tipset is returned when F3 is turned off, has no finalized tipset or F3 isn't ready.
  • F3 finalized is returned otherwise.

Support three categories of selectors under TipSetSelector:

  • By tag: either "latest" or "finalized."
  • By height: epoch, plus optional fallback to previous non-null tipset.
  • By tipset key.

The open API spec and Markdown generation is excluded from this work to reduce the PR verbosity and will be included in a follow-up PR once the basic API primitives are reviewed.

Note that all v2 APIs are experimental and may change with no notice.

Relates to: #12719
Part of: #12990

@masih masih self-assigned this Apr 2, 2025
@github-project-automation github-project-automation bot moved this to 📌 Triage in FilOz Apr 2, 2025
Introduce the first API for Lotus v2, focusing on `ChainGetTipSet`
within the `Chain` group. Define `TipSetSelector` for advanced
tipset retrieval options, and create a compact JSON-RPC call format.

Gracefully accommodate both EC and F3 finalized tipsets based on node
configuration, where:
* EC finalized tipset is returned when F3 is turned off, has no
  finalized tipset or F3 isn't ready.
* F3 finalized is returned otherwise.

Support three categories of selectors under `TipSetSelector`:
* By tag: either "latest" or "finalized."
* By height: epoch, plus optional fallback to previous non-null tipset.
* By tipset key.

The selection falls back to tag "latest" if the user specifies no
selection criterion.

The JSON-RPC format is designed to use JSON Object as the parameters
passed to the RPC call to remain compact, and extensible. Examples
include:

* Latest Tipset:
  ```json
  {"jsonrpc":"2.0","method":"Filecoin.ChainGetTipSet","id":1}
  ```
* Finalized Tipset:
  ```json
  {"jsonrpc":"2.0","method":"Filecoin.ChainGetTipSet","params":{"tag":"finalized"},"id":1}
  ```

* Tipset at specific height:
  ```json
  {"jsonrpc":"2.0","method":"Filecoin.ChainGetTipSet","params":{"height":{"at":1413,"previous":true}},"id":1}
  ```

The open API spec and Markdown generation is excluded from this work to
reduce the PR verbosity and will be included in a follow-up PR once the
basic API primitives are reviewed.

Note that all v2 APIs are experimental and may change with no notice.

Relates to: #12719
Part of: #12990
@masih masih force-pushed the masih/v2api-chain-get-tipset branch from 20231de to 905966a Compare April 2, 2025 15:42
@masih masih marked this pull request as ready for review April 2, 2025 15:45
@masih masih requested review from rvagg and Kubuxu April 2, 2025 15:45
@rvagg
Copy link
Member

rvagg commented Apr 3, 2025

I can live with this I think. I don't love the verbosity of the nesting but at least you've come up with short descriptors, "tag" and "at", I like that 👍. Good stuff.

@rvagg
Copy link
Member

rvagg commented Apr 3, 2025

Unrelated flaky itest recorded @ #13005

Introduce the concept of tipset anchor, to which a selection by height
is anchored, i.e. parent of the target tipset.

The anchor may be a tipset tag or a tipset key.

Reflect the changes across tests .
@masih masih requested a review from rvagg April 3, 2025 13:08
@Kubuxu
Copy link
Contributor

Kubuxu commented Apr 3, 2025

Initial quick skim:

  • Height is a bit of a miss-nomer, we don't track height but an epoch in filecoin as height originally refers to the height of the hash tree.
  • Why the generics complexity logic in the type? Why not use the TipSetCriterion directly with additional validation checks in the API?

@BigLep BigLep added this to F3 Apr 3, 2025
@masih
Copy link
Member Author

masih commented Apr 3, 2025

Discussed at stand up:

Height is a bit of a miss-nomer, we don't track height but an epoch in filecoin as height originally refers to the height of the hash tree.

It is. From internal notion doc discussions I got the sense that we want to switch to "height" as the new terminology. Happy to revert back to Epoch, or move on with Epoch. Either works.

@Kubuxu kindly pointed out that the Go SDK to interact with selectors is slightly cumbersome. I am open to suggestions on this one and I think we can iterate on it as non-breaking API changes later. For now I am keen to pin down the primitives for the Lotus V2 API first and then iterate.

Why the generics complexity logic in the type? Why not use the TipSetCriterion directly with additional validation checks in the API?

The generics is used solely as Type Constraint to NewTipSetSelector. We can't use TipSetCriterion as argument to ChainGetTipSet directly, because it is not a type alias of jsonrpc.RawParams without losing the ability to accept requests with no params specified. This is a limitation inherited by the implementation of jsonrpc library used by Lotus.

During the discussion, the choice of default param as "Heaviest" if nothing is specified came up:

  • @Kubuxu mentioned that maybe we should explicitly ask for what the user wants and have no default if no selector is specified.
  • I have no strong opinions on the defaults. I do see that explicit is better ( which is the rationale behind changes here compared to prior iterations). However, we would lose the "short-and-sweet" aspect of API calls by users in other languages, a non-functional requirement for the v2 API as I understand it.

@Kubuxu please add anything that I might have missed.

@rvagg thoughts?

@Kubuxu
Copy link
Contributor

Kubuxu commented Apr 3, 2025

Discussed it a bit with Masih.

  1. The weird typing is required if we want to have Filecoin.ChainGetTipSet with no params to work across JSOR-RPC
  2. IDK if I like that you will be required to type,
ts, err := api.ChainGetTipSet(ctx, types.NewTipSetSelector(types.TipSetHeight{
	At:       123,
	Previous: true,
	Anchor: &types.TipSetAnchor{
		Tag: &types.TipSetTags.Finalized,
		Key: &types.TipSetKey{},
	},
}))
  1. I'm 90% that the latest on empty is not what we want. Split across "empty invalid" and empty returning finalized.

written before Masih wore the above ^^

Copy link
Member

@BigLep BigLep left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking comments from me. Just following along.

I wanted to confirm we are good with how calling these APIs looks like from other languages (particularly JS).

@BigLep
Copy link
Member

BigLep commented Apr 3, 2025

I'm 90% that the latest on empty is not what we want. Split across "empty invalid" and empty returning finalized.

(Not an expert opinion here), but I agree. I think it's important to help builders "avoid running into trouble unknowingly". We can do this by either (as Kuba says):

  1. Be explicit - no default
  2. Use finalized

That said, maybe best to match ETH ecosystem here? Looking at getBlockByNumber (which I assume is a good corollary), it looks like there is no default.

@masih
Copy link
Member Author

masih commented Apr 4, 2025

Going ahead with no default.

IDK if I like that you will be required to type

Let's see what things look like after explicit param is needed and we can iterate. Probably faster to sit down and go through it synchronously to distill actionables on this one.

@masih
Copy link
Member Author

masih commented Apr 4, 2025

@Kubuxu Do you want to get rid of the default anchor for getting tipset by height too or leave it as optional?

Remove transparent fallback to heaviest tipset when no selector is
specified.
@masih
Copy link
Member Author

masih commented Apr 4, 2025

@Kubuxu

  • A selector is now strictly required; no defaults.
  • We can't get rid of the type alias to jsonrpc.RawParams without making request body more ugly:
    • The jsonrpc library lotus uses would implicitly switch to using JSON array as parms if the type of an argument is not jsonrpc.RawParams.
    • This means request params on the wire would have to have an array wrapping, e.g.: {"jsonrpc":"2.0","method":"Filecoin.ChainGetTipSet","params":[{"tag":"finalized"}],"id":1} instead of {"jsonrpc":"2.0","method":"Filecoin.ChainGetTipSet","params":{"tag":"finalized"},"id":1}.
    • The choices are to: use raw params type alias, live with the redundant wrapping, or fix go-jsonrpc to do things differently.
      • I don't think it makes sense to use the mix-and-match style params experimented with in feat(api): v2 API integrating with F3 #12719 because for me that conflicts with "Explicit APIs" we desire. My vote is to keep the parameters keyed if the API arg needs to be extensible.
    • My pick is to use the the params type alias as it is the most straightforward way to get what we want while keeping the API extensible if we want to add anything else.
  • I have not touched the Go API for constructing selectors. This one is mostly bike shedding and best done synchronously.

@Kubuxu
Copy link
Contributor

Kubuxu commented Apr 4, 2025

the params being array is fully expected. It is a single positional argument, which is an object.
I haven't seen the named parameters from JSON-RPC spec used anywhere.
They are allowed by not that popular.

@rvagg
Copy link
Member

rvagg commented Apr 7, 2025

Looks really good, I'll have to try and restart my Eth API work ontop of this for /v2 to remind myself about all the module hackery needed to make this work. But I like how the TipSetSelector has worked out here and my initial take is that the the ChainGetTipSet v2 wrapper is nice and clean, although I wonder how we're going to keep it simple when we need to deal with selecting a tipset for all the other methods (thinking especially the State* ones) that currently take an explicit tipset or EmptyTSK. Perhaps it's the v1 APIs that have to get wrapped to suit the internal v2 methods where a TipSetSelector is what we accept on the internal functions.

@masih
Copy link
Member Author

masih commented Apr 7, 2025

how we're going to keep it simple when we need to deal with selecting a tipset for all the other methods (thinking especially the State* ones

Thanks for pointing that out. I will pick up a State API next to flesh out any complexities before making Chain* more featureful.

Perhaps it's the v1 APIs that have to get wrapped to suit the internal v2 methods

This has been my thinking so far: to eventually turn v1 APIs into a wrapper of v2.

masih added 2 commits April 7, 2025 12:47
Make `at` in by-height selector a pointer and validate that it is
specifically set. This is to avoid zero-valued Go types shadow
accidental valid calls.
Unpack the fullnode interfaces because docsgen garbage heap does not
support nested types.

Generate docs.
Copy link
Member

@BigLep BigLep left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sweet to see the doc generation happening.

I left a couple of comments when wearing the lens of "learn about new APIs from the generated docs". (I'm not up on all the limitations of our custom doc generation system.)

@masih masih requested review from rvagg and Kubuxu April 8, 2025 16:21
@masih
Copy link
Member Author

masih commented Apr 8, 2025

@Kubuxu @rvagg can I get a review on this PR so that we can distill what's missing from this PR to land please?

Copy link
Contributor

@Kubuxu Kubuxu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SGWM

@github-project-automation github-project-automation bot moved this from 📌 Triage to ✔️ Approved by reviewer in FilOz Apr 9, 2025
@masih masih enabled auto-merge (squash) April 9, 2025 15:28
@masih masih merged commit a1f1d51 into master Apr 9, 2025
90 of 91 checks passed
@masih masih deleted the masih/v2api-chain-get-tipset branch April 9, 2025 16:09
@github-project-automation github-project-automation bot moved this from ✔️ Approved by reviewer to 🎉 Done in FilOz Apr 9, 2025
@github-project-automation github-project-automation bot moved this from In review to Done in F3 Apr 9, 2025
@@ -10,6 +10,7 @@
# UNRELEASED

- fix(eth): always return nil for eth transactions not found ([filecoin-project/lotus#12999](https://github.com/filecoin-project/lotus/pull/12999))
- feat: add experimental v2 APIs that are "F3 aware." (TODO: expand this section significantly to cover where someone learns about the new APIs, how they enable them, and what expectations they should have around them—i.e., they may change)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also TODO, when we revisit this, we need the link to this PR; I'll try and remember to add it when I follow-up with Eth stuff if it's not done already by then

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Help Me Obi-Wan Kenobi Generative AI, You’re My Only Hope.

Copy link
Member

@rvagg rvagg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry I was slow on the draw here, but a post-merge LGTM from me

@BigLep
Copy link
Member

BigLep commented Apr 11, 2025

Let's discuss further in stand up (about named parameters).

Inevitably this is going to come up again. I think we should capture the rationale for the decision. I extracted some of the statements I saw to https://www.notion.so/filecoindev/Filecoin-V2-APIs-1d0dc41950c1808b914de5966d501658?pvs=4#1d2dc41950c180019135f921e171e984 but it needs work with happened verbally.

@elmattic elmattic mentioned this pull request May 7, 2025
54 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Status: 🎉 Done
Development

Successfully merging this pull request may close these issues.

4 participants