Skip to content

Conversation

stickies-v
Copy link
Contributor

@stickies-v stickies-v commented Jan 18, 2022

In RESTful APIs, typically path parameters (e.g. /some/unique/resource/) are used to represent resources, and query parameters (e.g. ?sort=asc) are used to control how these resources are being loaded through e.g. sorting, pagination, filtering, ...

As first discussed in #17631, the current REST api contains two endpoints /headers/ and /blockfilterheaders/ that rather unexpectedly use path parameters to control how many (filter) headers are returned in the response. While this is no critical issue, it is unintuitive and we are still early enough to easily phase this behaviour out and ensure new endpoints (if any) do not have to stick to non-standard behaviour just for internal consistency.

In this PR, a new HTTPRequest::GetQueryParameter method is introduced to easily parse query parameters, as well as two new /headers/ and /blockfilterheaders/ endpoints that use a count query parameter are introduced. The old path parameter-based endpoints are kept without too much overhead, but the documentation now points to the new query parameter-based endpoints as the default interface to encourage standardness.

Behaviour change

New endpoints and default values

/headers/ and /blockfilterheaders/ now have 2 new endpoints that contain query parameters (?count=<count>) instead of path parameters (/<count>/), as described in REST-interface.md. Since query parameters can easily have default values, I have set this at 5 for both endpoints.

headers
GET /rest/headers/<BLOCK-HASH>.<bin|hex|json>?count=<COUNT=5>
should now be used instead of
GET /rest/headers/<COUNT>/<BLOCK-HASH>.<bin|hex|json>

blockfilterheaders
GET /rest/blockfilterheaders/<FILTERTYPE>/<BLOCK-HASH>.<bin|hex|json>?count=<COUNT=5>
should now be used instead of
GET /rest/blockfilterheaders/<FILTERTYPE>/<COUNT>/<BLOCK-HASH>.<bin|hex|json>

Some previously invalid API calls are now valid

API calls that contained query strings in the URI could not be parsed prior to this PR. This PR changes behaviour in that previously invalid calls (e.g. GET /rest/headers/5/somehash.json?someunusedparam=foo) would now become valid, as the query parameters are properly parsed, and discarded if unused.
For example, prior to this PR, adding an irrelevant someparam parameter would be illegal:

GET /rest/headers/5/0000004c6aad0c89c1c060e8e116dcd849e0554935cd78ff9c6a398abeac6eda.json?someparam=true
->
Invalid hash: 0000004c6aad0c89c1c060e8e116dcd849e0554935cd78ff9c6a398abeac6eda.json?someparam=true

This behaviour change affects all rest endpoints, not just the 2 new ones introduced here.

(Note: I'd be open to implementing additional logic to refuse requests containing unrecognized query parameters to minimize behaviour change, but for the endpoints that we currently have I don't really see the point for that added complexity. E.g. I don't see any scenarios where misspelling a parameter could lead to harmful outcomes)

Using the REST API

To run the API HTTP server, start a bitcoind instance with the -rest flag enabled. To use the
blockfilterheaders endpoint, you'll also need to set -blockfilterindex=1:

./bitcoind -signet -rest -blockfilterindex=1

As soon as bitcoind is fully up and running, you should be able to query the API, for example by
using curl on the command line: curl "127.0.0.1:38332/rest/chaininfo.json".
To more easily parse the JSON output, you can also use tools like 'jq' or json_pp, e.g.:

curl -s "localhost:38332/rest/blockfilterheaders/basic/0000004c6aad0c89c1c060e8e116dcd849e0554935cd78ff9c6a398abeac6eda.json?count=2" | json_pp .

To do

  • update doc/release-notes

Feedback

This is my first PR (hooray!). Please don't hold back on any feedback/comments/nits/... you may have, big or small, whether they are code, process, language, ... related. I welcome private messages too if there's anything you don't want to clutter the PR with. I'm here to learn and am grateful for everyone's input.

@DrahtBot
Copy link
Contributor

DrahtBot commented Jan 19, 2022

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #24456 (blockman: Properly guard blockfile members by dongcarl)
  • #23599 (Tidy up RPCTxSerializationFlags by MarcoFalke)
  • #23507 (Refactor: Improve API design of ScriptToUniv, TxToUniv etc to return the UniValue instead of mutating a parameter/reference by mjdietzx)
  • #22953 (refactor: introduce single-separator split helper (boost::split replacement) by theStack)
  • #22564 (refactor: Move mutable globals cleared in ::UnloadBlockIndex to BlockManager by dongcarl)
  • #21422 (Add feerate histogram to getmempoolinfo by kiminuo)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@stickies-v
Copy link
Contributor Author

Force pushed to rebase after #24054 was merged; no other changes.

Copy link
Contributor

@brunoerg brunoerg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK

Tested ?count=1, ?count=5 and ?count=600 (didn't paste here the result of it, obviously haha) for /rest/headers/

http://127.0.0.1:8332/rest/headers/000000001aeae195809d120b5d66a39c83eb48792e068f8ea1fea19d84a4278a.json?count=1

[{"hash":"000000001aeae195809d120b5d66a39c83eb48792e068f8ea1fea19d84a4278a","confirmations":237003,"height":50000,"version":1,"versionHex":"00000001","merkleroot":"27f1d66f8a1ee5280f4e92508dcb647e954d53004905d08a75574daee4988360","time":1270916538,"mediantime":1270915355,"nonce":58489010,"bits":"1c2a1115","difficulty":6.085476906000794,"chainwork":"00000000000000000000000000000000000000000000000000014a426593e15b","nTx":1,"previousblockhash":"000000000845517b31c6820d83f25cff46429bf136a7515fe504116427e60f8e","nextblockhash":"000000001c920d495e1eeef2452b6d1c6c229a919b28196c103ecffebabee141"}]

http://127.0.0.1:8332/rest/headers/000000001aeae195809d120b5d66a39c83eb48792e068f8ea1fea19d84a4278a.json?count=5

[{"hash":"000000001aeae195809d120b5d66a39c83eb48792e068f8ea1fea19d84a4278a","confirmations":240714,"height":50000,"version":1,"versionHex":"00000001","merkleroot":"27f1d66f8a1ee5280f4e92508dcb647e954d53004905d08a75574daee4988360","time":1270916538,"mediantime":1270915355,"nonce":58489010,"bits":"1c2a1115","difficulty":6.085476906000794,"chainwork":"00000000000000000000000000000000000000000000000000014a426593e15b","nTx":1,"previousblockhash":"000000000845517b31c6820d83f25cff46429bf136a7515fe504116427e60f8e","nextblockhash":"000000001c920d495e1eeef2452b6d1c6c229a919b28196c103ecffebabee141"},{"hash":"000000001c920d495e1eeef2452b6d1c6c229a919b28196c103ecffebabee141","confirmations":240713,"height":50001,"version":1,"versionHex":"00000001","merkleroot":"ee3a2d2b895cafacff526d06a55b55e049cf84a9735e4a63f7fd08f96d0f4649","time":1270917100,"mediantime":1270915522,"nonce":56717043,"bits":"1c2a1115","difficulty":6.085476906000794,"chainwork":"00000000000000000000000000000000000000000000000000014a487b7bc7c6","nTx":3,"previousblockhash":"000000001aeae195809d120b5d66a39c83eb48792e068f8ea1fea19d84a4278a","nextblockhash":"000000002066d7f9b134c30b5005b7bf5fbfa52f279883f1fde793dcdc964266"},{"hash":"000000002066d7f9b134c30b5005b7bf5fbfa52f279883f1fde793dcdc964266","confirmations":240712,"height":50002,"version":1,"versionHex":"00000001","merkleroot":"5164dc2785549f9efe14eb1c54522ec1874a02b7eda164fde370c05412f037ad","time":1270917181,"mediantime":1270915874,"nonce":9804838,"bits":"1c2a1115","difficulty":6.085476906000794,"chainwork":"00000000000000000000000000000000000000000000000000014a4e9163ae31","nTx":1,"previousblockhash":"000000001c920d495e1eeef2452b6d1c6c229a919b28196c103ecffebabee141","nextblockhash":"00000000237ce1b692f5ef9f1f9ac64d62d5e6a24dcfec08633fa2d339434e2a"},{"hash":"00000000237ce1b692f5ef9f1f9ac64d62d5e6a24dcfec08633fa2d339434e2a","confirmations":240711,"height":50003,"version":1,"versionHex":"00000001","merkleroot":"792c4a613e51966fc1c852cd8d94fd10a05efa55fffdd6a41281c8d139540b23","time":1270917916,"mediantime":1270916181,"nonce":311792473,"bits":"1c2a1115","difficulty":6.085476906000794,"chainwork":"00000000000000000000000000000000000000000000000000014a54a74b949c","nTx":1,"previousblockhash":"000000002066d7f9b134c30b5005b7bf5fbfa52f279883f1fde793dcdc964266","nextblockhash":"000000000528e8aac90e3096abb7b1f3d52efa2cd8f12754614788cb43eefe11"},{"hash":"000000000528e8aac90e3096abb7b1f3d52efa2cd8f12754614788cb43eefe11","confirmations":240710,"height":50004,"version":1,"versionHex":"00000001","merkleroot":"3cfbd9a91f033e37f05764215626c542c735ed71cd5b53467834cf881c0b9ebd","time":1270918264,"mediantime":1270916354,"nonce":78553962,"bits":"1c2a1115","difficulty":6.085476906000794,"chainwork":"00000000000000000000000000000000000000000000000000014a5abd337b07","nTx":1,"previousblockhash":"00000000237ce1b692f5ef9f1f9ac64d62d5e6a24dcfec08633fa2d339434e2a","nextblockhash":"0000000000e1ad737f11fe4cb3d126408b2cbcc28ab0d83293f408e413733338"}]

I tested the same values but using /rest/headers/<COUNT>/<BLOCK-HASH>.<bin|hex|json> format and it worked as well. I didn't understand Deprecated (but not removed) since 23.0:, will both ways be kept?

@maflcko
Copy link
Member

maflcko commented Jan 19, 2022

Is there a large cost to maintaining both ways? If not, it might be better to keep both to not upset users for a weak motivation.

@stickies-v
Copy link
Contributor Author

Is there a large cost to maintaining both ways? If not, it might be better to keep both to not upset users for a weak motivation.

The PR is already setup to maintain both ways, so anyone using the (to be) deprecated endpoints should not be affected. This is documented in REST-interface.md too, so users are aware it is still supported but that they should probably use the new endpoint for new applications. The maintenance overhead is minimal, basically in rest_headers and rest_filter_header we have an if/else branch that adds 3 LoC to maintain the old endpoints too. Some additional refactoring could be done to simplify without the two branches, but it's no big deal.

Personally, I would definitely not remove the old endpoints in the next release, and probably not until some bigger API refactoring would make this desirable, in which case we should notify users more explicitly. I did not add any deprecation logging to avoid behaviour change, and because I think we don't have to push users away from the old endpoints yet.

@stickies-v
Copy link
Contributor Author

I didn't understand Deprecated (but not removed) since 23.0:, will both ways be kept?

Thank you for the review! Correct, I argue both should be kept at least for now, and that's how it's currently implemented. I'm open to suggestions for better phrasing, but typically deprecated means a feature is no longer developed and may be removed in the future, but is still available. See e.g. Microsoft's definition. Does that clear things up?

@jsarenik
Copy link

@stickies-v instead of deprecating the old behavior the new can be just called backwards-compatible. By the way, an example of an externalized backward-compatible REST-service (in a way similar to what you have here) using Caddy webserver can be seen at https://github.com/jsarenik/bitcoin-faucet-shell/blob/master/Caddyfile.txt

Copy link

@jsarenik jsarenik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK 0415aaa

@stickies-v
Copy link
Contributor Author

@stickies-v instead of deprecating the old behavior the new can be just called backwards-compatible.

We can, but I'm not convinced we should? I think deprecating the old endpoints has a clear benefit of keeping the documented interface simpler and better adhering to common REST API practices. Simultaneously, since they're not removed, existing users are not affected, just gently nudged towards the new interface if and when it suits them.
Thank you for your review!

Copy link
Contributor

@promag promag left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see a strong reason for this change TBH.

@@ -47,18 +47,24 @@ The HTTP request and response are both handled entirely in-memory.
With the /notxdetails/ option JSON response will only contain the transaction hash instead of the complete transaction details. The option only affects the JSON response.

#### Blockheaders
`GET /rest/headers/<COUNT>/<BLOCK-HASH>.<bin|hex|json>`
`GET /rest/headers/<BLOCK-HASH>.<bin|hex|json>?count=<COUNT=5>`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why only count as a query parameter? How about

GET /rest/headers?block_hash=&count=&format=

block_hash defaults to tip
count defaults to 5
format defaults to request accept header, JSON if not specified

Copy link
Contributor Author

@stickies-v stickies-v Jan 24, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree and considered that, but decided to omit this for now to keep this PR's scope limited. Would you be okay with this being incorporated in a follow up PR (which I'm happy to take on right away?)

Edit: after reading jnewbery's comment, I realized I didn't see you also converted block_hash into a query parameter, which I don't agree with as per his arguments. I still agree with your suggestion for format.

Edit 2: after reading your latest comment, I realize I did not fully think this through. I have some further thoughts but will hold off on sharing these until after the PR review club to encourage open discussion.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To conclude, I agree with promag that /headers/ and /blockfilterheaders/ are collection endpoints since we're not obtaining a resource identified by the specified block_hash but rather use it as a pagination parameter. I didn't properly realise this at first.

Given that 1) this PR has had quite a bit of review time already which I'd rather not overhaul too much and 2) these and the other updates (moving format and notxdetails to a query parameter) make sense to implement together, I'll implement this in a follow up PR (I've already briefly started but it will take a bit more time as I'm thinking about ways to minimise the maintenance burden of keeping backwards compatibility).

Copy link
Contributor

@fjahr fjahr Mar 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW, I agree with @promag that this would be more properly following REST best practices, which seems to be the main point of this PR.

Do I understand correctly that you want to keep maintaining the old structure, this structure and then also future structures like suggested above @stickies-v ? That does seem more laborious than necessary to me. You could merge these first parts of changes as intermediary steps and then add follow-ups but I would not "officially" support it by adding it to docs and release notes yet. You could make the follow-up changes and only afterwards add documentation and release notes when everything is final, so that there are at least only two versions to maintain and not one per follow-up PR since without docs and release notes it can be reasonably assumed that nobody is building a client on top of the intermediary version.

Alternatively if you really want to include docs I would suggest in this intermediate step instead of marking the old structure as deprecated, add the new structure marked as experimental and add a comment that it is still object to change.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think your approach makes a lot of sense, fjahr. However, since there has already been significant review time spent on this and this PR seems ready for merge, I don't think this is big enough of a problem to ask for another round of review. Even if it's not a huge update, it still would touch quite a few commits (dropping release notes, modifying REST interface docs, updating error return messages and inline docs). I'll definitely consider the approach you suggest for further PRs because I think it is a more careful approach, but for now I'd prefer to leave things as they are and move forward.

@stickies-v stickies-v force-pushed the rest/use-query-parameters branch from 0415aaa to e1822e7 Compare January 24, 2022 15:14
@stickies-v
Copy link
Contributor Author

Rebased and:

Thank you @promag for your review. To address your comment:

I don't see a strong reason for this change TBH

I agree this is not a critical issue. However, I think everyone can agree my proposal is the more standard way of organizing API endpoints. If that's true (and so far no one argued the opposite), it's better to refactor this earlier rather than later, because for example in #17631 we (I think rightfully) preferred internal consistency over adhering to best practices. We now also have the functionality in place for other existing or new endpoints to easily accept query parameters, including your proposal for the data format.

Note: I've just learned I should only rebase in case of merge conflicts. My apologies for the added review difficulty, won't happen going forward.

Copy link
Contributor

@jnewbery jnewbery left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK. A few minor comments inline.

Why only count as a query parameter? How about

GET /rest/headers?block_hash=&count=&format=

block_hash defaults to tip
count defaults to 5
format defaults to request accept header, JSON if not specified

Each /headers/block is a unique resource, so I think it makes sense to keep the block hash as part of the URI path. Making format a query string key seems ok to me.

I also think that the /rest/block/notxdetails/ endpoint could be deprecated and merged into /rest/block/<BLOCKHASH>?txdetails=true|false or similar in a follow up.

@stickies-v stickies-v force-pushed the rest/use-query-parameters branch 2 times, most recently from 8868896 to 9d9f177 Compare January 28, 2022 02:48
@stickies-v
Copy link
Contributor Author

Thank you for the extensive review jnewbery, I've incorporated all of your comments into the latest force push 9d9f177.

Each /headers/block is a unique resource, so I think it makes sense to keep the block hash as part of the URI path. Making format a query string key seems ok to me.

I also think that the /rest/block/notxdetails/ endpoint could be deprecated and merged into /rest/block/<BLOCKHASH>?txdetails=true|false or similar in a follow up.

I agree with both of your comments, and am happy to take them on in a follow up PR. Since all of the query parameter logic is already in place now, this should be quite limited in scope too.

@stickies-v stickies-v force-pushed the rest/use-query-parameters branch from 25cd29e to fe95d30 Compare March 2, 2022 16:50
Copy link
Contributor

@theStack theStack left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK fe95d30 🌱

Copy link
Contributor

@jnewbery jnewbery left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nearly ACK fe95d30

I've left a few minor comments. The only one that I'd definitely like to see resolved before merge is the build failure on the first two commits.

This facilitates unit testing
As RetFormat is now exposed in a header, it is renamed to the more
understandable RESTResponseFormat
-BEGIN VERIFY SCRIPT-
s() { sed -i 's/RetFormat/RESTResponseFormat/g' $1; }
s src/rest.cpp
s src/rest.h
-END VERIFY SCRIPT-
URLs may contain a query string (prefixed with '?') and this should be ignored when parsing
the data format.

To facilitate testing this functionality, ParseDataFormat has been made non-static.
Easily get the query parameter from the URI, with optional default value.
@stickies-v stickies-v force-pushed the rest/use-query-parameters branch from fe95d30 to 9aac438 Compare March 10, 2022 11:32
@stickies-v
Copy link
Contributor Author

Force pushed from fe95d30 to 9aac438 to address jnewbery's comments, namely not each commit being able to compile and several style updates regarding spaces and brace initialization. There are no functional/behaviour changes.

The range-diff (git range-diff 219d728 fe95d30 9aac438) is a bit noisy unfortunately, you can also compare the commits individually:

git diff 1cc759af212c91807735e6c9bc2c998468c7252a..9f1c54787c81177dd56a31c881a9ad2834a122dc
git diff 40830ae7ea72c6ac6082bbfd30d26d1f749c46df..c1aad1b3b95b7c6bdf05e0c2095aba2f2db8310b
git diff 2c84d3916b710fbf6029858acff941549e875cb7..fff771ee864975cee8c831651239bac95503c37a
git diff ac3c007a393d3493ad9023c71d4ed3c38a8baabd..a09497614e9bb603fff36286d9611a25b23eeb02
git diff f610294c449aefc4bdae9175292fedbe5dd1d5f5..46af138c216d5fa7ac512a42c8219706e97d88fa
git diff fe95d30ae4ffd1946823ec3e2d6a59c063d62893..9aac4381897130ce630c4c3f5fca6690cd148330

@jnewbery
Copy link
Contributor

Code review ACK 9aac438

Tested range-diff and that all commits build.

Copy link
Contributor

@theStack theStack left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re-ACK 9aac438

In most RESTful APIs, path parameters are used to represent resources, and
query parameters are used to control how these resources are being filtered/sorted/...

The old /<count>/ functionality is kept alive to maintain backwards compatibility,
but new paths with query parameters are introduced and documented as the default
interface so future API methods don't break consistency by using query parameters.
@stickies-v stickies-v force-pushed the rest/use-query-parameters branch from 9aac438 to 54b39cf Compare April 5, 2022 17:25
@stickies-v
Copy link
Contributor Author

I've had to push a tiny doc update to REST-interface.md (git range-diff 219d728 9aac438 54b39cf) since this was not merged for v23, but since there are no significant changes beyond theStack and jnewbery's ACKs I think this PR is now ready to be considered for merging? @MarcoFalke

git range-diff 219d728 9aac438 54b39cf
1:  9f1c54787 = 1:  9f1c54787 Refactoring: move declarations to rest.h
2:  c1aad1b3b = 2:  c1aad1b3b scripted-diff: rename RetFormat to RESTResponseFormat
3:  fff771ee8 = 3:  fff771ee8 Handle query string when parsing data format
4:  a09497614 = 4:  a09497614 Add GetQueryParameter helper function
5:  46af138c2 ! 5:  f959fc039 Update /<count>/ endpoints to use a '?count=' query parameter instead
    @@ doc/REST-interface.md: The HTTP request and response are both handled entirely i
      Given a block hash: returns <COUNT> amount of blockheaders in upward direction.
      Returns empty if the block doesn't exist or it isn't in the active chain.
      
    -+*Deprecated (but not removed) since 23.0:*
    ++*Deprecated (but not removed) since 24.0:*
     +`GET /rest/headers/<COUNT>/<BLOCK-HASH>.<bin|hex|json>`
     +
      #### Blockfilter Headers
    @@ doc/REST-interface.md: The HTTP request and response are both handled entirely i
      direction for the filter type <FILTERTYPE>.
      Returns empty if the block doesn't exist or it isn't in the active chain.
      
    -+*Deprecated (but not removed) since 23.0:*
    ++*Deprecated (but not removed) since 24.0:*
     +`GET /rest/blockfilterheaders/<FILTERTYPE>/<COUNT>/<BLOCK-HASH>.<bin|hex|json>`
     +
      #### Blockfilters
6:  9aac43818 = 6:  54b39cfb3 Add release notes

@maflcko maflcko requested review from theStack and jnewbery April 5, 2022 18:34
Copy link
Contributor

@theStack theStack left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re-ACK 54b39cf

Verified that the only change since my last ACK was bumping the version number in the REST documentation from v23 to v24 (since this PR didn't make it into v23, v24 will be the first release where the old endpoints are deprecated).

@jnewbery
Copy link
Contributor

jnewbery commented Apr 5, 2022

ACK 54b39cf

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.