-
Notifications
You must be signed in to change notification settings - Fork 37.7k
rest: Use query parameters to control resource loading #24098
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
bdea37c
to
a857479
Compare
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
a857479
to
f861b6d
Compare
Force pushed to rebase after #24054 was merged; no other changes. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Concept ACK
Tested ?count=1, ?count=5 and ?count=600 (didn't paste here the result of it, obviously haha) for /rest/headers/
http://127.0.0.1:8332/rest/headers/000000001aeae195809d120b5d66a39c83eb48792e068f8ea1fea19d84a4278a.json?count=1
[{"hash":"000000001aeae195809d120b5d66a39c83eb48792e068f8ea1fea19d84a4278a","confirmations":237003,"height":50000,"version":1,"versionHex":"00000001","merkleroot":"27f1d66f8a1ee5280f4e92508dcb647e954d53004905d08a75574daee4988360","time":1270916538,"mediantime":1270915355,"nonce":58489010,"bits":"1c2a1115","difficulty":6.085476906000794,"chainwork":"00000000000000000000000000000000000000000000000000014a426593e15b","nTx":1,"previousblockhash":"000000000845517b31c6820d83f25cff46429bf136a7515fe504116427e60f8e","nextblockhash":"000000001c920d495e1eeef2452b6d1c6c229a919b28196c103ecffebabee141"}]
http://127.0.0.1:8332/rest/headers/000000001aeae195809d120b5d66a39c83eb48792e068f8ea1fea19d84a4278a.json?count=5
[{"hash":"000000001aeae195809d120b5d66a39c83eb48792e068f8ea1fea19d84a4278a","confirmations":240714,"height":50000,"version":1,"versionHex":"00000001","merkleroot":"27f1d66f8a1ee5280f4e92508dcb647e954d53004905d08a75574daee4988360","time":1270916538,"mediantime":1270915355,"nonce":58489010,"bits":"1c2a1115","difficulty":6.085476906000794,"chainwork":"00000000000000000000000000000000000000000000000000014a426593e15b","nTx":1,"previousblockhash":"000000000845517b31c6820d83f25cff46429bf136a7515fe504116427e60f8e","nextblockhash":"000000001c920d495e1eeef2452b6d1c6c229a919b28196c103ecffebabee141"},{"hash":"000000001c920d495e1eeef2452b6d1c6c229a919b28196c103ecffebabee141","confirmations":240713,"height":50001,"version":1,"versionHex":"00000001","merkleroot":"ee3a2d2b895cafacff526d06a55b55e049cf84a9735e4a63f7fd08f96d0f4649","time":1270917100,"mediantime":1270915522,"nonce":56717043,"bits":"1c2a1115","difficulty":6.085476906000794,"chainwork":"00000000000000000000000000000000000000000000000000014a487b7bc7c6","nTx":3,"previousblockhash":"000000001aeae195809d120b5d66a39c83eb48792e068f8ea1fea19d84a4278a","nextblockhash":"000000002066d7f9b134c30b5005b7bf5fbfa52f279883f1fde793dcdc964266"},{"hash":"000000002066d7f9b134c30b5005b7bf5fbfa52f279883f1fde793dcdc964266","confirmations":240712,"height":50002,"version":1,"versionHex":"00000001","merkleroot":"5164dc2785549f9efe14eb1c54522ec1874a02b7eda164fde370c05412f037ad","time":1270917181,"mediantime":1270915874,"nonce":9804838,"bits":"1c2a1115","difficulty":6.085476906000794,"chainwork":"00000000000000000000000000000000000000000000000000014a4e9163ae31","nTx":1,"previousblockhash":"000000001c920d495e1eeef2452b6d1c6c229a919b28196c103ecffebabee141","nextblockhash":"00000000237ce1b692f5ef9f1f9ac64d62d5e6a24dcfec08633fa2d339434e2a"},{"hash":"00000000237ce1b692f5ef9f1f9ac64d62d5e6a24dcfec08633fa2d339434e2a","confirmations":240711,"height":50003,"version":1,"versionHex":"00000001","merkleroot":"792c4a613e51966fc1c852cd8d94fd10a05efa55fffdd6a41281c8d139540b23","time":1270917916,"mediantime":1270916181,"nonce":311792473,"bits":"1c2a1115","difficulty":6.085476906000794,"chainwork":"00000000000000000000000000000000000000000000000000014a54a74b949c","nTx":1,"previousblockhash":"000000002066d7f9b134c30b5005b7bf5fbfa52f279883f1fde793dcdc964266","nextblockhash":"000000000528e8aac90e3096abb7b1f3d52efa2cd8f12754614788cb43eefe11"},{"hash":"000000000528e8aac90e3096abb7b1f3d52efa2cd8f12754614788cb43eefe11","confirmations":240710,"height":50004,"version":1,"versionHex":"00000001","merkleroot":"3cfbd9a91f033e37f05764215626c542c735ed71cd5b53467834cf881c0b9ebd","time":1270918264,"mediantime":1270916354,"nonce":78553962,"bits":"1c2a1115","difficulty":6.085476906000794,"chainwork":"00000000000000000000000000000000000000000000000000014a5abd337b07","nTx":1,"previousblockhash":"00000000237ce1b692f5ef9f1f9ac64d62d5e6a24dcfec08633fa2d339434e2a","nextblockhash":"0000000000e1ad737f11fe4cb3d126408b2cbcc28ab0d83293f408e413733338"}]
I tested the same values but using /rest/headers/<COUNT>/<BLOCK-HASH>.<bin|hex|json>
format and it worked as well. I didn't understand Deprecated (but not removed) since 23.0:
, will both ways be kept?
Is there a large cost to maintaining both ways? If not, it might be better to keep both to not upset users for a weak motivation. |
f861b6d
to
0415aaa
Compare
The PR is already setup to maintain both ways, so anyone using the (to be) deprecated endpoints should not be affected. This is documented in REST-interface.md too, so users are aware it is still supported but that they should probably use the new endpoint for new applications. The maintenance overhead is minimal, basically in Personally, I would definitely not remove the old endpoints in the next release, and probably not until some bigger API refactoring would make this desirable, in which case we should notify users more explicitly. I did not add any deprecation logging to avoid behaviour change, and because I think we don't have to push users away from the old endpoints yet. |
Thank you for the review! Correct, I argue both should be kept at least for now, and that's how it's currently implemented. I'm open to suggestions for better phrasing, but typically deprecated means a feature is no longer developed and may be removed in the future, but is still available. See e.g. Microsoft's definition. Does that clear things up? |
@stickies-v instead of deprecating the old behavior the new can be just called backwards-compatible. By the way, an example of an externalized backward-compatible REST-service (in a way similar to what you have here) using Caddy webserver can be seen at https://github.com/jsarenik/bitcoin-faucet-shell/blob/master/Caddyfile.txt |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
utACK 0415aaa
We can, but I'm not convinced we should? I think deprecating the old endpoints has a clear benefit of keeping the documented interface simpler and better adhering to common REST API practices. Simultaneously, since they're not removed, existing users are not affected, just gently nudged towards the new interface if and when it suits them. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't see a strong reason for this change TBH.
@@ -47,18 +47,24 @@ The HTTP request and response are both handled entirely in-memory. | |||
With the /notxdetails/ option JSON response will only contain the transaction hash instead of the complete transaction details. The option only affects the JSON response. | |||
|
|||
#### Blockheaders | |||
`GET /rest/headers/<COUNT>/<BLOCK-HASH>.<bin|hex|json>` | |||
`GET /rest/headers/<BLOCK-HASH>.<bin|hex|json>?count=<COUNT=5>` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why only count
as a query parameter? How about
GET /rest/headers?block_hash=&count=&format=
block_hash defaults to tip
count defaults to 5
format defaults to request accept header, JSON if not specified
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree and considered that, but decided to omit this for now to keep this PR's scope limited. Would you be okay with this being incorporated in a follow up PR (which I'm happy to take on right away?)
Edit: after reading jnewbery's comment, I realized I didn't see you also converted block_hash
into a query parameter, which I don't agree with as per his arguments. I still agree with your suggestion for format
.
Edit 2: after reading your latest comment, I realize I did not fully think this through. I have some further thoughts but will hold off on sharing these until after the PR review club to encourage open discussion.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To conclude, I agree with promag that /headers/
and /blockfilterheaders/
are collection endpoints since we're not obtaining a resource identified by the specified block_hash
but rather use it as a pagination parameter. I didn't properly realise this at first.
Given that 1) this PR has had quite a bit of review time already which I'd rather not overhaul too much and 2) these and the other updates (moving format
and notxdetails
to a query parameter) make sense to implement together, I'll implement this in a follow up PR (I've already briefly started but it will take a bit more time as I'm thinking about ways to minimise the maintenance burden of keeping backwards compatibility).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FWIW, I agree with @promag that this would be more properly following REST best practices, which seems to be the main point of this PR.
Do I understand correctly that you want to keep maintaining the old structure, this structure and then also future structures like suggested above @stickies-v ? That does seem more laborious than necessary to me. You could merge these first parts of changes as intermediary steps and then add follow-ups but I would not "officially" support it by adding it to docs and release notes yet. You could make the follow-up changes and only afterwards add documentation and release notes when everything is final, so that there are at least only two versions to maintain and not one per follow-up PR since without docs and release notes it can be reasonably assumed that nobody is building a client on top of the intermediary version.
Alternatively if you really want to include docs I would suggest in this intermediate step instead of marking the old structure as deprecated, add the new structure marked as experimental and add a comment that it is still object to change.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think your approach makes a lot of sense, fjahr. However, since there has already been significant review time spent on this and this PR seems ready for merge, I don't think this is big enough of a problem to ask for another round of review. Even if it's not a huge update, it still would touch quite a few commits (dropping release notes, modifying REST interface docs, updating error return messages and inline docs). I'll definitely consider the approach you suggest for further PRs because I think it is a more careful approach, but for now I'd prefer to leave things as they are and move forward.
0415aaa
to
e1822e7
Compare
Rebased and:
Thank you @promag for your review. To address your comment:
I agree this is not a critical issue. However, I think everyone can agree my proposal is the more standard way of organizing API endpoints. If that's true (and so far no one argued the opposite), it's better to refactor this earlier rather than later, because for example in #17631 we (I think rightfully) preferred internal consistency over adhering to best practices. We now also have the functionality in place for other existing or new endpoints to easily accept query parameters, including your proposal for the data format. Note: I've just learned I should only rebase in case of merge conflicts. My apologies for the added review difficulty, won't happen going forward. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Concept ACK. A few minor comments inline.
Why only count as a query parameter? How about
GET /rest/headers?block_hash=&count=&format=
block_hash defaults to tip
count defaults to 5
format defaults to request accept header, JSON if not specified
Each /headers/block
is a unique resource, so I think it makes sense to keep the block hash as part of the URI path. Making format
a query string key seems ok to me.
I also think that the /rest/block/notxdetails/
endpoint could be deprecated and merged into /rest/block/<BLOCKHASH>?txdetails=true|false
or similar in a follow up.
8868896
to
9d9f177
Compare
Thank you for the extensive review jnewbery, I've incorporated all of your comments into the latest force push 9d9f177.
I agree with both of your comments, and am happy to take them on in a follow up PR. Since all of the query parameter logic is already in place now, this should be quite limited in scope too. |
25cd29e
to
fe95d30
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ACK fe95d30 🌱
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nearly ACK fe95d30
I've left a few minor comments. The only one that I'd definitely like to see resolved before merge is the build failure on the first two commits.
This facilitates unit testing
As RetFormat is now exposed in a header, it is renamed to the more understandable RESTResponseFormat -BEGIN VERIFY SCRIPT- s() { sed -i 's/RetFormat/RESTResponseFormat/g' $1; } s src/rest.cpp s src/rest.h -END VERIFY SCRIPT-
URLs may contain a query string (prefixed with '?') and this should be ignored when parsing the data format. To facilitate testing this functionality, ParseDataFormat has been made non-static.
Easily get the query parameter from the URI, with optional default value.
fe95d30
to
9aac438
Compare
Force pushed from fe95d30 to 9aac438 to address jnewbery's comments, namely not each commit being able to compile and several style updates regarding spaces and brace initialization. There are no functional/behaviour changes. The range-diff (
|
Code review ACK 9aac438 Tested range-diff and that all commits build. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re-ACK 9aac438
In most RESTful APIs, path parameters are used to represent resources, and query parameters are used to control how these resources are being filtered/sorted/... The old /<count>/ functionality is kept alive to maintain backwards compatibility, but new paths with query parameters are introduced and documented as the default interface so future API methods don't break consistency by using query parameters.
9aac438
to
54b39cf
Compare
I've had to push a tiny doc update to git range-diff 219d728 9aac438 54b39cf
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re-ACK 54b39cf
Verified that the only change since my last ACK was bumping the version number in the REST documentation from v23 to v24 (since this PR didn't make it into v23, v24 will be the first release where the old endpoints are deprecated).
ACK 54b39cf |
In RESTful APIs, typically path parameters (e.g.
/some/unique/resource/
) are used to represent resources, and query parameters (e.g.?sort=asc
) are used to control how these resources are being loaded through e.g. sorting, pagination, filtering, ...As first discussed in #17631, the current REST api contains two endpoints
/headers/
and/blockfilterheaders/
that rather unexpectedly use path parameters to control how many (filter) headers are returned in the response. While this is no critical issue, it is unintuitive and we are still early enough to easily phase this behaviour out and ensure new endpoints (if any) do not have to stick to non-standard behaviour just for internal consistency.In this PR, a new
HTTPRequest::GetQueryParameter
method is introduced to easily parse query parameters, as well as two new/headers/
and/blockfilterheaders/
endpoints that use a count query parameter are introduced. The old path parameter-based endpoints are kept without too much overhead, but the documentation now points to the new query parameter-based endpoints as the default interface to encourage standardness.Behaviour change
New endpoints and default values
/headers/
and/blockfilterheaders/
now have 2 new endpoints that contain query parameters (?count=<count>
) instead of path parameters (/<count>/
), as described in REST-interface.md. Since query parameters can easily have default values, I have set this at 5 for both endpoints.headers
GET /rest/headers/<BLOCK-HASH>.<bin|hex|json>?count=<COUNT=5>
should now be used instead of
GET /rest/headers/<COUNT>/<BLOCK-HASH>.<bin|hex|json>
blockfilterheaders
GET /rest/blockfilterheaders/<FILTERTYPE>/<BLOCK-HASH>.<bin|hex|json>?count=<COUNT=5>
should now be used instead of
GET /rest/blockfilterheaders/<FILTERTYPE>/<COUNT>/<BLOCK-HASH>.<bin|hex|json>
Some previously invalid API calls are now valid
API calls that contained query strings in the URI could not be parsed prior to this PR. This PR changes behaviour in that previously invalid calls (e.g.
GET /rest/headers/5/somehash.json?someunusedparam=foo
) would now become valid, as the query parameters are properly parsed, and discarded if unused.For example, prior to this PR, adding an irrelevant
someparam
parameter would be illegal:This behaviour change affects all rest endpoints, not just the 2 new ones introduced here.
(Note: I'd be open to implementing additional logic to refuse requests containing unrecognized query parameters to minimize behaviour change, but for the endpoints that we currently have I don't really see the point for that added complexity. E.g. I don't see any scenarios where misspelling a parameter could lead to harmful outcomes)
Using the REST API
To run the API HTTP server, start a bitcoind instance with the
-rest
flag enabled. To use theblockfilterheaders
endpoint, you'll also need to set-blockfilterindex=1
:As soon as bitcoind is fully up and running, you should be able to query the API, for example by
using curl on the command line:
curl "127.0.0.1:38332/rest/chaininfo.json"
.To more easily parse the JSON output, you can also use tools like 'jq' or
json_pp
, e.g.:To do
doc/release-notes
Feedback
This is my first PR (hooray!). Please don't hold back on any feedback/comments/nits/... you may have, big or small, whether they are code, process, language, ... related. I welcome private messages too if there's anything you don't want to clutter the PR with. I'm here to learn and am grateful for everyone's input.