Skip to content

Conversation

sipa
Copy link
Member

@sipa sipa commented Jun 8, 2016

There are 3 ways through which main.cpp (and by extension, the getrawtransaction and REST tx lookup) find transactions:

  • In the mempool (works always)
  • In the block index (works only when -txindex is enabled).
  • In the UTXO set (inefficient, and only for confirmed transactions that have some of their outputs not yet spent by other confirmed transactions)

This third option is confusing because the conditions under which it works are complicated, and can make it seem like it always works even without -txindex. Remove it. If you need to call getrawtransaction for confirmed transactions, you almost certainly need -txindex.

This only impacts the getrawtransaction and REST tx calls, so document those.

@laanwj
Copy link
Member

laanwj commented Jun 8, 2016

What is the migration path for users that were using this to query the utxo set?
I suppose gettxout and REST /rest/getutxos cover that?

@sipa
Copy link
Member Author

sipa commented Jun 8, 2016

Yes, use UTXO query functions to query the UTXO set.

@laanwj
Copy link
Member

laanwj commented Jun 9, 2016

This definitely needs mention in the release notes.
Maybe deprecate this functionality in 0.13 (put it behind an option that is disabled by default) then actually remove it in 0.14?

@laanwj
Copy link
Member

laanwj commented Jun 9, 2016

abandonconflict.py:
Initializing test directory /tmp/testvmbco_b4/4
start_node: bitcoind started, waiting for RPC to come up
start_node: RPC succesfully started
start_node: bitcoind started, waiting for RPC to come up
start_node: RPC succesfully started
JSONRPC error: No information available about transaction. Use `gettransaction` for wallet transactions, or enable -txindex to use getrawtransaction on confirmed transactions.
Stopping nodes
Not cleaning up dir /tmp/testvmbco_b4/4
Failed
stderr:
   File "/home/travis/build/bitcoin/bitcoin/bitcoin-i686-pc-linux-gnu/qa/rpc-tests/test_framework/test_framework.py", line 144, in main
    self.run_test()
  File "/home/travis/build/bitcoin/bitcoin/bitcoin-i686-pc-linux-gnu/qa/rpc-tests/abandonconflict.py", line 42, in run_test
    nA = next(i for i, vout in enumerate(self.nodes[0].getrawtransaction(txA, 1)["vout"]) if vout["value"] == Decimal("10"))
  File "/home/travis/build/bitcoin/bitcoin/bitcoin-i686-pc-linux-gnu/qa/rpc-tests/test_framework/coverage.py", line 49, in __call__
    return_val = self.auth_service_proxy_instance.__call__(*args, **kwargs)
  File "/home/travis/build/bitcoin/bitcoin/bitcoin-i686-pc-linux-gnu/qa/rpc-tests/test_framework/authproxy.py", line 144, in __call__
    raise JSONRPCException(response['error'])
Pass: False, Duration: 6 s

@dcousens
Copy link
Contributor

dcousens commented Jun 9, 2016

concept ACK

@gmaxwell
Copy link
Contributor

gmaxwell commented Jun 9, 2016

I agree that it's so incomprehensible (and not desirable) in its behavior that it's unlikely anyone was using it (or at least, using it and not silently getting screwed over by it). I do wish any of these queries that only worked with txindex had a non-txindex mechanism that took a location parameter (e.g. block height.)

@maflcko
Copy link
Member

maflcko commented Jun 9, 2016

I don't think we need to deprecate this behind a command line flag. (Keep in mind that there still would be no way for us to find out how many would toggle the flag)

Considering that this is not really a functionality someone can rely on and it appears unlikely someone (other than our test framework) relies on it, it is probably safe to just remove it.

@laanwj
Copy link
Member

laanwj commented Jun 9, 2016

I don't think we need to deprecate this behind a command line flag. (Keep in mind that there still would be no way for us to find out how many would toggle the flag)

That's true, but it would give people the opportunity to complain about it, while not having to forego the upgrade.

I don't feel strongly about it though, if you guys believe that this is a change that can be made without warning in advance I'm fine with that. Depends on how urgent this change is as well.

I do wish any of these queries that only worked with txindex had a non-txindex mechanism that took a location parameter (e.g. block height.)

One radical idea would be to expose CDiskTxPos fields on the interface here and there, and add a REST/RPC call to get a transaction from such a (blockfile_id,offset) tuple. I wonder if that would make some things easier/possible, such as external indices without having to store the block chain twice, but with fast access. Probably these 'pointers' should be obfuscated somehow to discourage direct access to the block files from outside bitcoind.

@sipa
Copy link
Member Author

sipa commented Jun 9, 2016

Depends on how urgent this change is as well.

The changes here are not urgent. If we choose to not adopt them quickly, I think at least the updated error message in getrawtransaction is useful.

@petertodd
Copy link
Contributor

One of my clients has something that was using this FWIW, so I'd target actual removal for v0.14

@laanwj
Copy link
Member

laanwj commented Jun 16, 2016

One of my clients has something that was using this FWIW, so I'd target actual removal for v0.14

That's interesting. I somehow expected that, if some functionality is available for so long, some people will come to rely on it.
What is their use case? Could they (in principle, given warning upfront to update their code) use the other UTXO query commands as replacement?

@petertodd
Copy link
Contributor

@laanwj Basically their wallet code happens to depend on it; with a fair amount of new code they could work around the problem (remember that simply saving the txs when created isn't enough due to malleability).

I don't think that's a show-stopper, but we should give users some kind of warning period. Perhaps we could make v0.13.0 appear to not have this support by default, but then re-enable it with a command line flag, that's marked as "will be removed in v0.14.0"?

@laanwj
Copy link
Member

laanwj commented Jun 18, 2016

Perhaps we could make v0.13.0 appear to not have this support by default, but then re-enable it with a command line flag, that's marked as "will be removed in v0.14.0"?

+1. That's exactly what I suggested too:

Maybe deprecate this functionality in 0.13 (put it behind an option that is disabled by default) then actually remove it in 0.14?

@petertodd
Copy link
Contributor

@laanwj Cool, sounds like we're in agreement. :)

@sipa
Copy link
Member Author

sipa commented Jun 18, 2016 via email

@petertodd
Copy link
Contributor

@sipa Sounds good to me, thanks!

@laanwj
Copy link
Member

laanwj commented Aug 2, 2016

Looks like this causes a few RPC tests to fail:

wallet.py                      | False  | 9 s
rest.py                        | False  | 7 s
abandonconflict.py             | False  | 7 s

@dcousens
Copy link
Contributor

dcousens commented Aug 2, 2016

utACK c3af143, but needs the command line option (or maybe just a short-term compile flag?)

@laanwj
Copy link
Member

laanwj commented Oct 18, 2016

Still failing travis.

@gmaxwell
Copy link
Contributor

gmaxwell commented Dec 4, 2016

@sipa Needs rebase.

@morcos
Copy link
Contributor

morcos commented Dec 5, 2016

I'm ever so slightly against this change. I find it's routinely useful for me when I'm trying to debug behavior on a random node.

@sipa
Copy link
Member Author

sipa commented Dec 5, 2016

@morcos Perhaps that's a good reason to introduce a getmempooltransaction RPC? My reason for this (or a related) change is that getrawtransaction has very unclear conditions under which it works, and I keep seeing people say it doesn't work.

@laanwj
Copy link
Member

laanwj commented Dec 21, 2016

Perhaps that's a good reason to introduce a getmempooltransaction RPC

Agree with @morcos that the way toward fixing this could also be make the API clearer instead of removing functionality (that is apparently used by some people sometimes, even if only by developers for troubleshooting).

+1 for splitting up getrawtransaction into calls that look in one place only.

@sipa sipa closed this Jan 11, 2017
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants