Skip to content

Conversation

S3RK
Copy link
Contributor

@S3RK S3RK commented Jun 3, 2020

Rationale: Verify mempool.dat compatibility between versions

The format of mempool.dat has been changed in #18038
The tests verifies the fix made in #18807 and ensures that the file format is compatible between current version and v0.19.1
The test verifies both backward and forward compatibility.

This PR also adds a log when we fail to add a tx loaded from mempool.dat.
It was useful when debugging this test and could be potentially useful to debug other scenarios as well.

Closes #19037

@fanquake fanquake added the Tests label Jun 3, 2020
@S3RK S3RK changed the title Feature/mempool compatibility test test: mempool compatibility test Jun 3, 2020
@S3RK S3RK force-pushed the feature/mempool_compatibility_test branch from 7c24bae to fe4a271 Compare June 3, 2020 10:27
Copy link
Member

@maflcko maflcko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice first contribution! ACK with or without the nit addressed

@practicalswift
Copy link
Contributor

Concept ACK

What a nice first-time contribution!

Warm welcome as a contributor @S3RK - I hope to see more great contributions from you in the future :)

@S3RK S3RK force-pushed the feature/mempool_compatibility_test branch from fe4a271 to cd58338 Compare June 4, 2020 08:09
@laanwj
Copy link
Member

laanwj commented Jun 4, 2020

Concept ACK. I don't think we can always guarantee that the mempool file will be backwards compatible, but it's good to have a test to be sure that compatibility at least isn't accidentally broken.

Copy link
Member

@Sjors Sjors left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK. f0b2579 looks good modulo some comments.

Suggest making a separate PR for cd58338, ideally with a test.

"""Test that mempool.dat is compatible between versions

Download node binaries:
contrib/devtools/previous_release.sh -b v0.19.1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I prefer to have the same instruction in all test files that require previous releases, because otherwise other tests fail when they have some versions but not all. Try:

contrib/devtools/previous_release.sh -b v0.19.1 v0.18.1 v0.17.1 v0.16.3 v0.15.2

Only v0.19.1 is required by this test. The rest is used in other backwards compatibility tests.


def setup_nodes(self):
self.add_nodes(self.num_nodes, versions=[
190100,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest documenting what's interesting about that version, could be a link to Github issue.

# Copyright (c) 2017-2020 The Bitcoin Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
"""Test that mempool.dat is compatible between versions
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest adding a comment like what @laanwj said:

I don't think we can always guarantee that the mempool file will be backwards compatible, but it's good to have a test to be sure that compatibility at least isn't accidentally broken.

@S3RK
Copy link
Contributor Author

S3RK commented Jun 9, 2020

@Sjors thanks for the review. I've pushed a fixup commit to address your suggestions. Once the PR is ready to be merged I will squash it into the original commit.

Suggest making a separate PR for cd58338, ideally with a test.

I'm not sure this tiny change deserves a PR and a separate discussion on its own 😄. To clarify the impact to log chattiness, the code path in question is executed only during initial mempool load, so it won't repeatedly produce tons of logs. But maybe I'm failing to consider some other possible negative outcomes.

@Sjors
Copy link
Member

Sjors commented Jun 9, 2020

The new changes look good.

Log chattiness could be one concern, e.g. when loading a 1 GB mempool with broken entries, but also there's no test. Touching validation.cpp generally instills fear :-)

@S3RK S3RK force-pushed the feature/mempool_compatibility_test branch from aec13a6 to 16d4b3f Compare June 10, 2020 04:06
@S3RK
Copy link
Contributor Author

S3RK commented Jun 10, 2020

Squashed the changes, removed log commit

150200, # oldest version supported by the test framework
None,
])
adjust_bitcoin_conf_for_pre_17(self.nodes[0].bitcoinconf)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like other instances of adjust_bitcoin_conf_for_pre_17 have left comments as to why it's used. I think it'd be helpful to include here as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you believe the code is unclear, we need to think of a better way to fix it rather than putting the same comment before each function call. For example: give this function a better name. But as for me, I think the function name speaks for itself and it doesn't need any clarifying comment.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's fine to call this without comment; it's always necessary for nodes before 0.17. Ideally the test framework (add_nodes) takes care of it automagically.

@S3RK S3RK requested a review from Sjors June 16, 2020 09:56
Copy link
Member

@Sjors Sjors left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tACK 16d4b3f

P.S. for increased glory you could sign your commits

150200, # oldest version supported by the test framework
None,
])
adjust_bitcoin_conf_for_pre_17(self.nodes[0].bitcoinconf)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's fine to call this without comment; it's always necessary for nodes before 0.17. Ideally the test framework (add_nodes) takes care of it automagically.

@maflcko maflcko merged commit 0afbeb7 into bitcoin:master Jun 16, 2020
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Jul 7, 2020
16d4b3f test: mempool.dat compatibility between versions (Ivan Metlushko)

Pull request description:

  Rationale: Verify mempool.dat compatibility between versions

  The format of mempool.dat has been changed in bitcoin#18038
  The tests verifies the fix made in bitcoin#18807 and ensures that the file format is compatible between current version and v0.19.1
  The test verifies both backward and forward compatibility.

  This PR also adds a log when we fail to add a tx loaded from mempool.dat.
  It was useful when debugging this test and could be potentially useful to debug other scenarios as well.

  Closes bitcoin#19037

ACKs for top commit:
  Sjors:
    tACK 16d4b3f

Tree-SHA512: 00a38bf528c6478cb0da467af216488f83c1e3ca4d9166c109202ea8284023e99d87a3d6e252c4d88d08d9b5ed1a730b3e1970d6e5c0aef526fa7ced40de7490
@S3RK S3RK deleted the feature/mempool_compatibility_test branch September 25, 2020 02:07
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Feb 15, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Write compatibility tests for mempool.dat
7 participants