Skip to content

Remove genesis persistence in state db #1017

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 7 commits into from
Aug 30, 2023
Merged

Remove genesis persistence in state db #1017

merged 7 commits into from
Aug 30, 2023

Conversation

drklee3
Copy link
Contributor

@drklee3 drklee3 commented Jun 22, 2023 •
edited by sergio-mena
Loading

This avoids saving the genesisDoc to database. When using goleveldb and ~4GiB+ genesis files, it causes a panic during snappy encoding (panic: snappy: decoded block is too large). We run into this with a large Ethermint state in our mainnet exports.

The main file to review is node/setup.go, with changes mostly from tendermint/tendermint#6554

Contributes to #1037

PR checklist

  • Tests written/updated
  • Changelog entry added in .changelog (we use unclog to manage our changelog)
  • Updated relevant documentation (docs/ or spec/) and code comments

@drklee3 drklee3 marked this pull request as ready for review June 22, 2023 18:50
@drklee3 drklee3 requested a review from a team as a code owner June 22, 2023 18:50
This was referenced Jun 23, 2023
Closed
Closed
@github-actions
Copy link

github-actions bot commented Jul 3, 2023

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale For use by stalebot label Jul 3, 2023
@sergio-mena sergio-mena removed the stale For use by stalebot label Jul 3, 2023
mzabaluev
mzabaluev reviewed Jul 4, 2023
View reviewed changes
mzabaluev
mzabaluev reviewed Jul 4, 2023
View reviewed changes
Copy link
Contributor

@mzabaluev mzabaluev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good quality code.
One concern voiced about this change is, we'd like to keep a hash of the genesis doc in the store for a sanity check.

mzabaluev
mzabaluev reviewed Jul 4, 2023
View reviewed changes
@jmalicevic
Copy link
Contributor

Hey @drklee3 , thanks a lot for doing this. As @mzabaluev above said there are a few more options we are considering adding to this solution (like the hash of the doc in the store etc.) , its impact on the retrieval of the genesis doc via RPC and whether it can be improved when we have a solution for storing the file like this. We will explore them and build on the solution this quarter though : see #1037 .

@drklee3 drklee3 requested a review from a team July 6, 2023 18:34
@drklee3 drklee3 mentioned this pull request Jul 12, 2023
Closed
@alpha-omega-labs
Copy link

We need that fix to be able to import large (~10GB) genesis.json is kava or cronos fork for GenesisL1 blockchain.
Thank you for addressing that issue, its very helpful!

@alpha-omega-labs alpha-omega-labs mentioned this pull request Jul 22, 2023
Closed
@github-actions
Copy link

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale For use by stalebot label Jul 26, 2023
@github-actions github-actions bot closed this Jul 30, 2023
@jmalicevic jmalicevic added wip Work in progress and removed stale For use by stalebot labels Jul 31, 2023
@jmalicevic
Copy link
Contributor

Re-opening this because it is waiting for us to take a look at it once ADR-101 PRs are merged.

@jmalicevic jmalicevic reopened this Jul 31, 2023
@jmalicevic jmalicevic added storage P:storage-optimization Priority: Give operators greater control over storage and storage optimization labels Jul 31, 2023
@troian troian mentioned this pull request Feb 5, 2025
Closed
troian added a commit to akash-network/cometbft that referenced this pull request Feb 5, 2025
@troian
refactor: backport remove genesis persistence from state
c995c54 
This avoids saving the genesisDoc to database.
When using goleveldb and ~4GiB+ genesis files, it causes a panic
during snappy encoding (panic: snappy: decoded block is too large).

refs akash-network/support#280

backports:
- cometbft#1017
- cometbft#1293

Signed-off-by: Artur Troian <troian.ap@gmail.com>
@troian troian mentioned this pull request Feb 6, 2025
Merged
troian added a commit to akash-network/cometbft that referenced this pull request Feb 6, 2025
@troian
refactor: backport remove genesis persistence from state
de27f10 
This avoids saving the genesisDoc to database.
When using goleveldb and ~4GiB+ genesis files, it causes a panic
during snappy encoding (panic: snappy: decoded block is too large).

refs akash-network/support#280

backports:
- cometbft#1017
- cometbft#1293

Signed-off-by: Artur Troian <troian.ap@gmail.com>
troian added a commit to akash-network/cometbft that referenced this pull request Feb 6, 2025
@troian
refactor: backport remove genesis persistence from state
b6d56ca 
This avoids saving the genesisDoc to database.
When using goleveldb and ~4GiB+ genesis files, it causes a panic
during snappy encoding (panic: snappy: decoded block is too large).

refs akash-network/support#280

backports:
- cometbft#1017
- cometbft#1293

Signed-off-by: Artur Troian <troian.ap@gmail.com>
troian added a commit to akash-network/cometbft that referenced this pull request Feb 6, 2025
@troian
refactor: backport remove genesis persistence from state
c474940 
This avoids saving the genesisDoc to database.
When using goleveldb and ~4GiB+ genesis files, it causes a panic
during snappy encoding (panic: snappy: decoded block is too large).

refs akash-network/support#280

backports:
- cometbft#1017
- cometbft#1293

Signed-off-by: Artur Troian <troian.ap@gmail.com>
troian added a commit to akash-network/cometbft that referenced this pull request Feb 6, 2025
@troian
refactor: backport remove genesis persistence from state
9f07441 
This avoids saving the genesisDoc to database.
When using goleveldb and ~4GiB+ genesis files, it causes a panic
during snappy encoding (panic: snappy: decoded block is too large).

refs akash-network/support#280

backports:
- cometbft#1017
- cometbft#1293

Signed-off-by: Artur Troian <troian.ap@gmail.com>
troian added a commit to akash-network/cometbft that referenced this pull request Feb 6, 2025
@troian
refactor: backport remove genesis persistence from state
eee4224 
This avoids saving the genesisDoc to database.
When using goleveldb and ~4GiB+ genesis files, it causes a panic
during snappy encoding (panic: snappy: decoded block is too large).

refs akash-network/support#280

backports:
- cometbft#1017
- cometbft#1293

Signed-off-by: Artur Troian <troian.ap@gmail.com>
troian added a commit to akash-network/cometbft that referenced this pull request Feb 6, 2025
@troian
refactor: backport remove genesis persistence from state
23fb733 
This avoids saving the genesisDoc to database.
When using goleveldb and ~4GiB+ genesis files, it causes a panic
during snappy encoding (panic: snappy: decoded block is too large).

refs akash-network/support#280

backports:
- cometbft#1017
- cometbft#1293

Signed-off-by: Artur Troian <troian.ap@gmail.com>
troian added a commit to akash-network/cometbft that referenced this pull request Feb 6, 2025
@troian
refactor: backport remove genesis persistence from state
01e2719 
This avoids saving the genesisDoc to database.
When using goleveldb and ~4GiB+ genesis files, it causes a panic
during snappy encoding (panic: snappy: decoded block is too large).

refs akash-network/support#280

backports:
- cometbft#1017
- cometbft#1293

Signed-off-by: Artur Troian <troian.ap@gmail.com>
troian added a commit to akash-network/cometbft that referenced this pull request Feb 6, 2025
@troian
refactor: backport remove genesis persistence from state
03c1418 
This avoids saving the genesisDoc to database.
When using goleveldb and ~4GiB+ genesis files, it causes a panic
during snappy encoding (panic: snappy: decoded block is too large).

refs akash-network/support#280

backports:
- cometbft#1017
- cometbft#1293

Signed-off-by: Artur Troian <troian.ap@gmail.com>
troian added a commit to akash-network/cometbft that referenced this pull request Feb 6, 2025
@troian
refactor: backport remove genesis persistence from state
adca6e0 
This avoids saving the genesisDoc to database.
When using goleveldb and ~4GiB+ genesis files, it causes a panic
during snappy encoding (panic: snappy: decoded block is too large).

refs akash-network/support#280

backports:
- cometbft#1017
- cometbft#1293

Signed-off-by: Artur Troian <troian.ap@gmail.com>
troian added a commit to akash-network/cometbft that referenced this pull request Feb 6, 2025
@troian
refactor: backport remove genesis persistence from state
7d76c7d 
This avoids saving the genesisDoc to database.
When using goleveldb and ~4GiB+ genesis files, it causes a panic
during snappy encoding (panic: snappy: decoded block is too large).

refs akash-network/support#280

backports:
- cometbft#1017
- cometbft#1293

Signed-off-by: Artur Troian <troian.ap@gmail.com>
troian added a commit to akash-network/cometbft that referenced this pull request Feb 6, 2025
@troian
refactor: backport remove genesis persistence from state
ac9717a 
This avoids saving the genesisDoc to database.
When using goleveldb and ~4GiB+ genesis files, it causes a panic
during snappy encoding (panic: snappy: decoded block is too large).

refs akash-network/support#280

backports:
- cometbft#1017
- cometbft#1293

Signed-off-by: Artur Troian <troian.ap@gmail.com>
troian added a commit to akash-network/cometbft that referenced this pull request Feb 6, 2025
@troian
refactor: backport remove genesis persistence from state
7d25c53 
This avoids saving the genesisDoc to database.
When using goleveldb and ~4GiB+ genesis files, it causes a panic
during snappy encoding (panic: snappy: decoded block is too large).

refs akash-network/support#280

backports:
- cometbft#1017
- cometbft#1293

Signed-off-by: Artur Troian <troian.ap@gmail.com>
troian added a commit to akash-network/cometbft that referenced this pull request Feb 6, 2025
@troian
refactor: backport remove genesis persistence from state
4967515 
This avoids saving the genesisDoc to database.
When using goleveldb and ~4GiB+ genesis files, it causes a panic
during snappy encoding (panic: snappy: decoded block is too large).

refs akash-network/support#280

backports:
- cometbft#1017
- cometbft#1293

Signed-off-by: Artur Troian <troian.ap@gmail.com>
troian added a commit to akash-network/cometbft that referenced this pull request Feb 6, 2025
@troian
refactor: backport remove genesis persistence from state
9fa3277 
This avoids saving the genesisDoc to database.
When using goleveldb and ~4GiB+ genesis files, it causes a panic
during snappy encoding (panic: snappy: decoded block is too large).

refs akash-network/support#280

backports:
- cometbft#1017
- cometbft#1293

Signed-off-by: Artur Troian <troian.ap@gmail.com>
troian added a commit to akash-network/cometbft that referenced this pull request Feb 6, 2025
@troian
refactor: backport remove genesis persistence from state
14e504a 
This avoids saving the genesisDoc to database.
When using goleveldb and ~4GiB+ genesis files, it causes a panic
during snappy encoding (panic: snappy: decoded block is too large).

refs akash-network/support#280

backports:
- cometbft#1017
- cometbft#1293

Signed-off-by: Artur Troian <troian.ap@gmail.com>
troian added a commit to akash-network/cometbft that referenced this pull request Feb 6, 2025
@troian
refactor: backport remove genesis persistence from state
236b2fa 
This avoids saving the genesisDoc to database.
When using goleveldb and ~4GiB+ genesis files, it causes a panic
during snappy encoding (panic: snappy: decoded block is too large).

refs akash-network/support#280

backports:
- cometbft#1017
- cometbft#1293

Signed-off-by: Artur Troian <troian.ap@gmail.com>
troian added a commit to akash-network/cometbft that referenced this pull request Feb 14, 2025
@troian
refactor: backport remove genesis persistence from state
db91a67 
This avoids saving the genesisDoc to database.
When using goleveldb and ~4GiB+ genesis files, it causes a panic
during snappy encoding (panic: snappy: decoded block is too large).

refs akash-network/support#280

backports:
- cometbft#1017
- cometbft#1293

Signed-off-by: Artur Troian <troian.ap@gmail.com>
troian added a commit to akash-network/cometbft that referenced this pull request Feb 14, 2025
@troian
refactor: backport remove genesis persistence from state
439b327 
This avoids saving the genesisDoc to database.
When using goleveldb and ~4GiB+ genesis files, it causes a panic
during snappy encoding (panic: snappy: decoded block is too large).

refs akash-network/support#280

backports:
- cometbft#1017
- cometbft#1293

Signed-off-by: Artur Troian <troian.ap@gmail.com>
troian added a commit to akash-network/cometbft that referenced this pull request Feb 14, 2025
@troian
refactor: backport remove genesis persistence from state
a090994 
This avoids saving the genesisDoc to database.
When using goleveldb and ~4GiB+ genesis files, it causes a panic
during snappy encoding (panic: snappy: decoded block is too large).

refs akash-network/support#280

backports:
- cometbft#1017
- cometbft#1293

Signed-off-by: Artur Troian <troian.ap@gmail.com>
troian added a commit to akash-network/cometbft that referenced this pull request Jul 7, 2025
@troian
refactor: backport remove genesis persistence from state
fccdd02 
This avoids saving the genesisDoc to database.
When using goleveldb and ~4GiB+ genesis files, it causes a panic
during snappy encoding (panic: snappy: decoded block is too large).

refs akash-network/support#280

backports:
- cometbft#1017
- cometbft#1293

Signed-off-by: Artur Troian <troian.ap@gmail.com>
troian added a commit to akash-network/cometbft that referenced this pull request Jul 14, 2025
@troian
feat: backport remove genesis persistence from state
5b275a2 
This avoids saving the genesisDoc to database.
When using goleveldb and ~4GiB+ genesis files, it causes a panic
during snappy encoding (panic: snappy: decoded block is too large).

refs akash-network/support#280

ports:
- cometbft#1017
- cometbft#1293

Signed-off-by: Artur Troian <troian@users.noreply.github.com>
@troian troian mentioned this pull request Jul 14, 2025
Merged
troian added a commit to akash-network/cometbft that referenced this pull request Jul 14, 2025
@troian
feat: port remove genesis persistence from state
99e424a 
This avoids saving the genesisDoc to database.
When using goleveldb and ~4GiB+ genesis files, it causes a panic
during snappy encoding (panic: snappy: decoded block is too large).

refs akash-network/support#280

ports:
- cometbft#1017
- cometbft#1293

Signed-off-by: Artur Troian <troian@users.noreply.github.com>
troian added a commit to akash-network/cometbft that referenced this pull request Jul 14, 2025
@troian
feat: port remove genesis persistence from state
e837d39 
This avoids saving the genesisDoc to database.
When using goleveldb and ~4GiB+ genesis files, it causes a panic
during snappy encoding (panic: snappy: decoded block is too large).

refs akash-network/support#280

ports:
- cometbft#1017
- cometbft#1293

Signed-off-by: Artur Troian <troian@users.noreply.github.com>
troian added a commit to akash-network/cometbft that referenced this pull request Jul 14, 2025
@troian
feat: port remove genesis persistence from state
7b57a83 
This avoids saving the genesisDoc to database.
When using goleveldb and ~4GiB+ genesis files, it causes a panic
during snappy encoding (panic: snappy: decoded block is too large).

refs akash-network/support#280

ports:
- cometbft#1017
- cometbft#1293

Signed-off-by: Artur Troian <troian@users.noreply.github.com>
troian added a commit to akash-network/cometbft that referenced this pull request Jul 14, 2025
@troian
feat: port remove genesis persistence from state
a668bba 
This avoids saving the genesisDoc to database.
When using goleveldb and ~4GiB+ genesis files, it causes a panic
during snappy encoding (panic: snappy: decoded block is too large).

refs akash-network/support#280

ports:
- cometbft#1017
- cometbft#1293

Signed-off-by: Artur Troian <troian@users.noreply.github.com>
troian added a commit to akash-network/cometbft that referenced this pull request Jul 14, 2025
@troian
feat: port remove genesis persistence from state
8f42896 
This avoids saving the genesisDoc to database.
When using goleveldb and ~4GiB+ genesis files, it causes a panic
during snappy encoding (panic: snappy: decoded block is too large).

refs akash-network/support#280

ports:
- cometbft#1017
- cometbft#1293

Signed-off-by: Artur Troian <troian@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sergio-mena sergio-mena sergio-mena approved these changes

@andynog andynog andynog approved these changes

+1 more reviewer

@mzabaluev mzabaluev mzabaluev left review comments

Reviewers whose approvals may not affect merge requirements
Assignees
No one assigned
Labels
P:storage-optimization Priority: Give operators greater control over storage and storage optimization storage
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@drklee3 @jmalicevic @alpha-omega-labs @andynog @mzabaluev @sergio-mena