Skip to content

nethermind panic when using a debug_traceTransaction. #8600

@readygo67

Description

@readygo67

Description
panic when using a debug_traceTransaction.

Steps to Reproduce
on mainnet

curl -X POST http://localhost:28545   -H "Content-Type: application/json"   -d '{
    "jsonrpc":"2.0",                                                                                                           "method":"debug_traceTransaction",
    "params": [
      "0x9f969d92a369c96e567a3c23e917881fc5eb4673e0dcc5683fb04359af667d59",
      {}
    ],
    "id":1
  }'

Actual behavior
it returns

{"jsonrpc":"2.0","error":{"code":-32603,"message":"Internal error","data":"Nethermind.Trie.MissingTrieNodeException: Node A: P: H:0x08cf3393d291cf67a3070dc7e18ef3528bcc38765f6d579161005b17bf9e1a62 is missing from the DB\n   at Nethermind.Trie.Pruning.ReadOnlyTrieStore.Nethermind.Trie.Pruning.ITrieStoreInternal.LoadRlp(Hash256 address, TreePath& path, Hash256 hash, ReadFlags flags)\n   at Nethermind.Trie.Pruning.OverlayTrieStore.LoadRlp(Hash256 address, TreePath& path, Hash256 hash, ReadFlags flags) in /src/Nethermind/Nethermind.Trie/Pruning/OverlayTrieStore.cs:line 27\n   at Nethermind.Trie.PatriciaTree.ResolveNode(TrieNode node, TraverseContext& traverseContext, TreePath& path) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 573\n   at Nethermind.Trie.PatriciaTree.Get(ReadOnlySpan`1 rawKey, Hash256 rootHash) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 335\n   at Nethermind.State.StateProvider.<.ctor>b__13_0(AddressAsKey address) in /src/Nethermind/Nethermind.State/StateProvider.cs:line 60\n   at Nethermind.State.StateProvider.GetAndAddToCache(Address address) in /src/Nethermind/Nethermind.State/StateProvider.cs:line 711\n   at Nethermind.State.StateProvider.GetThroughCache(Address address) in /src/Nethermind/Nethermind.State/StateProvider.cs:line 732\n   at Nethermind.State.WorldState.IsContract(Address address) in /src/Nethermind/Nethermind.State/WorldState.cs:line 84\n   at Nethermind.Consensus.Processing.BlockProcessor.ProcessBlock(Block block, IBlockTracer blockTracer, ProcessingOptions options) in /src/Nethermind/Nethermind.Consensus/Processing/BlockProcessor.cs:line 326\n   at Nethermind.Consensus.Processing.BlockProcessor.ProcessOne(Block suggestedBlock, ProcessingOptions options, IBlockTracer blockTracer) in /src/Nethermind/Nethermind.Consensus/Processing/BlockProcessor.cs:line 282\n   at Nethermind.Consensus.Processing.BlockProcessor.Process(Hash256 newBranchStateRoot, List`1 suggestedBlocks, ProcessingOptions options, IBlockTracer blockTracer) in /src/Nethermind/Nethermind.Consensus/Processing/BlockProcessor.cs:line 148\n   at Nethermind.Consensus.Processing.BlockchainProcessor.ProcessBranch(ProcessingBranch& processingBranch, ProcessingOptions options, IBlockTracer tracer, String& error)\n   at Nethermind.Consensus.Processing.BlockchainProcessor.Process(Block suggestedBlock, ProcessingOptions options, IBlockTracer tracer, String& error) in /src/Nethermind/Nethermind.Consensus/Processing/BlockchainProcessor.cs:line 366\n   at Nethermind.Consensus.Processing.BlockchainProcessor.Process(Block suggestedBlock, ProcessingOptions options, IBlockTracer tracer) in /src/Nethermind/Nethermind.Consensus/Processing/BlockchainProcessor.cs:line 333\n   at Nethermind.Consensus.Processing.OneTimeChainProcessor.Process(Block block, ProcessingOptions options, IBlockTracer tracer) in /src/Nethermind/Nethermind.Consensus/Processing/OneTimeProcessor.cs:line 42\n   at Nethermind.Consensus.Tracing.GethStyleTracer.Trace(Block block, Hash256 txHash, CancellationToken cancellationToken, GethTraceOptions options, ProcessingOptions processingOptions) in /src/Nethermind/Nethermind.Consensus/Tracing/GethStyleTracer.cs:line 186\n   at Nethermind.Consensus.Tracing.GethStyleTracer.Trace(Hash256 txHash, GethTraceOptions traceOptions, CancellationToken cancellationToken) in /src/Nethermind/Nethermind.Consensus/Tracing/GethStyleTracer.cs:line 94\n   at Nethermind.JsonRpc.Modules.DebugModule.DebugBridge.GetTransactionTrace(Hash256 transactionHash, CancellationToken cancellationToken, GethTraceOptions gethTraceOptions) in /src/Nethermind/Nethermind.JsonRpc/Modules/DebugModule/DebugBridge.cs:line 115\n   at Nethermind.JsonRpc.Modules.DebugModule.DebugRpcModule.debug_traceTransaction(Hash256 transactionHash, GethTraceOptions options) in /src/Nethermind/Nethermind.JsonRpc/Modules/DebugModule/DebugRpcModule.cs:line 61\n   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)\n   at System.Reflection.MethodInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs)\n   at System.Reflection.MethodInvoker.InvokeDirectByRef(Object obj, Object arg1, Object arg2, Object arg3, Object arg4)\n   at System.Reflection.MethodInvoker.InvokeImpl(Object obj, Object arg1, Object arg2, Object arg3, Object arg4)\n   at System.Reflection.MethodInvoker.Invoke(Object obj, Span`1 arguments)\n   at Nethermind.JsonRpc.JsonRpcService.ExecuteAsync(JsonRpcRequest request, String methodName, ResolvedMethodInfo method, JsonRpcContext context) in /src/Nethermind/Nethermind.JsonRpc/JsonRpcService.cs:line 175"},"id":1}

nethmind's logs

07 May 16:44:19 | Storing beacon block root for block 22432673 (0x75b2814e12b3c751bb8c6a9a132357d3df89e36532878847196b584ea419353f) failed: Nethermind.Trie.MissingTrieNodeException: Node A: P: H:0x08cf3393d291cf67a3070dc7e18ef3528bcc38765f6d579161005b17bf9e1a62 is missing from the DB
   at Nethermind.Trie.Pruning.ReadOnlyTrieStore.Nethermind.Trie.Pruning.ITrieStoreInternal.LoadRlp(Hash256 address, TreePath& path, Hash256 hash, ReadFlags flags)
   at Nethermind.Trie.Pruning.OverlayTrieStore.LoadRlp(Hash256 address, TreePath& path, Hash256 hash, ReadFlags flags) in /src/Nethermind/Nethermind.Trie/Pruning/OverlayTrieStore.cs:line 27
   at Nethermind.Trie.PatriciaTree.ResolveNode(TrieNode node, TraverseContext& traverseContext, TreePath& path) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 573
   at Nethermind.Trie.PatriciaTree.Get(ReadOnlySpan`1 rawKey, Hash256 rootHash) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 335
   at Nethermind.State.StateProvider.<.ctor>b__13_0(AddressAsKey address) in /src/Nethermind/Nethermind.State/StateProvider.cs:line 60
   at Nethermind.State.StateProvider.GetAndAddToCache(Address address) in /src/Nethermind/Nethermind.State/StateProvider.cs:line 711
   at Nethermind.State.StateProvider.AccountExists(Address address) in /src/Nethermind/Nethermind.State/StateProvider.cs:line 110
   at Nethermind.Blockchain.BeaconBlockRoot.BeaconBlockRootHandler.BeaconRootsAccessList(Block block, IReleaseSpec spec, Boolean includeStorageCells) in /src/Nethermind/Nethermind.Blockchain/BeaconBlockRoot/BeaconBlockRootHandler.cs:line 33
   at Nethermind.Blockchain.BeaconBlockRoot.BeaconBlockRootHandler.StoreBeaconRoot(Block block, IReleaseSpec spec, ITxTracer tracer) in /src/Nethermind/Nethermind.Blockchain/BeaconBlockRoot/BeaconBlockRootHandler.cs:line 51
   at Nethermind.Consensus.Processing.BlockProcessor.StoreBeaconRoot(Block block, IReleaseSpec spec) in /src/Nethermind/Nethermind.Consensus/Processing/BlockProcessor.cs:line 388
07 May 16:44:19 | Encountered exception Nethermind.Trie.MissingTrieNodeException: Node A: P: H:0x08cf3393d291cf67a3070dc7e18ef3528bcc38765f6d579161005b17bf9e1a62 is missing from the DB
   at Nethermind.Trie.Pruning.ReadOnlyTrieStore.Nethermind.Trie.Pruning.ITrieStoreInternal.LoadRlp(Hash256 address, TreePath& path, Hash256 hash, ReadFlags flags)
   at Nethermind.Trie.Pruning.OverlayTrieStore.LoadRlp(Hash256 address, TreePath& path, Hash256 hash, ReadFlags flags) in /src/Nethermind/Nethermind.Trie/Pruning/OverlayTrieStore.cs:line 27
   at Nethermind.Trie.PatriciaTree.ResolveNode(TrieNode node, TraverseContext& traverseContext, TreePath& path) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 573
   at Nethermind.Trie.PatriciaTree.Get(ReadOnlySpan`1 rawKey, Hash256 rootHash) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 335
   at Nethermind.State.StateProvider.<.ctor>b__13_0(AddressAsKey address) in /src/Nethermind/Nethermind.State/StateProvider.cs:line 60
   at Nethermind.State.StateProvider.GetAndAddToCache(Address address) in /src/Nethermind/Nethermind.State/StateProvider.cs:line 711
   at Nethermind.State.StateProvider.GetThroughCache(Address address) in /src/Nethermind/Nethermind.State/StateProvider.cs:line 732
   at Nethermind.State.WorldState.IsContract(Address address) in /src/Nethermind/Nethermind.State/WorldState.cs:line 84
   at Nethermind.Consensus.Processing.BlockProcessor.ProcessBlock(Block block, IBlockTracer blockTracer, ProcessingOptions options) in /src/Nethermind/Nethermind.Consensus/Processing/BlockProcessor.cs:line 326
   at Nethermind.Consensus.Processing.BlockProcessor.ProcessOne(Block suggestedBlock, ProcessingOptions options, IBlockTracer blockTracer) in /src/Nethermind/Nethermind.Consensus/Processing/BlockProcessor.cs:line 282
   at Nethermind.Consensus.Processing.BlockProcessor.Process(Hash256 newBranchStateRoot, List`1 suggestedBlocks, ProcessingOptions options, IBlockTracer blockTracer) in /src/Nethermind/Nethermind.Consensus/Processing/BlockProcessor.cs:line 148 while processing blocks.
07 May 16:44:19 | Error during method execution, request: Id:1, debug_traceTransaction([
      "0x9f969d92a369c96e567a3c23e917881fc5eb4673e0dcc5683fb04359af667d59",
      {}
    ]) Nethermind.Trie.MissingTrieNodeException: Node A: P: H:0x08cf3393d291cf67a3070dc7e18ef3528bcc38765f6d579161005b17bf9e1a62 is missing from the DB
   at Nethermind.Trie.Pruning.ReadOnlyTrieStore.Nethermind.Trie.Pruning.ITrieStoreInternal.LoadRlp(Hash256 address, TreePath& path, Hash256 hash, ReadFlags flags)
   at Nethermind.Trie.Pruning.OverlayTrieStore.LoadRlp(Hash256 address, TreePath& path, Hash256 hash, ReadFlags flags) in /src/Nethermind/Nethermind.Trie/Pruning/OverlayTrieStore.cs:line 27
   at Nethermind.Trie.PatriciaTree.ResolveNode(TrieNode node, TraverseContext& traverseContext, TreePath& path) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 573
   at Nethermind.Trie.PatriciaTree.Get(ReadOnlySpan`1 rawKey, Hash256 rootHash) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 335
   at Nethermind.State.StateProvider.<.ctor>b__13_0(AddressAsKey address) in /src/Nethermind/Nethermind.State/StateProvider.cs:line 60
   at Nethermind.State.StateProvider.GetAndAddToCache(Address address) in /src/Nethermind/Nethermind.State/StateProvider.cs:line 711
   at Nethermind.State.StateProvider.GetThroughCache(Address address) in /src/Nethermind/Nethermind.State/StateProvider.cs:line 732
   at Nethermind.State.WorldState.IsContract(Address address) in /src/Nethermind/Nethermind.State/WorldState.cs:line 84
   at Nethermind.Consensus.Processing.BlockProcessor.ProcessBlock(Block block, IBlockTracer blockTracer, ProcessingOptions options) in /src/Nethermind/Nethermind.Consensus/Processing/BlockProcessor.cs:line 326
   at Nethermind.Consensus.Processing.BlockProcessor.ProcessOne(Block suggestedBlock, ProcessingOptions options, IBlockTracer blockTracer) in /src/Nethermind/Nethermind.Consensus/Processing/BlockProcessor.cs:line 282
   at Nethermind.Consensus.Processing.BlockProcessor.Process(Hash256 newBranchStateRoot, List`1 suggestedBlocks, ProcessingOptions options, IBlockTracer blockTracer) in /src/Nethermind/Nethermind.Consensus/Processing/BlockProcessor.cs:line 148
   at Nethermind.Consensus.Processing.BlockchainProcessor.ProcessBranch(ProcessingBranch& processingBranch, ProcessingOptions options, IBlockTracer tracer, String& error)
   at Nethermind.Consensus.Processing.BlockchainProcessor.Process(Block suggestedBlock, ProcessingOptions options, IBlockTracer tracer, String& error) in /src/Nethermind/Nethermind.Consensus/Processing/BlockchainProcessor.cs:line 366
   at Nethermind.Consensus.Processing.BlockchainProcessor.Process(Block suggestedBlock, ProcessingOptions options, IBlockTracer tracer) in /src/Nethermind/Nethermind.Consensus/Processing/BlockchainProcessor.cs:line 333
   at Nethermind.Consensus.Processing.OneTimeChainProcessor.Process(Block block, ProcessingOptions options, IBlockTracer tracer) in /src/Nethermind/Nethermind.Consensus/Processing/OneTimeProcessor.cs:line 42
   at Nethermind.Consensus.Tracing.GethStyleTracer.Trace(Block block, Hash256 txHash, CancellationToken cancellationToken, GethTraceOptions options, ProcessingOptions processingOptions) in /src/Nethermind/Nethermind.Consensus/Tracing/GethStyleTracer.cs:line 186
   at Nethermind.Consensus.Tracing.GethStyleTracer.Trace(Hash256 txHash, GethTraceOptions traceOptions, CancellationToken cancellationToken) in /src/Nethermind/Nethermind.Consensus/Tracing/GethStyleTracer.cs:line 94
   at Nethermind.JsonRpc.Modules.DebugModule.DebugBridge.GetTransactionTrace(Hash256 transactionHash, CancellationToken cancellationToken, GethTraceOptions gethTraceOptions) in /src/Nethermind/Nethermind.JsonRpc/Modules/DebugModule/DebugBridge.cs:line 115
   at Nethermind.JsonRpc.Modules.DebugModule.DebugRpcModule.debug_traceTransaction(Hash256 transactionHash, GethTraceOptions options) in /src/Nethermind/Nethermind.JsonRpc/Modules/DebugModule/DebugRpcModule.cs:line 61
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs)
   at System.Reflection.MethodInvoker.InvokeDirectByRef(Object obj, Object arg1, Object arg2, Object arg3, Object arg4)
   at System.Reflection.MethodInvoker.InvokeImpl(Object obj, Object arg1, Object arg2, Object arg3, Object arg4)
   at System.Reflection.MethodInvoker.Invoke(Object obj, Span`1 arguments)
   at Nethermind.JsonRpc.JsonRpcService.ExecuteAsync(JsonRpcRequest request, String methodName, ResolvedMethodInfo method, JsonRpcContext context) in /src/Nethermind/Nethermind.JsonRpc/JsonRpcService.cs:line 175
07 May 16:44:19 | Error when handling Id:1, debug_traceTransaction([
      "0x9f969d92a369c96e567a3c23e917881fc5eb4673e0dcc5683fb04359af667d59",
      {}
    ]) | {
  "jsonrpc": "2.0",
  "error": {
    "code": -32603,
    "message": "Internal error",
    "data": "Nethermind.Trie.MissingTrieNodeException: Node A: P: H:0x08cf3393d291cf67a3070dc7e18ef3528bcc38765f6d579161005b17bf9e1a62 is missing from the DB\n   at Nethermind.Trie.Pruning.ReadOnlyTrieStore.Nethermind.Trie.Pruning.ITrieStoreInternal.LoadRlp(Hash256 address, TreePath& path, Hash256 hash, ReadFlags flags)\n   at Nethermind.Trie.Pruning.OverlayTrieStore.LoadRlp(Hash256 address, TreePath& path, Hash256 hash, ReadFlags flags) in /src/Nethermind/Nethermind.Trie/Pruning/OverlayTrieStore.cs:line 27\n   at Nethermind.Trie.PatriciaTree.ResolveNode(TrieNode node, TraverseContext& traverseContext, TreePath& path) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 573\n   at Nethermind.Trie.PatriciaTree.Get(ReadOnlySpan`1 rawKey, Hash256 rootHash) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 335\n   at Nethermind.State.StateProvider.<.ctor>b__13_0(AddressAsKey address) in /src/Nethermind/Nethermind.State/StateProvider.cs:line 60\n   at Nethermind.State.StateProvider.GetAndAddToCache(Address address) in /src/Nethermind/Nethermind.State/StateProvider.cs:line 711\n   at Nethermind.State.StateProvider.GetThroughCache(Address address) in /src/Nethermind/Nethermind.State/StateProvider.cs:line 732\n   at Nethermind.State.WorldState.IsContract(Address address) in /src/Nethermind/Nethermind.State/WorldState.cs:line 84\n   at Nethermind.Consensus.Processing.BlockProcessor.ProcessBlock(Block block, IBlockTracer blockTracer, ProcessingOptions options) in /src/Nethermind/Nethermind.Consensus/Processing/BlockProcessor.cs:line 326\n   at Nethermind.Consensus.Processing.BlockProcessor.ProcessOne(Block suggestedBlock, ProcessingOptions options, IBlockTracer blockTracer) in /src/Nethermind/Nethermind.Consensus/Processing/BlockProcessor.cs:line 282\n   at Nethermind.Consensus.Processing.BlockProcessor.Process(Hash256 newBranchStateRoot, List`1 suggestedBlocks, ProcessingOptions options, IBlockTracer blockTracer) in /src/Nethermind/Nethermind.Consensus/Processing/BlockProcessor.cs:line 148\n   at Nethermind.Consensus.Processing.BlockchainProcessor.ProcessBranch(ProcessingBranch& processingBranch, ProcessingOptions options, IBlockTracer tracer, String& error)\n   at Nethermind.Consensus.Processing.BlockchainProcessor.Process(Block suggestedBlock, ProcessingOptions options, IBlockTracer tracer, String& error) in /src/Nethermind/Nethermind.Consensus/Processing/BlockchainProcessor.cs:line 366\n   at Nethermind.Consensus.Processing.BlockchainProcessor.Process(Block suggestedBlock, ProcessingOptions options, IBlockTracer tracer) in /src/Nethermind/Nethermind.Consensus/Processing/BlockchainProcessor.cs:line 333\n   at Nethermind.Consensus.Processing.OneTimeChainProcessor.Process(Block block, ProcessingOptions options, IBlockTracer tracer) in /src/Nethermind/Nethermind.Consensus/Processing/OneTimeProcessor.cs:line 42\n   at Nethermind.Consensus.Tracing.GethStyleTracer.Trace(Block block, Hash256 txHash, CancellationToken cancellationToken, GethTraceOptions options, ProcessingOptions processingOptions) in /src/Nethermind/Nethermind.Consensus/Tracing/GethStyleTracer.cs:line 186\n   at Nethermind.Consensus.Tracing.GethStyleTracer.Trace(Hash256 txHash, GethTraceOptions traceOptions, CancellationToken cancellationToken) in /src/Nethermind/Nethermind.Consensus/Tracing/GethStyleTracer.cs:line 94\n   at Nethermind.JsonRpc.Modules.DebugModule.DebugBridge.GetTransactionTrace(Hash256 transactionHash, CancellationToken cancellationToken, GethTraceOptions gethTraceOptions) in /src/Nethermind/Nethermind.JsonRpc/Modules/DebugModule/DebugBridge.cs:line 115\n   at Nethermind.JsonRpc.Modules.DebugModule.DebugRpcModule.debug_traceTransaction(Hash256 transactionHash, GethTraceOptions options) in /src/Nethermind/Nethermind.JsonRpc/Modules/DebugModule/DebugRpcModule.cs:line 61\n   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)\n   at System.Reflection.MethodInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs)\n   at System.Reflection.MethodInvoker.InvokeDirectByRef(Object obj, Object arg1, Object arg2, Object arg3, Object arg4)\n   at System.Reflection.MethodInvoker.InvokeImpl(Object obj, Object arg1, Object arg2, Object arg3, Object arg4)\n   at System.Reflection.MethodInvoker.Invoke(Object obj, Span`1 arguments)\n   at Nethermind.JsonRpc.JsonRpcService.ExecuteAsync(JsonRpcRequest request, String methodName, ResolvedMethodInfo method, JsonRpcContext context) in /src/Nethermind/Nethermind.JsonRpc/JsonRpcService.cs:line 175"
  },
  "id": 1
}

Expected behavior
even though missing from the DB, return gracefully rather than panic.

Screenshots

Desktop (please complete the following information):
Please provide the following information regarding your setup:

  • Operating System: [unbutu]
  • Version: [e.g. 1.17.0]
  • Installation Method: [Docker]
  • Consensus Client: [nimbus v1.4.3]

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions