Pruned or corrupt world state > Shutdown due to a VM fatal error.

[AionMiner]

Description

Since v1.6 cannot keep SPREAD node alive, non-graceful shutdown occurs then having restarting (due rocksDB lock) and reverting several thousand blocks, node will be in sync for some hours before non-graceful shutdown occurs again (observed 3 times over 3 days).

Have since swapped to FULL node and no issues.

... Node looks to import side chain block as best 
    (block 7375873, hash = 6f96c7 does not match that in mainnet explorer) ...

20-12-07 15:49:51.045 INFO  SYNC [p2p-recv-39]: <import-status: node = de921c, hash = 6f96c7, number = 7375873, txs = 0, result = IMPORTED_BEST, time elapsed = 32 ms>

... p2p-status logs for some minutes here, no branching occurs
    Pruned/corrupt notice then references a block some 270+ blocks behind aforementioned block ...

20-12-07 15:50:02.565 INFO  CONS [sync-ib]: Pruned or corrupt world state at block hash: 51189c, number: 7375602. Looking for ancestor block with valid world state ...
20-12-07 15:50:02.770 INFO  CONS [sync-ib]: Valid state found at block hash: 79a1d4, number: 7373812.
20-12-07 15:50:02.770 INFO  CONS [sync-ib]: Rebuilding block hash: 362425, number: 7373813, txs: 4.
20-12-07 15:50:02.783 WARN  CONS [sync-ib]: Block 7373813 has an invalid block header
20-12-07 15:50:02.783 ERROR CONS [sync-ib]: Attempting to add INVALID block.
20-12-07 15:50:02.783 INFO  CONS [sync-ib]: Rebuilding block hash: f41b59, number: 7373814, txs: 0.
20-12-07 15:50:02.787 INFO  CONS [sync-ib]: Rebuilding block hash: a94a7c, number: 7373815, txs: 4.
20-12-07 15:50:02.787 WARN  CONS [sync-ib]: Block 7373815 has an invalid block header
20-12-07 15:50:02.787 ERROR CONS [sync-ib]: Attempting to add INVALID block.
20-12-07 15:50:02.788 INFO  CONS [sync-ib]: Rebuilding block hash: c7faef, number: 7373816, txs: 0.
20-12-07 15:50:02.789 INFO  CONS [sync-ib]: Rebuilding block hash: 25bfc4, number: 7373817, txs: 1.
20-12-07 15:50:02.790 WARN  CONS [sync-ib]: Block 7373817 has an invalid block header
20-12-07 15:50:02.790 ERROR CONS [sync-ib]: Attempting to add INVALID block.
20-12-07 15:50:02.790 INFO  CONS [sync-ib]: Rebuilding block hash: fcc492, number: 7373818, txs: 2.
20-12-07 15:50:02.791 WARN  CONS [sync-ib]: Invalid transaction: Tx nonce 40609 != expected nonce 0 (parent nonce: 0): TransactionData [TransactionData [hash=89a68ca3bec3b3ad52696835ebd0087ba8b1271f25e319c62c23bc532bccb8ad, nonce=40609, destinationAddress=a057233584c9cd8e1ce2413b6adc94bcdc841dcb9246a669a6566e6e18edd6be, value=1139244436200678700, data=, energyLimit=90000, energyPrice=10000000000], txType=1, timeStamp=1607329835788934, signature=[pk: 60bec71fc0ffb6620d70203bf172a7b5482dd416ab9fbbf974ab603a4a689cc6 address: a026a32cd824232dfee3c34ecb90571ebf4102cab484e0c48543e2cbc00f07b7 signature: 2eb1a0f6d0064119af0527c6187906293f7d20bf489481bc5acba2a612c8efc6f97f170026eef855b5fef54fbe42400b4ff9fefd54f97efe72f002f7b53ff707], beaconHash=null]
20-12-07 15:50:02.791 ERROR CONS [sync-ib]: Attempting to add INVALID block.
20-12-07 15:50:02.791 INFO  CONS [sync-ib]: Rebuilding block hash: 276201, number: 7373819, txs: 0.
20-12-07 15:50:02.791 WARN  CONS [sync-ib]: Block 7373819 has an invalid block header

... continues block rebuild and invalid block/transactions over many lines
    before shutdown then occurs ...

20-12-07 15:50:03.480 ERROR CONS [sync-ib]: Attempting to add INVALID block.
20-12-07 15:50:03.480 INFO  CONS [sync-ib]: Rebuilding block hash: 7deec7, number: 7374155, txs: 7.
20-12-07 15:50:03.480 WARN  CONS [sync-ib]: Block 7374155 has an invalid block header
20-12-07 15:50:03.480 ERROR CONS [sync-ib]: Attempting to add INVALID block.
20-12-07 15:50:03.480 INFO  CONS [sync-ib]: Rebuilding block hash: a378e1, number: 7374156, txs: 1.
20-12-07 15:50:03.487 ERROR CONS [sync-ib]: Shutdown due to a VM fatal error.
org.aion.zero.impl.vm.common.VmFatalException: java.lang.RuntimeException: Invalid Trie state, missing node 55e0996d3031ee695bdc6a5654603dd726c7d553f2e794abc4eaacc6ca86be7a
	at org.aion.zero.impl.vm.avm.AvmTransactionExecutor.executeTransactions(AvmTransactionExecutor.java:115)
	at org.aion.zero.impl.vm.common.BulkExecutor.executeNextBatchOfAvmTransactions(BulkExecutor.java:340)
	at org.aion.zero.impl.vm.common.BulkExecutor.executeInternal(BulkExecutor.java:228)
	at org.aion.zero.impl.vm.common.BulkExecutor.executeAllTransactionsInBlock(BulkExecutor.java:102)
	at org.aion.zero.impl.blockchain.AionBlockchainImpl.applyBlock(AionBlockchainImpl.java:1891)
	at org.aion.zero.impl.blockchain.AionBlockchainImpl.processBlock(AionBlockchainImpl.java:1793)
	at org.aion.zero.impl.blockchain.AionBlockchainImpl.add(AionBlockchainImpl.java:1535)
	at org.aion.zero.impl.blockchain.AionBlockchainImpl.recoverWorldState(AionBlockchainImpl.java:2388)
	at org.aion.zero.impl.blockchain.AionBlockchainImpl.tryToConnectAndFetchSummary(AionBlockchainImpl.java:996)
	at org.aion.zero.impl.blockchain.AionBlockchainImpl.tryToConnectWithTimedExecution(AionBlockchainImpl.java:945)
	at org.aion.zero.impl.blockchain.AionBlockchainImpl.tryToConnect(AionBlockchainImpl.java:896)
	at org.aion.zero.impl.sync.TaskImportBlocks.processBatch(TaskImportBlocks.java:154)
	at org.aion.zero.impl.sync.TaskImportBlocks.importBlocks(TaskImportBlocks.java:56)
	at org.aion.zero.impl.sync.SyncMgr.lambda$filterBlocks$6(SyncMgr.java:431)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.RuntimeException: Invalid Trie state, missing node 55e0996d3031ee695bdc6a5654603dd726c7d553f2e794abc4eaacc6ca86be7a
	at org.aion.zero.impl.trie.TrieImpl.insert(TrieImpl.java:300)
	at org.aion.zero.impl.trie.TrieImpl.insert(TrieImpl.java:356)
	at org.aion.zero.impl.trie.TrieImpl.insert(TrieImpl.java:356)
	at org.aion.zero.impl.trie.TrieImpl.update(TrieImpl.java:193)
	at org.aion.zero.impl.trie.SecureTrie.update(SecureTrie.java:28)
	at org.aion.zero.impl.db.AionRepositoryImpl.updateAccountState(AionRepositoryImpl.java:698)
	at org.aion.zero.impl.db.AionRepositoryImpl.updateBatch(AionRepositoryImpl.java:425)
	at org.aion.zero.impl.db.AionRepositoryCache.flushTo(AionRepositoryCache.java:513)
	at org.aion.zero.impl.blockchain.AionBlockchainImpl.lambda$getPostExecutionWorkForApplyBlock$1(AionBlockchainImpl.java:400)
	at org.aion.zero.impl.vm.common.PostExecutionWork.doWork(PostExecutionWork.java:30)
	at org.aion.zero.impl.vm.avm.AvmTransactionExecutor.executeTransactions(AvmTransactionExecutor.java:100)
	... 16 common frames omitted
20-12-07 15:50:03.491 INFO  GEN  [shutdown]: Starting shutdown process...
20-12-07 15:50:03.491 INFO  GEN  [shutdown]: Shutting down RpcServer
20-12-07 15:50:03.510 INFO  GEN  [shutdown]: Shutdown RpcServer ... Done!
20-12-07 15:50:03.510 INFO  GEN  [shutdown]: Shutting down the AionHub...
20-12-07 15:50:03.510 INFO  GEN  [shutdown]: <KERNEL SHUTDOWN SEQUENCE>
20-12-07 15:50:12.454 INFO  P2P  [p2p-status]:

... continues print P2P status for several lines ...

20-12-07 15:52:03.512 ERROR SYNC [shutdown]: Pool did not terminate
20-12-07 15:52:03.512 INFO  GEN  [shutdown]: <shutdown-sync-mgr>
20-12-07 15:52:03.517 INFO  SYNC [shutdown]: BroadcastTxHandler shutting down!
20-12-07 15:52:03.531 INFO  GEN  [shutdown]: <shutdown-p2p-mgr>
20-12-07 15:52:07.533 INFO  GEN  [shutdown]: shutting down consensus...
20-12-07 15:52:07.533 INFO  GEN  [shutdown]: shutdown consensus... Done!

... does not shutdown DB, hangs here ...

System Information

I'm running:

• Which Aion version?: 1.6.1.0528110
• Which operating system?: Linux
• Which distributor and version?: Ubuntu 20.10
• How installed?: via binaries
• Are you fully synchronized?: yes
• Did you try to restart the node?: yes

[AionJayT]

@AionMiner Thanks for the issue report. We will investigate the issue. There is another mode called "TOP", it also saves DB spaces (Not like SPREAD mode, always prune the data, TOP mode keeps the full DB state of the top few hundreds block then prune it), if this node runs well. I think you can switch to this mode.

[AionMiner]

Have switched to TOP mode, working well 👍

[AionMiner]

Have encountered similar issue with TOP mode today (fresh 1.6.2 install, sync db from genesis online for ~3 days before issue).

Kernel was running fine until block height 7446583, then kernel outputs wall of WARN logs. ctrl+c graceful shutdown was successful but am now unable to get kernel to launch, attempted revert DB but similar notice as previous.

20-12-18 00:03:28.498 WARN  CONS [p2p-recv-23]: Invalid block#7446584: calculated total transaction energy consumed[4863125] mismatched with the energy used in header[5070887]
20-12-18 00:03:28.499 WARN  CONS [p2p-recv-23]: Tx[TransactionData [TransactionData [hash=cff92fc0c46220956b2b779a99d6eb1c5c1218f7473550396003c7f67d088ccf, nonce=31242, destinationAddress>
  , postTxState=c2bc88529d2a2f6d235fc42d23ba214b39df11561e1a08163df7c652752bcca2
  , error=
  , executionResult=
  , bloom=000000000000000000000000000000000000000000000000000000000000000000800000000000000000000000000000000010001000000000000000000000000000000000000000000000000000000000000000000000000>
  , logs=[Log { address = a0733306c2ee0c60224b0e59efeae8eee558c0ca1b39e7e5a14a575124549416, data = 59ff5405e4e562a0, topics = [ 426f6e64656400000000000000000000000000000000000000000000000>
  , nrgUsed=274535]]
20-12-18 00:03:28.520 WARN  CONS [p2p-recv-23]: Tx[TransactionData [TransactionData [hash=fd002eb3cb126077637908f91767c379efd45d841be426fa5a6174abb5f368b8, nonce=31243, destinationAddress>
  , postTxState=257857728798f86f307e180f851ded7e0f9175f01c703266dda8ed2b05d44ac7
  , error=
  , executionResult=
  , bloom=000000000000000000000000000000000000000000000000000000000000000000800000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000000000000>
  , logs=[Log { address = a0733306c2ee0c60224b0e59efeae8eee558c0ca1b39e7e5a14a575124549416, data = 1a70259aeca7a170, topics = [ 426f6e64656400000000000000000000000000000000000000000000000>
  , nrgUsed=274823]]
20-12-18 00:03:28.521 WARN  CONS [p2p-recv-23]: Tx[TransactionData [TransactionData [hash=c2d49c3ebb42794fe488f42bb6b326ec92ec3f1ea543b89f7243c1520d2d65d3, nonce=31244, destinationAddress>
  , postTxState=e6a62d38583cd71988bceeb944c521ecd6f219053984fb18ebac2d51ea5867f7
  , error=
  , executionResult=
  , bloom=000000000000000000000000000000000000000000000000000000000000000000800000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000000000000>
  , logs=[Log { address = a0733306c2ee0c60224b0e59efeae8eee558c0ca1b39e7e5a14a575124549416, data = 040cda01e686d9c3d5, topics = [ 426f6e646564000000000000000000000000000000000000000000000>
  , nrgUsed=274611]]
20-12-18 00:03:28.522 WARN  CONS [p2p-recv-23]: Tx[TransactionData [TransactionData [hash=18892beb6f0664dd3e347884de37bf07e12826c7881caaef2955f66e7be909bf, nonce=31245, destinationAddress>
  , postTxState=391cb13d527e8f15be61bc6e21e9765392db307d02a35ddefb511e670435fb41
  , error=
  , executionResult=
  , bloom=000000000000000000000000000000000000000000000000000000000000000000800800000000000000000002000000000000001000000000000000000000000000000000000000000000000000000000000000000000000>
  , logs=[Log { address = a0733306c2ee0c60224b0e59efeae8eee558c0ca1b39e7e5a14a575124549416, data = 02043af5275343, topics = [ 426f6e6465640000000000000000000000000000000000000000000000000>
  , nrgUsed=274747]]

Continues WARN output many lines until interrupt with ctrl+c graceful shutdown:

20-12-18 07:23:09.007 WARN  CONS [sync-ib]: Tx[TransactionData [TransactionData [hash=45552233a7e3a1f872a962e8f00844e4dcf50a1643295a7658bfdc85a0a5b36d, nonce=619646, destinationAddress=a008e42a76e2e779175c589efdb2a0e742b40d8d421df2b93a8a0b13090c7cc8, value=0, data=21001266696e616c697a65556e64656c6567617465060000000000000b56, energyLimit=2000000, energyPrice=10000000000], txType=1, timeStamp=1608249787018734, signature=[pk: 4ed53f2658f87719c2cf9d284b27037fe919dc>
  , postTxState=85e4241cff01ff4b799e35e734bf598415501ad2dae91e2b0bbf96117790b995
  , error=reverted
  , executionResult=
  , bloom=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000>
  , logs=[]
  , nrgUsed=54502]]
20-12-18 07:23:09.007 WARN  CONS [sync-ib]: Tx[TransactionData [TransactionData [hash=07e4715e311051b9af6e5839afc77ae8c35e702c2a35642ef7e1741d2ac9b1b7, nonce=619647, destinationAddress=a008e42a76e2e779175c589efdb2a0e742b40d8d421df2b93a8a0b13090c7cc8, value=0, data=21001266696e616c697a65556e64656c6567617465060000000000000b57, energyLimit=2000000, energyPrice=10000000000], txType=1, timeStamp=1608249788765874, signature=[pk: 4ed53f2658f87719c2cf9d284b27037fe919dc>
  , postTxState=6904cebc76d691a36cf01919f007876a148ed0f68a208e9b38496050cf31f88f
  , error=reverted
  , executionResult=
  , bloom=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000>
  , logs=[]
  , nrgUsed=54502]]
20-12-18 07:23:09.007 WARN  CONS [sync-ib]: Tx[TransactionData [TransactionData [hash=e4037856dbf40ae7be6d83a0746e0685e0bdcbdcd89f1f2392b325e12a9ea0fa, nonce=619648, destinationAddress=a008e42a76e2e779175c589efdb2a0e742b40d8d421df2b93a8a0b13090c7cc8, value=0, data=21001266696e616c697a65556e64656c6567617465060000000000000b58, energyLimit=2000000, energyPrice=10000000000], txType=1, timeStamp=1608249790511797, signature=[pk: 4ed53f2658f87719c2cf9d284b27037fe919dc>
  , postTxState=a2440b89d048b70f01e5746eee4d418a4a378c6881f387c5abe62f191718f184
  , error=reverted
  , executionResult=
  , bloom=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000>
  , logs=[]
  , nrgUsed=54502]]
20-12-18 07:23:09.007 INFO  GEN  [shutdown]: <shutdown-sync-mgr>
20-12-18 07:23:09.013 INFO  SYNC [shutdown]: BroadcastTxHandler shutting down!
20-12-18 07:23:09.026 INFO  GEN  [shutdown]: <shutdown-p2p-mgr>
20-12-18 07:23:13.028 INFO  GEN  [shutdown]: shutting down consensus...
20-12-18 07:23:13.029 INFO  GEN  [shutdown]: shutdown consensus... Done!
20-12-18 07:23:13.030 INFO  GEN  [shutdown]: shutting down DB...
20-12-18 07:23:13.061 INFO  GEN  [shutdown]: Details data source closed.
20-12-18 07:23:13.064 INFO  GEN  [shutdown]: contractInfoSource closed.
20-12-18 07:23:13.085 INFO  GEN  [shutdown]: State database closed.
20-12-18 07:23:13.089 INFO  GEN  [shutdown]: Transaction store closed.
20-12-18 07:23:13.111 INFO  GEN  [shutdown]: Block store closed.
20-12-18 07:23:13.118 INFO  GEN  [shutdown]: Pending block store closed.
20-12-18 07:23:13.121 INFO  GEN  [shutdown]: txPoolDatabase store closed.
20-12-18 07:23:13.123 INFO  GEN  [shutdown]: pendingTxCacheDatabase store closed.
20-12-18 07:23:13.124 INFO  GEN  [shutdown]: transformedCodeSource store closed.
20-12-18 07:23:13.124 INFO  GEN  [shutdown]: shutdown DB... Done!
20-12-18 07:23:13.124 INFO  GEN  [shutdown]: Shutting down the Aion Virtual Machine...
20-12-18 07:23:13.136 INFO  GEN  [shutdown]: ---------------------------------------------
20-12-18 07:23:13.136 INFO  GEN  [shutdown]: | Aion kernel graceful shutdown successful! |
20-12-18 07:23:13.136 INFO  GEN  [shutdown]: ---------------------------------------------

Attempting to launch kernel after DB revert (tried a few reverts to different heights, all report corrupt world state then RuntimeException follows):

20-12-18 07:25:30.498 INFO  GEN  [main]: The DB vendor is: ROCKSDB
20-12-18 07:25:30.804 INFO  GEN  [main]: Pruning ENABLED. Top block count set to 256.
20-12-18 07:25:30.809 INFO  GEN  [main]: Internal tx database disabled
20-12-18 07:25:30.829 INFO  GEN  [main]: Recovery initiated due to corrupt world state at block 7440000.
20-12-18 07:25:30.830 INFO  CONS [main]: Pruned or corrupt world state at block hash: 419c8c, number: 7440000. Looking for ancestor block with valid world state ...
20-12-18 07:25:30.917 INFO  CONS [main]: Valid state found at block hash: 9fadff, number: 7439440.
20-12-18 07:25:30.918 INFO  CONS [main]: Rebuilding block hash: a8d965, number: 7439441, txs: 1.
20-12-18 07:25:31.239 WARN  CONS [main]: Block 7439441 has an invalid block header
20-12-18 07:25:31.239 ERROR CONS [main]: Attempting to add INVALID block.
20-12-18 07:25:31.239 INFO  CONS [main]: Rebuilding block hash: 843646, number: 7439442, txs: 0.
20-12-18 07:25:31.246 INFO  CONS [main]: Rebuilding block hash: a54565, number: 7439443, txs: 0.
20-12-18 07:25:31.250 WARN  CONS [main]: Block 7439443 has an invalid block header
20-12-18 07:25:31.250 ERROR CONS [main]: Attempting to add INVALID block.
20-12-18 07:25:31.250 INFO  CONS [main]: Rebuilding block hash: 2827e9, number: 7439444, txs: 0.
Exception in thread "main" java.lang.ExceptionInInitializerError
	at org.aion.zero.impl.blockchain.AionImpl.inst(AionImpl.java:86)
	at org.aion.zero.impl.blockchain.AionFactory.create(AionFactory.java:12)
	at org.aion.Aion.main(Aion.java:209)
Caused by: java.lang.RuntimeException: Invalid Trie state, missing node 58daa6ff849fd36148ce2a101801ac41f94cd14f48567be14aa5b1cb84278b29
	at org.aion.zero.impl.trie.TrieImpl.insert(TrieImpl.java:300)
	at org.aion.zero.impl.trie.TrieImpl.insert(TrieImpl.java:356)
	at org.aion.zero.impl.trie.TrieImpl.update(TrieImpl.java:193)
	at org.aion.zero.impl.trie.SecureTrie.update(SecureTrie.java:28)
	at org.aion.zero.impl.db.AionRepositoryImpl.updateAccountState(AionRepositoryImpl.java:698)
	at org.aion.zero.impl.db.AionRepositoryImpl.updateBatch(AionRepositoryImpl.java:425)
	at org.aion.zero.impl.db.AionRepositoryCache.flushTo(AionRepositoryCache.java:513)
	at org.aion.zero.impl.blockchain.AionBlockchainImpl.add(AionBlockchainImpl.java:1551)
	at org.aion.zero.impl.blockchain.AionBlockchainImpl.recoverWorldState(AionBlockchainImpl.java:2409)
	at org.aion.zero.impl.blockchain.AionBlockchainImpl.load(AionBlockchainImpl.java:2720)
	at org.aion.zero.impl.blockchain.AionHub.initializeHub(AionHub.java:115)
	at org.aion.zero.impl.blockchain.AionHub.<init>(AionHub.java:91)
	at org.aion.zero.impl.blockchain.AionImpl.<init>(AionImpl.java:68)
	at org.aion.zero.impl.blockchain.AionImpl.<init>(AionImpl.java:37)
	at org.aion.zero.impl.blockchain.AionImpl$Holder.<clinit>(AionImpl.java:439)
	... 3 more

Update, tried reverting further back again but now kernel just hangs looking for ancestor block:

20-12-18 07:44:02.660 INFO  GEN  [main]: The DB vendor is: ROCKSDB
20-12-18 07:44:02.903 INFO  GEN  [main]: Pruning ENABLED. Top block count set to 256.
20-12-18 07:44:02.904 INFO  GEN  [main]: Internal tx database disabled
20-12-18 07:44:02.930 INFO  GEN  [main]: Recovery initiated due to corrupt world state at block 7400000.
20-12-18 07:44:02.931 INFO  CONS [main]: Pruned or corrupt world state at block hash: ac4e94, number: 7400000. Looking for ancestor block with valid world state ...

[AionJayT]

@AionMiner The WARN log of the Transaction is fine. Someone sent the invalid transaction to the network so has been reverted. You can set the logger Level to ERROR in the TX category or remove the category if you don't want to see the transaction log.
For the revert issue in the TOP mode. Yes, I can duplicate it. So I think the revert currently won't work for the TOP mode.

Do you still have the log when you see the launching issue when you wad shutting down the kernel at 7446583?

[AionMiner]

@AionJayT The WARN log was repeating itself over hundreds of thousands of lines, there was many log files created just full of the WARN notices and nothing else.

I noticed the TOP node out of sync from the connection to my FULL node and logged in to find the kernel just spitting out those WARN notices before I manually shut it down.

The launch log starting at 20-12-18 07:25:30.498 was directly after the actions: shutdown, launch (constant WARN logs again), shutdown, revert to 7440000 (reported success), launch (exception).

Unfortunately I no longer have access to the logs but I will spin up another server and see if it reproduces.

[AionMiner]

Has been over 10 days of uptime and TOP mode kernel has yet to reproduce the sync halt issue experienced previously. However, there's been no invalid block/tx logged since becoming fully sync'd (since block 7464349 till now 7550424).

I suspect the invalid block/tx event is what lead to the previous sync halt, being that the kernel was still running yet out of sync for ~7 hours (since the invalid block) with the only output whilst in this state being invalid block/tx WARN CONS until I intervened with shutdown.