feat: add db migration/patch tool #1908
Conversation
JayT106
requested review from
randy-cro and
thomas-nguy
and removed request for
a team
This comment has been minimized.
This comment has been minimized.
WalkthroughAdds a Changes
Sequence Diagram(s)sequenceDiagram
autonumber
actor User
participant CLI as "cronosd (root)"
participant DBCmd as "database cmd"
participant Cmd as "migrate / patch cmd"
participant DBMig as "dbmigrate (Migrate / PatchDatabase)"
participant Src as "Source DB"
participant Tgt as "Target DB"
User->>CLI: run "cronosd database migrate|patch"
CLI->>DBCmd: dispatch to database command
DBCmd->>Cmd: parse flags (backend, dbs, height, batch, verify/dry-run)
Cmd->>DBMig: call Migrate(opts) / PatchDatabase(opts)
DBMig->>Src: open source DB (with retry/backoff)
DBMig->>Tgt: open/create temp target DB
DBMig->>DBMig: count keys (apply height filter if provided)
loop per batch
DBMig->>Src: read batch keys/values
DBMig->>Tgt: write batch (apply conflict rules for patch)
DBMig->>DBMig: update stats, log progress
end
alt verify enabled
DBMig->>Src: compare keys/values
DBMig->>Tgt: detect extra keys
end
alt RocksDB target
DBMig->>Tgt: flush RocksDB memtables
end
DBMig-->>Cmd: return MigrationStats
Cmd-->>User: print summary and next steps
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes
Suggested reviewers
Poem
Pre-merge checks and finishing touches✅ Passed checks (3 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 9
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (24)
Makefile(1 hunks)cmd/cronosd/cmd/database.go(1 hunks)cmd/cronosd/cmd/migrate_db.go(1 hunks)cmd/cronosd/cmd/migrate_db_test.go(1 hunks)cmd/cronosd/cmd/patch_db.go(1 hunks)cmd/cronosd/cmd/root.go(1 hunks)cmd/cronosd/dbmigrate/QUICKSTART.md(1 hunks)cmd/cronosd/dbmigrate/README.md(1 hunks)cmd/cronosd/dbmigrate/height_filter.go(1 hunks)cmd/cronosd/dbmigrate/height_filter_test.go(1 hunks)cmd/cronosd/dbmigrate/height_parse_test.go(1 hunks)cmd/cronosd/dbmigrate/migrate.go(1 hunks)cmd/cronosd/dbmigrate/migrate_basic_test.go(1 hunks)cmd/cronosd/dbmigrate/migrate_dbname_test.go(1 hunks)cmd/cronosd/dbmigrate/migrate_no_rocksdb.go(1 hunks)cmd/cronosd/dbmigrate/migrate_rocksdb.go(1 hunks)cmd/cronosd/dbmigrate/migrate_rocksdb_test.go(1 hunks)cmd/cronosd/dbmigrate/migrate_test.go(1 hunks)cmd/cronosd/dbmigrate/patch.go(1 hunks)cmd/cronosd/dbmigrate/patch_test.go(1 hunks)cmd/cronosd/dbmigrate/swap-migrated-db.sh(1 hunks)x/cronos/keeper/keeper.go(3 hunks)x/cronos/rpc/api.go(2 hunks)x/e2ee/client/cli/encrypt.go(1 hunks)
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2024-10-12T22:09:46.096Z
Learnt from: yihuang
Repo: crypto-org-chain/cronos PR: 1618
File: memiavl/db_test.go:199-199
Timestamp: 2024-10-12T22:09:46.096Z
Learning: In unit tests within `memiavl/db_test.go`, converting `int64` to `uint32` is acceptable.
Applied to files:
cmd/cronosd/dbmigrate/migrate_basic_test.gocmd/cronosd/dbmigrate/migrate_test.gocmd/cronosd/dbmigrate/migrate_dbname_test.gocmd/cronosd/dbmigrate/patch_test.gocmd/cronosd/dbmigrate/height_filter_test.gocmd/cronosd/dbmigrate/migrate_rocksdb_test.go
🧬 Code graph analysis (15)
cmd/cronosd/dbmigrate/migrate_rocksdb.go (2)
cmd/cronosd/dbmigrate/migrate_no_rocksdb.go (1)
PrepareRocksDBOptions(13-15)cmd/cronosd/opendb/opendb_rocksdb.go (1)
NewRocksdbOptions(88-138)
cmd/cronosd/cmd/database.go (2)
cmd/cronosd/cmd/migrate_db.go (1)
MigrateCmd(301-306)cmd/cronosd/cmd/patch_db.go (1)
PatchCmd(306-311)
cmd/cronosd/cmd/migrate_db_test.go (1)
cmd/cronosd/cmd/migrate_db.go (3)
DBTypeApp(28-28)DBTypeCometBFT(29-29)DBTypeAll(30-30)
cmd/cronosd/dbmigrate/height_parse_test.go (1)
cmd/cronosd/dbmigrate/height_filter.go (2)
HeightRange(20-24)ParseHeightFlag(124-151)
cmd/cronosd/dbmigrate/migrate_no_rocksdb.go (1)
cmd/cronosd/dbmigrate/migrate_rocksdb.go (1)
PrepareRocksDBOptions(14-17)
cmd/cronosd/dbmigrate/migrate_basic_test.go (2)
cmd/cronosd/dbmigrate/migrate_test.go (7)
TestCountKeys(60-88)TestMigrationStats(91-106)TestMigrateLargeDatabase(109-140)TestMigrateEmptyDatabase(143-167)TestMigrationWithoutVerification(170-196)TestMigrationBatchSizes(199-229)TestMigrateSpecialKeys(334-423)cmd/cronosd/dbmigrate/migrate.go (3)
MigrateOptions(23-45)Migrate(74-233)MigrationStats(48-54)
cmd/cronosd/dbmigrate/migrate_test.go (2)
cmd/cronosd/dbmigrate/migrate_basic_test.go (7)
TestCountKeys(41-69)TestMigrationStats(102-117)TestMigrateLargeDatabase(120-151)TestMigrateEmptyDatabase(154-178)TestMigrationWithoutVerification(181-207)TestMigrationBatchSizes(210-240)TestMigrateSpecialKeys(243-288)cmd/cronosd/dbmigrate/migrate.go (3)
MigrationStats(48-54)MigrateOptions(23-45)Migrate(74-233)
cmd/cronosd/cmd/patch_db.go (5)
memiavl/types.go (1)
Logger(6-10)cmd/cronosd/dbmigrate/height_filter.go (2)
ParseHeightFlag(124-151)HeightRange(20-24)cmd/cronosd/dbmigrate/migrate_no_rocksdb.go (1)
PrepareRocksDBOptions(13-15)cmd/cronosd/dbmigrate/migrate_rocksdb.go (1)
PrepareRocksDBOptions(14-17)cmd/cronosd/dbmigrate/patch.go (3)
PatchOptions(46-59)ConflictAsk(36-36)PatchDatabase(62-170)
cmd/cronosd/cmd/migrate_db.go (4)
cmd/cronosd/dbmigrate/height_filter.go (2)
ParseHeightFlag(124-151)HeightRange(20-24)cmd/cronosd/dbmigrate/migrate_no_rocksdb.go (1)
PrepareRocksDBOptions(13-15)cmd/cronosd/dbmigrate/migrate_rocksdb.go (1)
PrepareRocksDBOptions(14-17)cmd/cronosd/dbmigrate/migrate.go (4)
MigrationStats(48-54)MigrateOptions(23-45)Migrate(74-233)DefaultBatchSize(17-17)
cmd/cronosd/dbmigrate/migrate_dbname_test.go (1)
cmd/cronosd/dbmigrate/migrate.go (2)
MigrateOptions(23-45)Migrate(74-233)
cmd/cronosd/cmd/root.go (1)
cmd/cronosd/cmd/database.go (1)
DatabaseCmd(8-29)
cmd/cronosd/dbmigrate/migrate.go (2)
memiavl/types.go (2)
Logger(6-10)NewNopLogger(18-18)cmd/cronosd/dbmigrate/height_filter.go (3)
HeightRange(20-24)DBNameBlockstore(14-14)DBNameTxIndex(15-15)
cmd/cronosd/dbmigrate/height_filter_test.go (1)
cmd/cronosd/dbmigrate/height_filter.go (1)
HeightRange(20-24)
cmd/cronosd/dbmigrate/patch.go (3)
memiavl/types.go (1)
Logger(6-10)cmd/cronosd/dbmigrate/height_filter.go (3)
HeightRange(20-24)DBNameBlockstore(14-14)DBNameTxIndex(15-15)cmd/cronosd/dbmigrate/migrate.go (1)
MigrationStats(48-54)
cmd/cronosd/dbmigrate/migrate_rocksdb_test.go (2)
cmd/cronosd/dbmigrate/migrate.go (2)
MigrateOptions(23-45)Migrate(74-233)memiavl/types.go (2)
Logger(6-10)NewNopLogger(18-18)
🪛 Gitleaks (8.29.0)
cmd/cronosd/dbmigrate/README.md
[high] 941-941: Detected a Generic API Key, potentially exposing access to various services and sensitive operations.
(generic-api-key)
🪛 LanguageTool
cmd/cronosd/dbmigrate/README.md
[style] ~493-~493: As an alternative to the over-used intensifier ‘very’, consider replacing this phrase.
Context: ...xample 6: Large Database Migration For very large databases, disable verification for fas...
(EN_WEAK_ADJECTIVE)
[grammar] ~1043-~1043: Ensure spelling is correct
Context: ... Safety and Best Practices #### Always Backup First ```bash # Timestamp your backups...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
[uncategorized] ~1168-~1168: If this is a compound adjective that modifies the following noun, use a hyphen.
Context: ...ase content and paths. 5. "Failed to open source database" ``` Error: failed to open ...
(EN_COMPOUND_ADJECTIVE_INTERNAL)
cmd/cronosd/dbmigrate/QUICKSTART.md
[grammar] ~51-~51: Ensure spelling is correct
Context: ...# or manually pkill cronosd ### 2. Backup Your Databases bash # Backup applic...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
[grammar] ~494-~494: Ensure spelling is correct
Context: ...| HDD (100MB/s) | SSD (500MB/s) | NVMe (3GB/s) | |--------------|---------------|--...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
🪛 markdownlint-cli2 (0.18.1)
cmd/cronosd/dbmigrate/README.md
268-268: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
583-583: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
663-663: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
677-677: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
704-704: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
714-714: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
729-729: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
737-737: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
744-744: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
759-759: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
765-765: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
769-769: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
781-781: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
929-929: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
935-935: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
1088-1088: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
1114-1114: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
1116-1116: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
1130-1130: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
1132-1132: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
1142-1142: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
1144-1144: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
1155-1155: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
1157-1157: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
1168-1168: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
1170-1170: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
1218-1218: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
1235-1235: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
1277-1277: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
1298-1298: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
cmd/cronosd/dbmigrate/QUICKSTART.md
126-126: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
138-138: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
355-355: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
360-360: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
750-750: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
756-756: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
762-762: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
768-768: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
794-794: Bare URL used
(MD034, no-bare-urls)
🪛 Shellcheck (0.11.0)
cmd/cronosd/dbmigrate/swap-migrated-db.sh
[warning] 212-212: FAILED_COUNT appears unused. Verify use (or export if used externally).
(SC2034)
[warning] 345-345: Don't use ls | grep. Use a glob or a for loop with a condition to allow non-alphanumeric filenames.
(SC2010)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (18)
- GitHub Check: integration_tests (ibc)
- GitHub Check: integration_tests (upgrade)
- GitHub Check: integration_tests (gas)
- GitHub Check: integration_tests (slow)
- GitHub Check: Run golangci-lint
- GitHub Check: integration_tests (unmarked)
- GitHub Check: integration_tests (ica)
- GitHub Check: integration_tests (gov)
- GitHub Check: integration_tests (ibc_timeout)
- GitHub Check: integration_tests (ibc_rly_evm)
- GitHub Check: integration_tests (ibc_update_client)
- GitHub Check: integration_tests (ibc_rly_gas)
- GitHub Check: Analyze (go)
- GitHub Check: unittest
- GitHub Check: build (macos-latest)
- GitHub Check: build (macos-14)
- GitHub Check: build (ubuntu-latest)
- GitHub Check: semgrep-cloud-platform/scan
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
cmd/cronosd/dbmigrate/migrate.go (1)
19-20: Remove unused exported constant.
DefaultWorkersis exported as part of the public API but is never referenced in the codebase. The migration runs single-threaded without worker pools.Apply this diff if concurrent workers are not planned:
- // DefaultWorkers is the number of concurrent workers for migration - DefaultWorkers = 4Otherwise, document its intended future use.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
cmd/cronosd/dbmigrate/migrate.go(1 hunks)
🧰 Additional context used
🧬 Code graph analysis (1)
cmd/cronosd/dbmigrate/migrate.go (3)
memiavl/types.go (2)
Logger(6-10)NewNopLogger(18-18)cmd/cronosd/dbmigrate/height_filter.go (3)
HeightRange(20-24)DBNameBlockstore(14-14)DBNameTxIndex(15-15)memiavl/db.go (2)
Load(146-266)DB(44-86)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (19)
- GitHub Check: Run golangci-lint
- GitHub Check: build (ubuntu-latest)
- GitHub Check: build (macos-14)
- GitHub Check: unittest
- GitHub Check: build (macos-latest)
- GitHub Check: integration_tests (ibc_timeout)
- GitHub Check: integration_tests (ica)
- GitHub Check: integration_tests (slow)
- GitHub Check: integration_tests (ibc_rly_gas)
- GitHub Check: integration_tests (gov)
- GitHub Check: integration_tests (gas)
- GitHub Check: integration_tests (ibc_rly_evm)
- GitHub Check: integration_tests (ibc)
- GitHub Check: integration_tests (ibc_update_client)
- GitHub Check: integration_tests (upgrade)
- GitHub Check: integration_tests (unmarked)
- GitHub Check: Analyze (go)
- GitHub Check: semgrep-cloud-platform/scan
- GitHub Check: semgrep-cloud-platform/scan
🔇 Additional comments (6)
cmd/cronosd/dbmigrate/migrate.go (6)
58-64: Floating-point progress calculation is acceptable here.Static analysis flagged the floating-point division as non-deterministic. For an offline migration tool's progress reporting, this is not a consensus concern and the implementation is appropriate.
183-240: Migration and verification flow is well-structured.The implementation correctly:
- Applies height filtering when specified (via
shouldIncludeKeyin helper functions)- Flushes RocksDB memtable before verification
- Releases locks by closing DBs before verification
- Provides clear instructions for finalizing the migration
Past concerns about height-range filtering in verification appear to be properly addressed.
257-427: Height filtering and migration logic correctly implemented.The code properly addresses the past concern about discrete height ranges by applying
shouldIncludeKeyfiltering in both:
- Line 294:
countKeysWithHeightFilter- ensures accurate counts- Line 377:
migrateWithIterator- ensures only requested keys are migratedThis prevents copying keys between min/max bounds that weren't explicitly requested.
429-482: Retry logic with exponential backoff is well-designed.The retry mechanisms handle OS-level file lock delays that can occur after database closure. The exponential backoff (50ms → 800ms over 5 attempts) is appropriate for this use case.
484-730: Comprehensive two-phase verification properly handles height filtering.The verification correctly:
- Applies height filtering in both phases (lines 557, 669) to avoid false missing-key errors
- Compares source → target (phase 1) and target → source (phase 2) to detect both missing and extra keys
- Reports detailed progress and mismatches
Multiple
time.Now()calls flagged by static analysis are acceptable for this offline migration tool's progress reporting.
115-126: Add nil check in defer to prevent panic.If
NewDBfails and returns a nil database, the deferred cleanup attempts to callClose()on a nil receiver, which may panic depending on the implementation.Apply this diff:
sourceDBClosed := false defer func() { - if !sourceDBClosed { + if !sourceDBClosed && sourceDB != nil { sourceDB.Close() } }()Likely an incorrect or invalid review comment.
Docstrings generation was requested by @JayT106. * #1908 (comment) The following files were modified: * `cmd/cronosd/cmd/database.go` * `cmd/cronosd/cmd/migrate_db.go` * `cmd/cronosd/cmd/patch_db.go` * `cmd/cronosd/cmd/root.go` * `cmd/cronosd/dbmigrate/height_filter.go` * `cmd/cronosd/dbmigrate/migrate.go` * `cmd/cronosd/dbmigrate/migrate_no_rocksdb.go` * `cmd/cronosd/dbmigrate/migrate_rocksdb.go` * `cmd/cronosd/dbmigrate/patch.go` * `cmd/cronosd/dbmigrate/swap-migrated-db.sh` * `x/e2ee/client/cli/encrypt.go`
|
@coderabbitai re-review |
|
@JayT106: I'll perform a comprehensive re-review of all the changes in this PR. ✅ Actions performedFull review triggered. |
There was a problem hiding this comment.
Actionable comments posted: 3
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
x/cronos/keeper/keeper.go (2)
152-162: Explicit return is good, but iterator must be closed.The explicit return improves clarity. However, the iterator created on line 154 is never closed, causing a resource leak.
Apply this diff to close the iterator:
func (k Keeper) GetExternalContracts(ctx sdk.Context) (out []types.TokenMapping) { store := ctx.KVStore(k.storeKey) iter := prefix.NewStore(store, types.KeyPrefixDenomToExternalContract).Iterator(nil, nil) + defer iter.Close() for ; iter.Valid(); iter.Next() { out = append(out, types.TokenMapping{ Denom: string(iter.Key()), Contract: common.BytesToAddress(iter.Value()).Hex(), }) } return out }
165-175: Explicit return is good, but iterator must be closed.The explicit return improves clarity. However, the iterator created on line 167 is never closed, causing a resource leak.
Apply this diff to close the iterator:
func (k Keeper) GetAutoContracts(ctx sdk.Context) (out []types.TokenMapping) { store := ctx.KVStore(k.storeKey) iter := prefix.NewStore(store, types.KeyPrefixDenomToAutoContract).Iterator(nil, nil) + defer iter.Close() for ; iter.Valid(); iter.Next() { out = append(out, types.TokenMapping{ Denom: string(iter.Key()), Contract: common.BytesToAddress(iter.Value()).Hex(), }) } return out }
♻️ Duplicate comments (2)
cmd/cronosd/dbmigrate/migrate_rocksdb.go (1)
14-16: Comment needs correction per past review.The inline comment states "false: disable read-only mode," but according to past review feedback, the second parameter to
NewRocksdbOptionsissstFileWriter bool, not a read-only mode flag.cmd/cronosd/dbmigrate/patch.go (1)
194-199: Define aminhelper (the build still fails).
min(...)is not defined anywhere (Go has no builtin), so this file will not compile. Please add a small helper and use it here.- logger.Debug("Blockstore key found", - "key_prefix", string(it.Key()[:min(10, len(it.Key()))]), - "key_hex", fmt.Sprintf("%x", it.Key()[:min(20, len(it.Key()))]), + logger.Debug("Blockstore key found", + "key_prefix", string(it.Key()[:minInt(10, len(it.Key()))]), + "key_hex", fmt.Sprintf("%x", it.Key()[:minInt(20, len(it.Key()))]),+func minInt(a, b int) int { + if a < b { + return a + } + return b +}
🧹 Nitpick comments (4)
Makefile (1)
116-116: Remove unnecessary trailing semicolons in per-target test commands.The per-target test commands (test-memiavl, test-store, test-versiondb) use
&&correctly for error propagation. However, the trailing;characters at the end of each line are unnecessary in Makefile recipes—they create empty commands that add no value. This also conflicts with the AI summary's claim that "trailing semicolons" were removed.Apply this diff to clean up the trailing semicolons:
test-memiavl: - @cd memiavl && go test -tags=objstore -v -mod=readonly ./... -coverprofile=$(COVERAGE) -covermode=atomic; + @cd memiavl && go test -tags=objstore -v -mod=readonly ./... -coverprofile=$(COVERAGE) -covermode=atomic test-store: - @cd store && go test -tags=objstore -v -mod=readonly ./... -coverprofile=$(COVERAGE) -covermode=atomic; + @cd store && go test -tags=objstore -v -mod=readonly ./... -coverprofile=$(COVERAGE) -covermode=atomic test-versiondb: - @cd versiondb && go test -tags=objstore,rocksdb -v -mod=readonly ./... -coverprofile=$(COVERAGE) -covermode=atomic; + @cd versiondb && go test -tags=objstore,rocksdb -v -mod=readonly ./... -coverprofile=$(COVERAGE) -covermode=atomicAlso applies to: 119-119, 122-122
cmd/cronosd/dbmigrate/migrate_rocksdb.go (1)
33-38: Consider requiring caller to provide RocksDB options.The fallback path creates default options when
opts == nil, but this introduces hidden resource allocation that the caller may not be aware of. SincePrepareRocksDBOptions()is provided for obtaining options, it might be clearer to require non-nil options and return an error if nil, forcing the caller to explicitly prepare options via the exported function.If you choose to keep the fallback, the current implementation with the
createdOptsflag and deferred cleanup is correct.cmd/cronosd/dbmigrate/README.md (1)
268-268: Consider adding language specifiers to code blocks.Multiple fenced code blocks lack language specifiers, which would enable syntax highlighting. While not critical, adding appropriate language tags (e.g.,
bash,go, ```text) would improve readability.Also applies to: 583-583, 663-663, 677-677, 704-704, 714-714, 729-729, 737-737, 744-744, 781-781, 929-929, 935-935, 1088-1088, 1116-1116, 1132-1132, 1144-1144, 1157-1157, 1170-1170, 1277-1277, 1298-1298
cmd/cronosd/dbmigrate/swap-migrated-db.sh (1)
348-348: Consider alternative tols | grepfor robustness.Shellcheck warns that piping
lstogrepcan break with non-alphanumeric filenames. While this is unlikely to cause issues in this context (database directory names), a more robust approach would use a glob pattern or a for loop with test conditions.Alternative implementation:
echo "" print_info "Current data directory contents:" for dir in "$DATA_DIR"/*; do if [[ -d "$dir" ]]; then size=$(du -sh "$dir" 2>/dev/null | awk '{print $1}') basename=$(basename "$dir") echo " $basename ($size)" fi done
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (25)
CHANGELOG.md(1 hunks)Makefile(1 hunks)cmd/cronosd/cmd/database.go(1 hunks)cmd/cronosd/cmd/migrate_db.go(1 hunks)cmd/cronosd/cmd/migrate_db_test.go(1 hunks)cmd/cronosd/cmd/patch_db.go(1 hunks)cmd/cronosd/cmd/root.go(1 hunks)cmd/cronosd/dbmigrate/QUICKSTART.md(1 hunks)cmd/cronosd/dbmigrate/README.md(1 hunks)cmd/cronosd/dbmigrate/height_filter.go(1 hunks)cmd/cronosd/dbmigrate/height_filter_test.go(1 hunks)cmd/cronosd/dbmigrate/height_parse_test.go(1 hunks)cmd/cronosd/dbmigrate/migrate.go(1 hunks)cmd/cronosd/dbmigrate/migrate_basic_test.go(1 hunks)cmd/cronosd/dbmigrate/migrate_dbname_test.go(1 hunks)cmd/cronosd/dbmigrate/migrate_no_rocksdb.go(1 hunks)cmd/cronosd/dbmigrate/migrate_rocksdb.go(1 hunks)cmd/cronosd/dbmigrate/migrate_rocksdb_test.go(1 hunks)cmd/cronosd/dbmigrate/migrate_test.go(1 hunks)cmd/cronosd/dbmigrate/patch.go(1 hunks)cmd/cronosd/dbmigrate/patch_test.go(1 hunks)cmd/cronosd/dbmigrate/swap-migrated-db.sh(1 hunks)x/cronos/keeper/keeper.go(3 hunks)x/cronos/rpc/api.go(2 hunks)x/e2ee/client/cli/encrypt.go(1 hunks)
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2024-10-12T22:09:46.096Z
Learnt from: yihuang
Repo: crypto-org-chain/cronos PR: 1618
File: memiavl/db_test.go:199-199
Timestamp: 2024-10-12T22:09:46.096Z
Learning: In unit tests within `memiavl/db_test.go`, converting `int64` to `uint32` is acceptable.
Applied to files:
cmd/cronosd/dbmigrate/migrate_test.gocmd/cronosd/dbmigrate/height_filter_test.gocmd/cronosd/dbmigrate/migrate_rocksdb_test.gocmd/cronosd/dbmigrate/migrate_dbname_test.gocmd/cronosd/dbmigrate/patch_test.gocmd/cronosd/dbmigrate/migrate_basic_test.go
🪛 GitHub Check: CodeQL
cmd/cronosd/dbmigrate/migrate.go
[warning] 522-522: Calling the system time
Calling the system time may be a possible source of non-determinism
[warning] 585-585: Calling the system time
Calling the system time may be a possible source of non-determinism
[warning] 644-644: Calling the system time
Calling the system time may be a possible source of non-determinism
cmd/cronosd/dbmigrate/patch.go
[warning] 77-77: Calling the system time
Calling the system time may be a possible source of non-determinism
[warning] 168-168: Calling the system time
Calling the system time may be a possible source of non-determinism
[warning] 695-828: Iteration over map
Iteration over map may be a possible source of non-determinism
[warning] 859-859: Calling the system time
Calling the system time may be a possible source of non-determinism
[warning] 1021-1021: Calling the system time
Calling the system time may be a possible source of non-determinism
[notice] 1013-1013: Floating point arithmetic
Floating point arithmetic operations are not associative and a possible source of non-determinism
[notice] 1234-1234: Floating point arithmetic
Floating point arithmetic operations are not associative and a possible source of non-determinism
🪛 Gitleaks (8.29.0)
cmd/cronosd/dbmigrate/README.md
[high] 941-941: Detected a Generic API Key, potentially exposing access to various services and sensitive operations.
(generic-api-key)
🪛 LanguageTool
cmd/cronosd/dbmigrate/QUICKSTART.md
[grammar] ~51-~51: Ensure spelling is correct
Context: ...# or manually pkill cronosd ### 2. Backup Your Databases bash # Backup applic...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
[grammar] ~494-~494: Ensure spelling is correct
Context: ...| HDD (100MB/s) | SSD (500MB/s) | NVMe (3GB/s) | |--------------|---------------|--...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
cmd/cronosd/dbmigrate/README.md
[style] ~493-~493: As an alternative to the over-used intensifier ‘very’, consider replacing this phrase.
Context: ...xample 6: Large Database Migration For very large databases, disable verification for fas...
(EN_WEAK_ADJECTIVE)
[grammar] ~1043-~1043: Ensure spelling is correct
Context: ... Safety and Best Practices #### Always Backup First ```bash # Timestamp your backups...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
[uncategorized] ~1168-~1168: If this is a compound adjective that modifies the following noun, use a hyphen.
Context: ...ase content and paths. 5. "Failed to open source database" ``` Error: failed to open ...
(EN_COMPOUND_ADJECTIVE_INTERNAL)
🪛 markdownlint-cli2 (0.18.1)
cmd/cronosd/dbmigrate/QUICKSTART.md
126-126: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
138-138: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
355-355: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
360-360: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
750-750: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
756-756: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
762-762: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
768-768: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
794-794: Bare URL used
(MD034, no-bare-urls)
cmd/cronosd/dbmigrate/README.md
268-268: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
583-583: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
663-663: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
677-677: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
704-704: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
714-714: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
729-729: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
737-737: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
744-744: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
759-759: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
765-765: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
769-769: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
781-781: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
929-929: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
935-935: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
1088-1088: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
1114-1114: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
1116-1116: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
1130-1130: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
1132-1132: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
1142-1142: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
1144-1144: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
1155-1155: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
1157-1157: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
1168-1168: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
1170-1170: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
1218-1218: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
1235-1235: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
1277-1277: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
1298-1298: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🪛 Shellcheck (0.11.0)
cmd/cronosd/dbmigrate/swap-migrated-db.sh
[warning] 348-348: Don't use ls | grep. Use a glob or a for loop with a condition to allow non-alphanumeric filenames.
(SC2010)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (16)
- GitHub Check: unittest
- GitHub Check: build (macos-latest)
- GitHub Check: build (macos-14)
- GitHub Check: build (ubuntu-latest)
- GitHub Check: Run golangci-lint
- GitHub Check: integration_tests (ibc_update_client)
- GitHub Check: integration_tests (gov)
- GitHub Check: integration_tests (unmarked)
- GitHub Check: integration_tests (slow)
- GitHub Check: integration_tests (ibc_timeout)
- GitHub Check: integration_tests (ibc_rly_evm)
- GitHub Check: integration_tests (gas)
- GitHub Check: integration_tests (upgrade)
- GitHub Check: integration_tests (ica)
- GitHub Check: integration_tests (ibc_rly_gas)
- GitHub Check: integration_tests (ibc)
🔇 Additional comments (24)
CHANGELOG.md (1)
5-5: Changelog entry is well‑formed and properly positioned.The new entry for PR #1908 follows the established format, contains an accurate description, and is correctly placed in the UNRELEASED section at the top of the changelog.
Makefile (1)
112-113: Test target simplified to single invocation—verify coverage behavior.The test target now runs all packages in a single
go testinvocation rather than as separate target dependencies. This is cleaner and should execute faster. Ensure that the unified coverage profile ($(COVERAGE)) correctly captures coverage across all packages as expected by CI/reporting tools.x/e2ee/client/cli/encrypt.go (1)
92-105: LGTM - explicit return improves clarity.The change from a bare
returntoreturn erris semantically equivalent with named returns but makes the intent more explicit. The error-handling pattern witherrors.Joinin the deferred function correctly captures bothio.Copyandw.Close()errors.x/cronos/rpc/api.go (2)
101-114: Explicit returns improve code clarity.The change from bare returns to explicit return statements makes the control flow clearer and easier to understand. Returning accumulated values on error paths is an interesting pattern that preserves partial context, though current callers don't use them (they check
errfirst).
387-387: LGTM: Explicit return improves readability.Making the return statement explicit rather than relying on named return values improves code clarity.
x/cronos/keeper/keeper.go (1)
113-119: LGTM: Explicit return improves clarity.The change from implicit to explicit return makes the function's intent clearer while maintaining correct logic.
cmd/cronosd/dbmigrate/migrate_rocksdb.go (2)
65-80: LGTM!The resource lifecycle management is correct. The database options are destroyed after opening (since RocksDB copies them internally), while the read/write options are passed to the wrapper and will be cleaned up when the database is closed.
82-93: LGTM!The flush logic correctly handles the RocksDB-specific case with proper resource cleanup for the flush options. The no-op behavior for non-RocksDB instances is appropriate.
cmd/cronosd/dbmigrate/README.md (2)
941-941: False positive: Example transaction hash, not a secret.The Gitleaks warning flagging line 941 is a false positive. The hex string shown is example debug output illustrating the format of a transaction hash in the documentation, not an actual API key or secret.
1-1360: Excellent comprehensive documentation.This README provides thorough documentation covering:
- Clear usage examples for both migrate and patch commands
- Detailed architecture and implementation notes
- Height filtering with bounded iterator optimization explained
- CometBFT key format documentation
- Safety guidelines and troubleshooting
The documentation quality is excellent and will help users understand and use these tools effectively.
cmd/cronosd/dbmigrate/migrate_no_rocksdb.go (1)
1-32: LGTM!The stub implementations correctly handle the case when RocksDB support is not compiled in. Error messages are clear and instruct users to rebuild with
-tags rocksdb, which is helpful.cmd/cronosd/dbmigrate/swap-migrated-db.sh (3)
205-212: LGTM!The dry-run handling correctly prevents side effects. The backup directory is only created when not in dry-run mode, addressing the previous review concern.
15-40: Excellent input validation.The
validate_backup_suffixfunction properly sanitizes and validates the backup suffix using a character whitelist. This prevents potential path traversal or injection issues. The immediate validation of both default and user-provided values is a good security practice.
256-294: LGTM!The swap logic correctly handles both existing and missing original databases, provides clear logging for both dry-run and actual execution, and properly tracks success counts.
cmd/cronosd/cmd/root.go (1)
196-199: LGTM!The integration of
DatabaseCmd()follows the existing pattern used forChangeSetCmd()and cleanly adds the new database management commands to the root command. The nil check allows for conditional compilation scenarios.cmd/cronosd/cmd/database.go (1)
7-29: LGTM!The
DatabaseCmd()function cleanly defines the database command group with a helpful alias and properly wires the migrate and patch subcommands. The documentation in the Long field clearly describes the available subcommands.cmd/cronosd/dbmigrate/patch_test.go (3)
13-42: LGTM!The
TestIncrementBytestest covers important cases including simple increment, carry-over propagation, and the all-FF edge case that returns nil (representing no upper bound for iterators).
44-78: LGTM!The
TestFormatKeyPrefixtest appropriately covers ASCII text, binary data hex formatting, and truncation behavior. UsingContainsassertions is suitable given that the exact formatting may include additional decorations.
80-114: LGTM!The
TestFormatValuetest covers the key scenarios: ASCII text, binary data, and empty value representation. The test strategy is sound.cmd/cronosd/dbmigrate/migrate_test.go (5)
19-58: LGTM!The
setupTestDBhelper properly creates test databases with configurable backends and populates them with sample data. The resource cleanup with deferred calls for RocksDB-specific options is correct.
60-89: LGTM!
TestCountKeysproperly validates the key counting functionality with both populated and empty databases, which is essential for progress reporting.
109-141: LGTM!
TestMigrateLargeDatabaseappropriately usestesting.Short()to skip expensive tests when running in short mode. The test validates migration with a realistic dataset size and smaller batch sizes.
232-332: LGTM!
TestVerifyMigrationcomprehensively tests the verification logic with multiple scenarios: identical databases (passing), value mismatches, extra keys, and missing keys (all failing). This ensures the verification catches data integrity issues.
334-424: Excellent edge case coverage.
TestMigrateSpecialKeysthoroughly tests migration with problematic key patterns including null bytes, Unicode, large keys, and edge cases. The test correctly handles databases that may not support certain key types (like empty keys) by logging and continuing rather than failing, while still ensuring that successfully written keys are properly migrated.
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
cmd/cronosd/dbmigrate/migrate_basic_test.go (2)
263-267: Consider simplifying the empty-key guard.All keys in
specialKeyshave length > 0, so the conditionif len(key) > 0is always true. If empty-key testing is not needed, remove the condition; if it is needed, add an empty-key case to the test data.Apply this diff to remove the unnecessary condition:
for i, key := range specialKeys { - if len(key) > 0 { // Skip empty key if not supported - value := []byte(fmt.Sprintf("value-%d", i)) - err := db.Set(key, value) - require.NoError(t, err) - } + value := []byte(fmt.Sprintf("value-%d", i)) + err := db.Set(key, value) + require.NoError(t, err) }
242-286: Verify migrated content for special keys.The test only checks that
stats.ProcessedKeys > 0but doesn't verify that special keys (null bytes, unicode, large keys) were correctly migrated. Consider opening the target database and verifying a subset of the special keys, similar to the thorough verification done inTestHeightFilteredVerification.Add verification after migration:
// Verify the migrated data targetDataDir := filepath.Join(targetDir, "data") targetDB, err := dbm.NewDB("application.migrate-temp", dbm.GoLevelDBBackend, targetDataDir) require.NoError(t, err) defer targetDB.Close() // Verify a few special keys testKey := []byte("🔑emoji-key") value, err := targetDB.Get(testKey) require.NoError(t, err) require.NotNil(t, value) require.Equal(t, []byte("value-4"), value) // Verify null byte key nullKey := []byte("\x00") value, err = targetDB.Get(nullKey) require.NoError(t, err) require.NotNil(t, value)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
cmd/cronosd/cmd/migrate_db.go(1 hunks)cmd/cronosd/cmd/patch_db.go(1 hunks)cmd/cronosd/dbmigrate/migrate_basic_test.go(1 hunks)x/cronos/keeper/keeper.go(2 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- cmd/cronosd/cmd/migrate_db.go
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2024-10-12T22:09:46.096Z
Learnt from: yihuang
Repo: crypto-org-chain/cronos PR: 1618
File: memiavl/db_test.go:199-199
Timestamp: 2024-10-12T22:09:46.096Z
Learning: In unit tests within `memiavl/db_test.go`, converting `int64` to `uint32` is acceptable.
Applied to files:
cmd/cronosd/dbmigrate/migrate_basic_test.go
🧬 Code graph analysis (3)
x/cronos/keeper/keeper.go (2)
x/cronos/types/cronos.pb.go (3)
TokenMapping(147-150)TokenMapping(154-154)TokenMapping(155-157)x/cronos/types/keys.go (1)
KeyPrefixDenomToAutoContract(36-36)
cmd/cronosd/dbmigrate/migrate_basic_test.go (2)
cmd/cronosd/dbmigrate/migrate.go (3)
MigrateOptions(24-46)Migrate(75-240)MigrationStats(49-55)cmd/cronosd/dbmigrate/height_filter.go (3)
DBNameBlockstore(14-14)HeightRange(20-24)DBNameTxIndex(15-15)
cmd/cronosd/cmd/patch_db.go (6)
memiavl/types.go (1)
Logger(6-10)cmd/cronosd/dbmigrate/height_filter.go (2)
ParseHeightFlag(124-151)HeightRange(20-24)cmd/cronosd/dbmigrate/migrate_rocksdb.go (1)
PrepareRocksDBOptions(14-16)cmd/cronosd/dbmigrate/migrate_no_rocksdb.go (1)
PrepareRocksDBOptions(13-15)cmd/cronosd/dbmigrate/patch.go (2)
PatchOptions(46-59)PatchDatabase(62-170)cmd/cronosd/dbmigrate/migrate.go (1)
DefaultBatchSize(18-18)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (18)
- GitHub Check: Run golangci-lint
- GitHub Check: build (macos-14)
- GitHub Check: build (macos-latest)
- GitHub Check: unittest
- GitHub Check: build (ubuntu-latest)
- GitHub Check: integration_tests (gov)
- GitHub Check: integration_tests (slow)
- GitHub Check: integration_tests (upgrade)
- GitHub Check: integration_tests (gas)
- GitHub Check: integration_tests (unmarked)
- GitHub Check: integration_tests (ibc_rly_gas)
- GitHub Check: integration_tests (ibc_update_client)
- GitHub Check: integration_tests (ibc_timeout)
- GitHub Check: integration_tests (ibc)
- GitHub Check: integration_tests (ica)
- GitHub Check: integration_tests (ibc_rly_evm)
- GitHub Check: Analyze (go)
- GitHub Check: semgrep-cloud-platform/scan
🔇 Additional comments (4)
x/cronos/keeper/keeper.go (3)
113-119: LGTM! Explicit return improves clarity.The explicit return statement is a good practice that makes the code more readable and maintainable, even though the named return parameters would have returned implicitly.
152-163: Critical fix: iterator cleanup prevents resource leak.The addition of
defer iter.Close()at line 155 is essential—it ensures the iterator is properly released even if the function exits early or panics. This fixes a potential resource leak. The explicit return statement also improves code clarity.
166-177: Iterator cleanup properly prevents resource leak.The addition of
defer iter.Close()at line 169 correctly ensures the iterator is properly released, preventing a resource leak. This implementation is consistent with the fix applied toGetExternalContractsand follows Go best practices for resource management.cmd/cronosd/dbmigrate/migrate_basic_test.go (1)
288-380: Excellent verification coverage!This test provides comprehensive validation of height-filtered migration: it verifies the correct key count, checks that expected keys exist in the target, and confirms that out-of-range keys are excluded. The debug logging and thorough assertions make this a strong reference for other migration tests.
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Signed-off-by: JayT106 <JayT106@users.noreply.github.com>
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
cmd/cronosd/dbmigrate/patch.go (1)
197-205: Build breaks:minhelper is missing.
min(...)isn’t defined anywhere (Go provides no builtin), so this file doesn’t compile—same failure that was flagged earlier. Please add a small helper and use it here.+func minInt(a, b int) int { + if a < b { + return a + } + return b +} + height, hasHeight := extractHeightFromBlockstoreKey(it.Key()) logger.Debug("Blockstore key found", - "key_prefix", string(it.Key()[:min(10, len(it.Key()))]), - "key_hex", fmt.Sprintf("%x", it.Key()[:min(20, len(it.Key()))]), + "key_prefix", string(it.Key()[:minInt(10, len(it.Key()))]), + "key_hex", fmt.Sprintf("%x", it.Key()[:minInt(20, len(it.Key()))]),
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (5)
cmd/cronosd/dbmigrate/README.md(1 hunks)cmd/cronosd/dbmigrate/height_filter.go(1 hunks)cmd/cronosd/dbmigrate/height_filter_test.go(1 hunks)cmd/cronosd/dbmigrate/migrate.go(1 hunks)cmd/cronosd/dbmigrate/patch.go(1 hunks)
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2024-10-12T22:09:46.096Z
Learnt from: yihuang
Repo: crypto-org-chain/cronos PR: 1618
File: memiavl/db_test.go:199-199
Timestamp: 2024-10-12T22:09:46.096Z
Learning: In unit tests within `memiavl/db_test.go`, converting `int64` to `uint32` is acceptable.
Applied to files:
cmd/cronosd/dbmigrate/height_filter_test.go
🧬 Code graph analysis (3)
cmd/cronosd/dbmigrate/migrate.go (2)
cmd/cronosd/dbmigrate/height_filter.go (3)
HeightRange(20-24)DBNameBlockstore(14-14)DBNameTxIndex(15-15)memiavl/db.go (2)
Load(146-266)DB(44-86)
cmd/cronosd/dbmigrate/height_filter_test.go (1)
cmd/cronosd/dbmigrate/height_filter.go (1)
HeightRange(20-24)
cmd/cronosd/dbmigrate/patch.go (2)
cmd/cronosd/dbmigrate/height_filter.go (3)
HeightRange(20-24)DBNameBlockstore(14-14)DBNameTxIndex(15-15)cmd/cronosd/dbmigrate/migrate.go (1)
MigrationStats(49-55)
🪛 Gitleaks (8.29.0)
cmd/cronosd/dbmigrate/README.md
[high] 951-951: Detected a Generic API Key, potentially exposing access to various services and sensitive operations.
(generic-api-key)
🪛 LanguageTool
cmd/cronosd/dbmigrate/README.md
[style] ~493-~493: As an alternative to the over-used intensifier ‘very’, consider replacing this phrase.
Context: ...xample 6: Large Database Migration For very large databases, disable verification for fas...
(EN_WEAK_ADJECTIVE)
[grammar] ~1053-~1053: Ensure spelling is correct
Context: ... Safety and Best Practices #### Always Backup First ```bash # Timestamp your backups...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
[uncategorized] ~1178-~1178: If this is a compound adjective that modifies the following noun, use a hyphen.
Context: ...ase content and paths. 5. "Failed to open source database" ``` Error: failed to open ...
(EN_COMPOUND_ADJECTIVE_INTERNAL)
🪛 markdownlint-cli2 (0.18.1)
cmd/cronosd/dbmigrate/README.md
268-268: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
583-583: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
663-663: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
677-677: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
704-704: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
715-715: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
731-731: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
739-739: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
746-746: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
761-761: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
767-767: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
771-771: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
783-783: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
939-939: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
945-945: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
1098-1098: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
1124-1124: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
1126-1126: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
1140-1140: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
1142-1142: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
1152-1152: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
1154-1154: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
1165-1165: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
1167-1167: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
1178-1178: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
1180-1180: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
1228-1228: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
1245-1245: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
1287-1287: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
1308-1308: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (19)
- GitHub Check: integration_tests (ibc_update_client)
- GitHub Check: integration_tests (ica)
- GitHub Check: integration_tests (ibc_rly_gas)
- GitHub Check: integration_tests (ibc_timeout)
- GitHub Check: integration_tests (upgrade)
- GitHub Check: integration_tests (gas)
- GitHub Check: integration_tests (gov)
- GitHub Check: integration_tests (slow)
- GitHub Check: integration_tests (unmarked)
- GitHub Check: integration_tests (ibc)
- GitHub Check: integration_tests (ibc_rly_evm)
- GitHub Check: build (ubuntu-latest)
- GitHub Check: build (macos-latest)
- GitHub Check: build (macos-14)
- GitHub Check: unittest
- GitHub Check: Run golangci-lint
- GitHub Check: Analyze (go)
- GitHub Check: semgrep-cloud-platform/scan
- GitHub Check: semgrep-cloud-platform/scan
…migrate, and update docs
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (5)
cmd/cronosd/dbmigrate/QUICKSTART.md(1 hunks)cmd/cronosd/dbmigrate/README.md(1 hunks)cmd/cronosd/dbmigrate/height_filter.go(1 hunks)cmd/cronosd/dbmigrate/migrate.go(1 hunks)cmd/cronosd/dbmigrate/migrate_basic_test.go(1 hunks)
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2024-10-12T22:09:46.096Z
Learnt from: yihuang
Repo: crypto-org-chain/cronos PR: 1618
File: memiavl/db_test.go:199-199
Timestamp: 2024-10-12T22:09:46.096Z
Learning: In unit tests within `memiavl/db_test.go`, converting `int64` to `uint32` is acceptable.
Applied to files:
cmd/cronosd/dbmigrate/migrate_basic_test.go
🧬 Code graph analysis (2)
cmd/cronosd/dbmigrate/migrate_basic_test.go (2)
cmd/cronosd/dbmigrate/migrate_test.go (7)
TestCountKeys(61-89)TestMigrationStats(92-107)TestMigrateLargeDatabase(110-141)TestMigrateEmptyDatabase(144-168)TestMigrationWithoutVerification(171-197)TestMigrationBatchSizes(200-230)TestMigrateSpecialKeys(335-423)cmd/cronosd/dbmigrate/migrate.go (3)
MigrateOptions(24-44)Migrate(73-204)MigrationStats(47-53)
cmd/cronosd/dbmigrate/migrate.go (2)
memiavl/types.go (2)
Logger(6-10)NewNopLogger(18-18)memiavl/db.go (2)
Load(146-266)DB(44-86)
🪛 Gitleaks (8.29.0)
cmd/cronosd/dbmigrate/README.md
[high] 874-874: Detected a Generic API Key, potentially exposing access to various services and sensitive operations.
(generic-api-key)
🪛 LanguageTool
cmd/cronosd/dbmigrate/README.md
[style] ~399-~399: As an alternative to the over-used intensifier ‘very’, consider replacing this phrase.
Context: ...xample 5: Large Database Migration For very large databases, disable verification for fas...
(EN_WEAK_ADJECTIVE)
[grammar] ~976-~976: Ensure spelling is correct
Context: ... Safety and Best Practices #### Always Backup First ```bash # Timestamp your backups...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
[uncategorized] ~1101-~1101: If this is a compound adjective that modifies the following noun, use a hyphen.
Context: ...ase content and paths. 5. "Failed to open source database" ``` Error: failed to open ...
(EN_COMPOUND_ADJECTIVE_INTERNAL)
cmd/cronosd/dbmigrate/QUICKSTART.md
[grammar] ~51-~51: Ensure spelling is correct
Context: ...# or manually pkill cronosd ### 2. Backup Your Databases bash # Backup applic...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
[grammar] ~458-~458: Ensure spelling is correct
Context: ...| HDD (100MB/s) | SSD (500MB/s) | NVMe (3GB/s) | |--------------|---------------|--...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
🪛 markdownlint-cli2 (0.18.1)
cmd/cronosd/dbmigrate/README.md
207-207: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
489-489: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
574-574: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
588-588: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
615-615: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
626-626: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
642-642: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
650-650: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
657-657: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
672-672: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
678-678: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
682-682: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
694-694: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
862-862: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
868-868: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
1021-1021: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
1047-1047: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
1049-1049: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
1063-1063: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
1065-1065: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
1075-1075: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
1077-1077: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
1088-1088: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
1090-1090: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
1101-1101: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
1103-1103: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
1139-1139: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
1156-1156: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
1179-1179: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
1200-1200: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
cmd/cronosd/dbmigrate/QUICKSTART.md
126-126: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
138-138: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
355-355: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
360-360: Emphasis used instead of a heading
(MD036, no-emphasis-as-heading)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (19)
- GitHub Check: unittest
- GitHub Check: build (macos-14)
- GitHub Check: build (macos-latest)
- GitHub Check: build (ubuntu-latest)
- GitHub Check: integration_tests (gas)
- GitHub Check: integration_tests (slow)
- GitHub Check: Run golangci-lint
- GitHub Check: integration_tests (ibc_rly_gas)
- GitHub Check: integration_tests (upgrade)
- GitHub Check: integration_tests (unmarked)
- GitHub Check: integration_tests (ibc)
- GitHub Check: integration_tests (ibc_update_client)
- GitHub Check: integration_tests (ibc_timeout)
- GitHub Check: integration_tests (ibc_rly_evm)
- GitHub Check: integration_tests (ica)
- GitHub Check: integration_tests (gov)
- GitHub Check: Analyze (go)
- GitHub Check: semgrep-cloud-platform/scan
- GitHub Check: semgrep-cloud-platform/scan
🔇 Additional comments (1)
cmd/cronosd/dbmigrate/height_filter.go (1)
383-427: Iterator strategy correctly implements prefix-bounded approach for height filtering.The functions create prefix-bounded iterators (e.g., all keys starting with
H:,P:, etc. for blockstore) which is the correct strategy for CometBFT's string-encoded heights. Since decimal strings don't sort lexicographically by numeric value (e.g., "H:20" > "H:150"), using exact height bounds would be incorrect.The design properly separates concerns:
- These iterators provide prefix-bounded iteration
shouldIncludeKey()provides application-level numeric filtering- Used together in patch operations (not full migrations) to selectively copy specific heights
This matches the documented Strategy B approach and is appropriate for the string-encoded key format.
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
cmd/cronosd/cmd/migrate_db.go(1 hunks)
🧰 Additional context used
🧬 Code graph analysis (1)
cmd/cronosd/cmd/migrate_db.go (4)
memiavl/types.go (1)
Logger(6-10)cmd/cronosd/dbmigrate/migrate_rocksdb.go (1)
PrepareRocksDBOptions(13-15)cmd/cronosd/dbmigrate/migrate_no_rocksdb.go (1)
PrepareRocksDBOptions(13-15)cmd/cronosd/dbmigrate/migrate.go (4)
MigrationStats(47-53)MigrateOptions(24-44)Migrate(73-204)DefaultBatchSize(18-18)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (19)
- GitHub Check: Run golangci-lint
- GitHub Check: integration_tests (ibc_rly_evm)
- GitHub Check: integration_tests (gas)
- GitHub Check: integration_tests (gov)
- GitHub Check: integration_tests (ica)
- GitHub Check: integration_tests (slow)
- GitHub Check: integration_tests (upgrade)
- GitHub Check: integration_tests (ibc_timeout)
- GitHub Check: integration_tests (ibc_update_client)
- GitHub Check: integration_tests (ibc)
- GitHub Check: integration_tests (ibc_rly_gas)
- GitHub Check: integration_tests (unmarked)
- GitHub Check: Analyze (go)
- GitHub Check: build (macos-latest)
- GitHub Check: unittest
- GitHub Check: build (macos-14)
- GitHub Check: build (ubuntu-latest)
- GitHub Check: semgrep-cloud-platform/scan
- GitHub Check: semgrep-cloud-platform/scan
🔇 Additional comments (5)
cmd/cronosd/cmd/migrate_db.go (5)
1-40: LGTM!The imports, flag constants, database type constants, and valid database names are well-structured and appropriate for the CLI command.
188-205: LGTM!The nil-pointer dereference issue flagged in the previous review has been correctly addressed. The error handling now safely checks if
stats != nilbefore accessing its fields, preventing panics whendbmigrate.Migratereturns(nil, err).
102-186: LGTM!The main migration workflow is well-structured:
- Properly validates that source and target backends differ
- Correctly handles flag precedence (--databases takes precedence over --db-type)
- Safely defaults targetHome to source home when not specified
- Conditionally prepares RocksDB options only when needed
- Comprehensive configuration logging
266-314: LGTM!The helper functions are well-implemented:
parseBackendTypecorrectly maps backend strings to types with appropriate error handlingparseDatabaseNamesproperly validates and normalizes comma-separated database lists, handling whitespace and empty entriesgetDBNamesFromTypecorrectly maps database type constants to their respective database sets
247-264: LGTM!The flag definitions use sensible defaults:
- Verify enabled by default for safety
- Conservative default of
DBTypeApp(only application database)- Proper default batch size referencing the constant
The
MigrateCmdwrapper correctly adapts the legacy command for use in the database command group.
|
|
||
| var verifiedKeys int64 | ||
| var mismatchCount int64 | ||
| lastProgressReport := time.Now() |
Check warning
Code scanning / CodeQL
Calling the system time Warning
| "verified", verifiedKeys, | ||
| "mismatches", mismatchCount, | ||
| ) | ||
| lastProgressReport = time.Now() |
Check warning
Code scanning / CodeQL
Calling the system time Warning
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
cmd/cronosd/cmd/migrate_db.go (1)
278-300: Consider deduplicating database names.The parsing logic correctly validates and trims database names, but doesn't deduplicate entries. For example,
--databases application,applicationwould attempt to migrate the same database twice, causing the second migration to fail when creating the temporary database.You can add deduplication using a map:
func parseDatabaseNames(databases string) ([]string, error) { if databases == "" { return nil, fmt.Errorf("no databases specified") } dbList := strings.Split(databases, ",") - var dbNames []string + seen := make(map[string]bool) + var dbNames []string for _, dbName := range dbList { dbName = strings.TrimSpace(dbName) if dbName == "" { continue } if !validDatabaseNames[dbName] { return nil, fmt.Errorf("invalid database name: %s (valid names: application, blockstore, state, tx_index, evidence)", dbName) } + if seen[dbName] { + continue + } + seen[dbName] = true dbNames = append(dbNames, dbName) } if len(dbNames) == 0 { return nil, fmt.Errorf("no valid databases specified in --databases flag") } return dbNames, nil }
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
cmd/cronosd/cmd/migrate_db.go(1 hunks)
🧰 Additional context used
🧬 Code graph analysis (1)
cmd/cronosd/cmd/migrate_db.go (4)
memiavl/types.go (1)
Logger(6-10)cmd/cronosd/dbmigrate/migrate_rocksdb.go (1)
PrepareRocksDBOptions(13-15)cmd/cronosd/dbmigrate/migrate_no_rocksdb.go (1)
PrepareRocksDBOptions(13-15)cmd/cronosd/dbmigrate/migrate.go (4)
MigrationStats(47-53)MigrateOptions(24-44)Migrate(73-204)DefaultBatchSize(18-18)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (19)
- GitHub Check: Run golangci-lint
- GitHub Check: build (macos-14)
- GitHub Check: build (macos-latest)
- GitHub Check: build (ubuntu-latest)
- GitHub Check: unittest
- GitHub Check: integration_tests (ica)
- GitHub Check: integration_tests (gov)
- GitHub Check: integration_tests (gas)
- GitHub Check: integration_tests (ibc_update_client)
- GitHub Check: integration_tests (slow)
- GitHub Check: integration_tests (upgrade)
- GitHub Check: integration_tests (ibc)
- GitHub Check: integration_tests (ibc_timeout)
- GitHub Check: integration_tests (unmarked)
- GitHub Check: integration_tests (ibc_rly_evm)
- GitHub Check: integration_tests (ibc_rly_gas)
- GitHub Check: Analyze (go)
- GitHub Check: semgrep-cloud-platform/scan
- GitHub Check: semgrep-cloud-platform/scan
🔇 Additional comments (7)
cmd/cronosd/cmd/migrate_db.go (7)
1-40: LGTM! Clean imports and well-defined constants.The package structure, imports, flag constants, database type constants, and validation map are all well-organized and follow Go best practices.
42-101: Excellent command documentation.The deprecation notice is clear, and the Long description provides comprehensive guidance including the migration process, database options, important warnings, and practical examples.
102-162: Configuration parsing and validation is solid.The backend type validation, target home defaulting with logging, and flag precedence logic (--databases takes priority over --db-type) are all implemented correctly. The configuration logging provides good observability for troubleshooting.
164-219: Well-implemented migration loop with proper error handling.The conditional RocksDB options preparation is correct, and the per-database migration loop properly accumulates statistics. The nil-safety check for stats before dereferencing (lines 190-203) correctly addresses the previous review concern about potential nil pointer dereference.
221-244: Clear and comprehensive success message.The final summary provides all necessary information and actionable next steps. The temporary database path format (line 237) now correctly matches the actual path created by the migration code (
{targetHome}/data/{dbName}.migrate-temp.db), addressing the previous review concern.
247-276: Sensible flag defaults and robust backend parsing.The flag defaults are well-chosen (verify=true for safety is particularly good), and the backend parser handles both "goleveldb" and "leveldb" aliases correctly with clear error messages.
302-314: LGTM! Database type mapping is clear and correct.The function correctly maps database types to their respective database lists, matching the documentation in the command's Long description.
3 participants
Implementation new CLI to support db migration or patch between goleveldb and rocksdb backend.
For details, can read the README in
cmd/cronosd/dbmigrateSummary by CodeRabbit
New Features
Documentation
Tests
Bug Fixes
Chores