Skip to content

batch fetch/export process for export command #19

@zeroXbrock

Description

@zeroXbrock

hindsight export currently loads ALL entries from the source db into memory before dumping them into the destination.

This is reeeal slow on a large dataset (1+ months).

proposed solution: query 1000 arbs at a time, export them as you go; maximize parallelization of load/save

  • file export will need modification to append to the file, also thread-safe lock on the file ofc
  • db ops can likely be done w/ parallel connections, as there won't be any write conflicts coming from the source

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions