DVC operations are becoming very slow, especially when working with many small files

[lakabosch]

When I run dvc add on a directory with 50,000 small images, it takes hours to complete and sometimes fails with "Killed" or appears to hang. My dataset is only 2GB total, which doesn't seem that large. What's causing this performance issue, and how can I fix it?

[aayostem]

The most effective solution is to archive your many small files into a few larger archives. This mean that DVC only needs to harsh and track one file.

# Instead of adding thousands of small files directly:
# dvc add images/

# Create an archive first
tar -czf images.tar.gz images/

# Then add the single archive to DVC
dvc add images.tar.gz

[skshetry]

When I run dvc add on a directory with 50,000 small images, it takes hours to complete and sometimes fails with "Killed" or appears to hang. My dataset is only 2GB total, which doesn't seem that large. What's causing this performance issue, and how can I fix it?

Could you please provide more information please? eg: dvc version, python version, platform information, etc. I cannot help otherwise.

Also, what traceback do you get when you kill? If you don't see it in normal mode, you can add --verbose which will show a traceback.