[Feature] Rotate dataDirectory data

[kevkevinpal]

Description

Currently, bitcoin-augur-reference stores to where persistence:dataDirectory is defined in the config.yml
However, as time passes, the machine will eventually run out of memory.

I am proposing we add an optional config value
dataRetentionPeriod or storageDuration or dataTTL

where, if it is not defined, default to the current behavior of unlimited; however, if defined, then delete any buckets past the retention period.

We can pass in as an int representing days

Example

(30 days) dataRetentionPeriod: 30
(1 year) dataRetentionPeriod: 365

Motivation

This would be useful where storage is a constraint. There are lots of full Bitcoin nodes that are running on 1 tb hard drives, and if they only need the last month of data, then holding onto anything past might be unnecessary.

Also, we should not be forcing users to clear out data manually

Note

I'm not sure how much time I have to work on this so if someone else want to pick it up feel free to do so. Otherwise, I might pick it up in the future

[laurenshareshian]

Thanks for bringing this up. I think this issue might fall under the same category as the other issue related to ensuring that the node is synced. At the least for the time being, we're considering this reference library just something for folks to get started with to see how Augur works. We don't intend for it to be used in a production environment. Meaning, we can assume the node is synced and that there isn't a crazy amount of data to be stored. (If fact, if this was used in a production environment, then folks would probably want to switch to a database instead of a file-based system).

[kevkevinpal]

Makes sense, is there any plan to make a production-ready implementation of Augur?

I can avoid trying to add unnecessary improvements if it is just meant to be a simple reference

[laurenshareshian]

Makes sense, is there any plan to make a production-ready implementation of Augur?

I can avoid trying to add unnecessary improvements if it is just meant to be a simple reference

Good question. We were not planning to turn this reference implementation into a full-featured production-ready service but to use it to showcase the new augur library and to give people a starting point. New features that allow for data exploration (like your historical estimates endpoint) are great because it allows users to better understand how augur works but we were not planning on adding all the bells and whistles that a production-ready fee estimation service would need.