BEST PRACTICES

Best practices about using, configuring and deploying Flume-ng-vfs.

Ideally Flume-vfs will have a good performance with multiples huge files. Having to cache and process small amounts of files with gigabytes of information is always preferable to a lot of small files, because although we implement concurrency, opening and closing file is always an expensive operation. Also, if we select do nothing (nor move either delete) after processing by flume, it will reduce the workload that the source has to do, obviously. So, why add a feature that can move or delete file when flume has finished it task?, because it is being useful in our use case, and we believe than it can also be for some users. Setting a timeout for files to be processed with delay, it will decrease performance. The timeout is used by a scheduled service, that will free resources when finished. This configurable parameter must be used with care.