Dataset should support binary Buffer payload

[monolithed]

Which package is the feature request for? If unsure which one to select, leave blank

@crawlee/core

Feature

It would be great to have a plugin or API that allows accessing the binary data directly, or at least the option to store it in a Dataset without serialization.

I see that the current interface allows restoring data via Buffer.from, but I’m not sure about the efficiency of this approach or the acceptable data size it can handle.

Motivation

I’d like to share my use case.

I have the Crawlee logic extracted into a separate package, and the result of this package’s work is a screenshot.
Right now, I have to upload it to S3 directly inside the requestHandler, which feels like a terrible anti-pattern.

Ideal solution or implementation, and any additional constraints

The most straightforward solution would be to provide an interface for working with S3.

Alternative solutions or implementations

No response

Other context

No response

[barjin]

Hello, and thank you for your interest in this project!

By definition, the Dataset is a storage for serializable "data objects", i.e. JSON objects with strings, numbers, arrays, other such objects etc.

Because of the dataset implementation, you cannot store binary data directly into it - the Dataset backend (both local and on Apify Platform) stores the individual Dataset items as serialized JSON objects.

If you want to store binary data from the Crawlee crawlers, you can use the Key-Value Store class (docs). This is a Crawlee-native S3-like storage that allows you to store arbitrary (binary) data under string keys.

Alternatively, if you have to store the data in the dataset, you could use base64 or a similar encoding to convert the bytes into a string. Keep in mind that base64 increases the size of the data by roughly 30%.

I'll close this discussion as solved, but feel free to ask additional questions if you have any. Cheers!

[monolithed]

@barjin, thanks for the reply!
I ran some tests, and for small binary data (including screenshots up to 1–2 MB) I can confirm that the Dataset handles it well.
I’m sure the KeyValueStore would work fine too. However, I have some doubts when it comes to handling larger data volumes and synchronization, and here’s why:

1. Disk operations in distributed systems are usually more expensive and slower than network operations.
2. Right now, getting a file handle is synchronous, which means IO will constantly be blocked and the worker can’t parallelize tasks effectively (I’ve already found the issue where this is planned to be fixed).
3. Maintaining a machine with a KeyValueStore would likely require a dedicated engineer to monitor disk usage and load.
4. If a task crashes, the worker won’t be able to recover, resume, or hand it off, since it might run on a different machine. The advantage of using something like S3 is that I can start a task on one worker and finish it on another - for example, to move it to a less loaded server. In the current setup, this becomes a bottleneck.