Skip to content

feat: Support Store S3 #520

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
speeddragon wants to merge 29 commits into
base: edge
Choose a base branch
from
+1,409 −3
Open

feat: Support Store S3 #520

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
be92000
feat(s3): s3 store
shyba Oct 9, 2025
a89746c
feat(s3): fix deps, fix scope, fix other bugs
shyba Oct 10, 2025
b076be7
style: code cleanup
nikooo777 Oct 11, 2025
e88a871
fix: Trying to fix link_fragment_test
speeddragon Oct 13, 2025
9321a4e
fix: reset call
speeddragon Oct 13, 2025
489626a
fix: Improve logic in not_found link. 1 test failing
speeddragon Oct 13, 2025
6e48ded
fix: tests
speeddragon Oct 13, 2025
20c0d39
test: Add s3 cache to hb_cache and hb_store tests
speeddragon Oct 13, 2025
efa25bc
test: Add hb_store_s3 to hb_cache and hb_store test suite
speeddragon Oct 14, 2025
12b787d
test: Do not run match on hb_store_s3
speeddragon Oct 14, 2025
b4fa6d4
impr: code error handling, tests and test documentation
speeddragon Oct 16, 2025
039900c
impr: Fix code style and minor issues
speeddragon Oct 16, 2025
421efd9
impr: Prefix config
speeddragon Oct 20, 2025
18e8598
impr: Remove hackney, support gun for S3, clean up code
speeddragon Oct 22, 2025
4426e36
impr: code clean up
speeddragon Oct 22, 2025
4b648c5
impr: small changes and code clean, remove prefix
speeddragon Oct 22, 2025
e74a4d6
fix: http requests with hb:start_mainnet/1
speeddragon Oct 23, 2025
e3047ec
impr: Move docs to S3, revert start fix with start_mainnet
speeddragon Oct 24, 2025
c065aca
fix: Changet boolean to binary, boolean not support as ao-type
speeddragon Oct 24, 2025
9d6bbfd
refactor: remove links and groups, simplify code
speeddragon Oct 29, 2025
c05c002
fix: missing cache bug
speeddragon Oct 29, 2025
80380c7
feat: Add sharding
speeddragon Oct 30, 2025
9bb98c8
fix: compile error on non s3 profile
speeddragon Oct 30, 2025
8b93474
fix: shard_key and resolve bugs
speeddragon Oct 31, 2025
0fb7ad3
style: Minor details
speeddragon Nov 3, 2025
13d4e80
fix: rebase
speeddragon Nov 3, 2025
e17a9a8
fix: Improvements
speeddragon Nov 7, 2025
b64dfce
fix: Add erlcloud to release under S3 profile
speeddragon Nov 7, 2025
4712906
fix: Proper fix to release with and without s3 profile
speeddragon Nov 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@ HyperBEAM supports several optional build profiles that enable additional featur

- `genesis_wasm`: Enables Genesis WebAssembly support
- `rocksdb`: Enables RocksDB storage backend (adds RocksDB v1.8.0 dependency)
- `s3`: Enables S3 storage backend
- `http3`: Enables HTTP/3 support via QUIC protocol


Expand Down
14 changes: 14 additions & 0 deletions rebar.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,17 @@
{d, 'ENABLE_ROCKSDB', true}
]}
]},
{s3, [
{deps, [
{erlcloud, {git, "https://github.com/erlcloud/erlcloud.git",
{ref, "fe58402b40b93a20176d12b4bc211ea6a5b0d915"}}
}
]},
{erl_opts, [
{d, 'ENABLE_S3', true}
]},
{relx, [{release, {'hb', "0.0.1"}, [hb, b64fast, cowboy, gun, luerl, prometheus, prometheus_cowboy, elmdb, erlcloud]}]}
]},
{http3, [
{deps, [
{quicer, {git, "https://github.com/emqx/quic.git",
Expand All @@ -55,6 +66,9 @@
{add, cowboy, [{erl_opts, [{d, 'COWBOY_QUICER', 1}]}]},
{add, gun, [{erl_opts, [{d, 'GUN_QUICER', 1}]}]}
]}
]},
{test, [
{deps, [{meck, "1.1.0"}]}
]}
]}.

Expand Down
16 changes: 13 additions & 3 deletions src/hb_store.erl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -440,7 +440,7 @@ test_stores() ->
}
]
}
] ++ rocks_stores().
] ++ rocks_stores() ++ s3_stores().

-ifdef(ENABLE_ROCKSDB).
rocks_stores() ->
Expand All @@ -453,16 +453,26 @@ rocks_stores() ->
-else.
rocks_stores() -> [].
-endif.

-ifdef(ENABLE_S3).
s3_stores() ->
[(hb_store_s3:default_test_opts())#{
<<"benchmark-scale">> => 0.01
}].
-else.
s3_stores() -> [].
-endif.
generate_test_suite(Suite) ->
generate_test_suite(Suite, test_stores()).
generate_test_suite(Suite, Stores) ->
hb:init(),
application:ensure_all_started(hb),
Copy link
Author

@speeddragon speeddragon Oct 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is necessary to use hb_http_client in hb_store_s3 requests.

lists:map(
fun(Store = #{<<"store-module">> := Mod}) ->
{foreach,
fun() ->
hb_store:start(Store)
hb_store:start(Store),
% If the test fails, the store isn't cleared.
hb_store:reset(Store)
end,
fun(_) ->
hb_store:reset(Store)
Expand Down
Loading