feat(rust): add configurable size guardrails by Zakir032002 · Pull Request #3421 · apache/fory

Zakir032002 · 2026-02-25T19:34:23Z

Summary

Adds three opt-in Fory builder methods that let callers enforce upper bounds on the size of data allocated during deserialization. Without these guards a crafted payload can contain an absurdly large length prefix, causing Vec::with_capacity / string allocation to exhaust heap memory before a single byte of real data is read.

let fory = Fory::default()
    .max_string_bytes(1024 * 1024)    
    .max_collection_size(100_000)     
    .max_map_size(100_000);

All three limits default to None (no limit), so this is 100 % backwards-compatible.

Testing

cargo fmt --check   ✓
cargo clippy --all-targets --all-features -- -D warnings   ✓
cargo test test_size_guardrails   → 6 passed, 0 failed
cargo test                        → all existing tests pass

…ollection_size, max_map_size) Adds three Fory builder methods that let callers cap the byte length of deserialized strings, element count of collections, and entry count of maps. When a limit is exceeded an informative Error is returned instead of a blind allocation, preventing OOM from crafted payloads. - config.rs: three Option<usize> fields - context.rs: check_string_bytes / check_collection_size / check_map_size helpers - fory.rs: builder methods for all three limits - buffer.rs: read_varuint36small() helper used by string check - serializer/string.rs: check before string allocation - serializer/collection.rs: check in generic Vec / collection read - serializer/primitive_list.rs: check for Vec<primitive> fast path - serializer/map.rs: check before HashMap / BTreeMap allocation - tests/tests/test_size_guardrails.rs: 6 integration tests - tests/tests/mod.rs: register new test module Fixes apache#3409

- Bug 1: move check_collection_size before polymorphic dispatch in read_collection_data so HashSet/LinkedList/BTreeSet with polymorphic elements no longer bypass the limit - Bug 2: adjust string byte budget for UTF-16 (len is code-units; multiply by 2 before checking) so an adversary cannot double the allocation limit by forcing UTF-16 encoding - Bug 3: remove duplicate check_bound in read_latin1_string that regressed every latin1/ASCII string on the hot deserialization path - Bug 4: move check_map_size before BTreeMap::new() for a consistent 'check before allocate' pattern matching the HashMap path

…alloc

Zakir032002 · 2026-02-25T19:35:43Z

@chaokunyang , let me know if you need any changes

chaokunyang · 2026-02-26T03:54:15Z

Please set default value to max value of int32. And also check length is less or equal than buffer remaining size

…g error message in read_collection_data - read_vec_data was missing the buffer-remaining check entirely; a crafted len value would pass check_collection_size (default i32::MAX) and immediately hit Vec::with_capacity causing OOM before any read. Added a precise per-element byte check using size_of::<T>() since the static type is known at this call site. - read_collection_data had a coarse lower-bound check (elements vs bytes) but the error message falsely implied a byte-level guarantee. Updated the message to accurately describe what the check ensures.

…a before Vec::with_capacity Previously a crafted size_bytes value (e.g. 400_000_000) would pass check_collection_size (default i32::MAX) and then allocate a massive Vec before read_bytes had a chance to catch the truncation. The new check compares size_bytes against the bytes actually remaining in the buffer, which is precise because primitive_list encodes the payload length in bytes — so no approximation is needed.

…_bytes check_collection_size and check_map_size are both marked #[inline(always)] but check_string_bytes was not, despite being called on every single string deserialization — the hottest deserialization path in virtually every real-world payload. Consistent with the other two guard methods.

…nd primitive_list - test_vec_buffer_truncation_rejected: exercises the new buffer-remaining guard in read_vec_data (collection.rs). Crafts a valid Vec<i32> payload then truncates it so the declared element count cannot be satisfied. - test_primitive_list_buffer_truncation_rejected: exercises the new guard in primitive_list.rs fory_read_data. Serializes a Vec<i64>, truncates the payload, and confirms rejection even with default (i32::MAX) limits. - Existing test_buffer_truncation_rejected covers the string path and remains unchanged.

…alization

chaokunyang · 2026-02-27T08:20:11Z

rust/fory-core/src/resolver/context.rs

@@ -315,6 +315,9 @@ pub struct ReadContext<'a> {
    xlang: bool,
    max_dyn_depth: u32,
    check_struct_version: bool,
+    max_string_bytes: usize,


Let's simplify the config options, reserve:

max_binary_size, set default value to 64MB

max_collection_size, set default value to 1_000_000

chaokunyang · 2026-02-27T08:20:47Z

rust/fory-core/src/resolver/context.rs

@@ -472,6 +478,39 @@ impl<'a> ReadContext<'a> {
        self.meta_string_resolver.read_meta_string(&mut self.reader)
    }

+    #[inline(always)]
+    pub fn check_string_bytes(&self, byte_len: usize) -> Result<(), Error> {


For string, we don't have to check bytes length, bevcause we don't have preallocaiton for string

- Add max_binary_size and max_collection_size guardrails - Enforce collection size limits in Vec and Map deserializers - Add buffer-remaining cross-checks to prevent pre-allocation attacks - Simplify configuration by consolidating string and map limits - Fixes apache#3409

Zakir032002 · 2026-02-27T18:15:00Z

Heyy @chaokunyang , Done with the changes as you requested

chaokunyang · 2026-02-28T08:50:19Z

rust/fory-core/src/resolver/context.rs

@@ -472,6 +476,28 @@ impl<'a> ReadContext<'a> {
        self.meta_string_resolver.read_meta_string(&mut self.reader)
    }

+    #[inline(always)]
+    pub fn check_binary_size(&self, byte_len: usize) -> Result<(), Error> {


I don't think we need such methods? ANd it's even not used? why you add an unused method?

Zakir032002 · 2026-03-01T10:10:04Z

@chaokunyang — You're right, that method was unnecessary and unused.
I've removed check_binary_size and the related ReadContext field.

The implementation now only enforces max_collection_size at actual preallocation sites (collections, maps, primitive lists), and includes the buffer-remaining check as requested.

Config has been simplified to keep only max_binary_size (reserved, default 64MB) and max_collection_size (default 1_000_000).

All checks are passing. Please let me know if anything else should be adjusted.

chaokunyang · 2026-03-01T12:24:45Z

rust/fory-core/src/serializer/map.rs

+            .bf
+            .len()
+            .saturating_sub(context.reader.cursor);
+        if len as usize > remaining {


This won't work for streaming deserialization

Could you fix this? in stream deserialization, the buffer length doesn't represent remaining bytes anymore

Zakir032002 added 3 commits February 26, 2026 00:27

fix(rust): remove duplicate check_bound + move BTreeMap guard before …

66c35f8

…alloc

Zakir032002 marked this pull request as ready for review February 25, 2026 19:35

Zakir032002 requested review from chaokunyang and theweipeng as code owners February 25, 2026 19:35

Zakir032002 added 12 commits February 26, 2026 22:33

final

7ffbc8c

fix ci

8dba330

fix ci

b59ac23

Add buffer-remaining check

65172ea

remove buff check

b48f931

addong buff rem checks

c547e7e

fix ci

6c2a02c

fix(rust): add size guardrails for string, collection, and map deseri…

4057e18

…alization

Zakir032002 force-pushed the issue-3409-size-guardrails branch from f2d745c to 4057e18 Compare February 27, 2026 06:41

chaokunyang reviewed Feb 27, 2026

View reviewed changes

Zakir032002 added 2 commits February 27, 2026 22:48

remove check_string_bytes

6bf3ec2

Zakir032002 requested a review from chaokunyang February 28, 2026 06:25

chaokunyang reviewed Feb 28, 2026

View reviewed changes

remove unused check_binary_size from ReadContext

7891f39

Zakir032002 force-pushed the issue-3409-size-guardrails branch from 8845190 to 7891f39 Compare March 1, 2026 09:23

Update buffer.rs

9b36d92

Zakir032002 requested a review from chaokunyang March 1, 2026 10:10

chaokunyang reviewed Mar 1, 2026

View reviewed changes

chaokunyang changed the title ~~feat(rust): add configurable size guardrails (max_string_bytes, max_collection_size, max_map_size)~~ feat(rust): add configurable size guardrails Mar 5, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(rust): add configurable size guardrails#3421

feat(rust): add configurable size guardrails#3421
Zakir032002 wants to merge 19 commits intoapache:mainfrom
Zakir032002:issue-3409-size-guardrails

Zakir032002 commented Feb 25, 2026 •

edited

Loading

Uh oh!

Zakir032002 commented Feb 25, 2026

Uh oh!

chaokunyang commented Feb 26, 2026 •

edited

Loading

Uh oh!

chaokunyang Feb 27, 2026

Uh oh!

chaokunyang Feb 27, 2026

Uh oh!

Zakir032002 commented Feb 27, 2026

Uh oh!

chaokunyang Feb 28, 2026

Uh oh!

Zakir032002 commented Mar 1, 2026

Uh oh!

chaokunyang Mar 1, 2026

Uh oh!

chaokunyang Mar 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Zakir032002 commented Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Uh oh!

Zakir032002 commented Feb 25, 2026

Uh oh!

chaokunyang commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chaokunyang Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

chaokunyang Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Zakir032002 commented Feb 27, 2026

Uh oh!

chaokunyang Feb 28, 2026

Choose a reason for hiding this comment

Uh oh!

Zakir032002 commented Mar 1, 2026

Uh oh!

chaokunyang Mar 1, 2026

Choose a reason for hiding this comment

Uh oh!

chaokunyang Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Zakir032002 commented Feb 25, 2026 •

edited

Loading

chaokunyang commented Feb 26, 2026 •

edited

Loading