fix: Fix checking value in Golang ValidateJSONB #2383

icbd · 2025-10-04T15:52:35Z

Description

hatchet/sdks/python/hatchet_sdk/worker/runner/runner.py

Lines 501 to 506 in 2a08cbf

    
                           Task outputs cannot contain the unicode null character \\u0000 
        
                           Please see this Discord thread: https://discord.com/channels/1088927970518909068/1384324576166678710/1386714014565928992 
        
                           Relevant Postgres documentation: https://www.postgresql.org/docs/current/datatype-json.html 
        
                           Use `hatchet_sdk.{remove_null_unicode_character.__name__}` to sanitize your output if you'd like to remove the character.

This comment has explained the significance of this method, but the implementation of this method has problems.
This PR fixes the value of this check.

Fixes # (issue)

When the object returned normally contains "\u0000", the task will be reset to failure.

Source Code:

hatchet/internal/services/dispatcher/server_v1.go

Lines 557 to 562 in 5bf9f97

    
           if request.EventType == contracts.StepActionEventType_STEP_EVENT_TYPE_COMPLETED { 
        
           	if err := repository.ValidateJSONB([]byte(request.EventPayload), "taskOutput"); err != nil { 
        
           		request.EventPayload = err.Error() 
        
           		request.EventType = contracts.StepActionEventType_STEP_EVENT_TYPE_FAILED 
        
           	} 
        
           }

When the object returned error contains "\u0000", the error content is displayed normally (No check for ValidateJSONB).

Type of change

Bug fix (non-breaking change which fixes an issue)
Documentation change (pure documentation change)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Refactor (non-breaking changes to code which doesn't change any behaviour)
CI (any automation pipeline changes)
Chore (changes which are not directly related to any business logic)
Test changes (add, refactor, improve or change a test)
This change requires a documentation update

What's Changed

Add a list of tasks or features here...

vercel · 2025-10-04T15:52:41Z

@icbd is attempting to deploy a commit to the Hatchet Team on Vercel.

A member of the Team first needs to authorize it.

icbd · 2025-10-04T16:09:17Z

internal/services/dispatcher/server_v1.go

 		s.l.Warn().Msg("retry count is nil, using task's current retry count")
 	}

-	if request.EventType == contracts.StepActionEventType_STEP_EVENT_TYPE_COMPLETED {


Need to double check here

I think we probably only want this for completed / failed?

I don't have a full development environment, so I've only run unit tests on this change.

Could you help run an E2E test to confirm the change? Thanks

yeah, will do tomorrow AM - CI should be a pretty good indicator though 👍

i think the core of this PR is changing \\u0000 to \u0000

Using bytes just improves performance.

I'm not sure if a couple new commits are directly related to this PR.
If it's not relevant, we can submit it first and rebase it.

WDYT？

For me, the issue here isn't that the Validate method isn't working (it is) but we just weren't calling it every place we needed to. Is that not your understanding? It's easy to verify - if you change the string thing to bytes, those python tests will fail and you'll see errors on the engine. I'm not sure why it doesn't work, but it doesn't 🤷

Also, I'm not too worried about the bytes -> string conversion for performance here, although it's a fair point

b0bee2b

This commit is what I want, everything else is optional.

JSON encoder will filter and resolve NUL（JSON encoder transfers \u0000 to \\u0000

\u0000 is one character，means NUL，pg will also reject it.

\\u0000 are six characters，are normal string.

so，you may check the invalid character NUL，instead of a normal str

vercel · 2025-10-07T02:58:23Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Preview	Comments	Updated (UTC)
hatchet-docs	Ready	Preview	Comment	Oct 7, 2025 3:03am
hatchet-v0-docs	Ready	Preview	Comment	Oct 7, 2025 3:03am

mrkaye97 · 2025-10-07T02:59:19Z

pkg/repository/jsonb.go

 	}

-	if strings.Contains(string(jsonb), "\u0000") {
+	if bytes.Contains(jsonb, []byte("\u0000")) {


fyi @icbd this didn't work so I reverted it - not completely sure why though tbh, it seemed fine at first glance

I think this broke a unit test btw, but I validated the behavior with an E2E test. Let me know if you disagree!

icbd added 5 commits October 4, 2025 23:09

Fix ValidateJSONB

b0bee2b

Add TestValidateJSONB

942c0ce

Update jsonb_test.go

f7f0912

Update jsonb.go

e9bfa9e

Update jsonb_test.go

48cd75e

icbd commented Oct 4, 2025

View reviewed changes

Update server_v1.go

381272c

icbd force-pushed the fix/ValidateJSONB branch from 1377b16 to 381272c Compare October 5, 2025 14:00

mrkaye97 added 2 commits October 6, 2025 16:53

fix: e2e test

8ebec8b

fix: revert change from string to bytes

eadf898

mrkaye97 reviewed Oct 7, 2025

View reviewed changes

vercel bot deployed to Preview – hatchet-docs October 7, 2025 03:03 View deployment

fix: rename test

bb45683

	Task outputs cannot contain the unicode null character \\u0000

	Please see this Discord thread: https://discord.com/channels/1088927970518909068/1384324576166678710/1386714014565928992
	Relevant Postgres documentation: https://www.postgresql.org/docs/current/datatype-json.html

	Use `hatchet_sdk.{remove_null_unicode_character.__name__}` to sanitize your output if you'd like to remove the character.

	if request.EventType == contracts.StepActionEventType_STEP_EVENT_TYPE_COMPLETED {
	if err := repository.ValidateJSONB([]byte(request.EventPayload), "taskOutput"); err != nil {
	request.EventPayload = err.Error()
	request.EventType = contracts.StepActionEventType_STEP_EVENT_TYPE_FAILED
	}
	}

Uh oh!

fix: Fix checking value in Golang ValidateJSONB #2383

Are you sure you want to change the base?

fix: Fix checking value in Golang ValidateJSONB #2383

Uh oh!

Conversation

icbd commented Oct 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

What's Changed

Uh oh!

vercel bot commented Oct 4, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

icbd Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

icbd Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vercel bot commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

icbd commented Oct 4, 2025 •

edited

Loading

icbd Oct 7, 2025 •

edited

Loading

icbd Oct 8, 2025 •

edited

Loading

vercel bot commented Oct 7, 2025 •

edited

Loading