fix(marshmallow): add compatibility layer for Flask-AppBuilder with marshmallow 4.x #35920

kevin-lann · 2025-10-31T01:47:19Z

SUMMARY

Upgrades marshmallow version to >= 4.0

Flask-AppBuilder 5.0.0 auto-generates schema fields for SQLAlchemy relationships that don't exist in marshmallow 4.x's stricter field validation, causing KeyError during schema initialization.
Our solution patches marshmallow's Schema._init_fields method to detect missing fields that Flask-AppBuilder 5.0.0 auto-generates, and then adds these missing fields dynamically as raw fields with appropriate flags & retry initialization until all missing fields are resolved.
So, for example, it can handle multiple fields in a sequence (permission_id, view_menu_id, db_id, chart_id, dashboard_id, user_id). All of this logic for this bullet point is encapsulated in the marshmallow_fix.py module, whose job is to specifically address the Flask-AppBuilder 5.0.0 incompatibility with marshmallow 4.x, and then app.py applies the patch by importing it from marshmallow_fix.py.
All changes in files other than app.py and marshmallow_fix.py are minor convention/syntax changes that are required for the Marshmallow upgrade to >= 4.0.0 (See https://marshmallow.readthedocs.io/en/stable/upgrading.html for more info)

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

BEFORE: Upon pinning marshmallow>=4 in pyproject.tomland initializing the superset app, an initialization error occurs.

AFTER: App runs without error

TESTING INSTRUCTIONS

Start superset application, see that no marshmallow related errors show up.

Run tests with docker compose run --rm superset python -m pytest tests/unit_tests/test_marshmallow_compatibility.py -v

Unit tests created:

test_patch_marshmallow_for_flask_appbuilder_applies_patch - Verifies the patch function correctly modifies the Schema class
test_patch_functionality_with_real_schema_creation - Tests actual schema creation works with the patch
test_patch_handles_schema_with_no_fields - Ensures empty schemas work properly
test_raw_field_creation_and_configuration - Validates Raw field configuration
test_print_function_can_be_mocked - Tests logging functionality can be tested
test_keyerror_exception_handling - Verifies KeyError exception handling
test_schema_declared_fields_manipulation - Tests field manipulation capabilities
test_flask_appbuilder_field_names_list - Validates the field names our fix handles
test_patch_function_is_callable - Ensures the patch function works correctly
test_marshmallow_schema_basic_functionality - Confirms basic marshmallow functionality still works

ADDITIONAL INFORMATION

Has associated issue: Fixes issue Make Superset compatible with marshmallow>=4.0.0 #33162
Required feature flags:
Changes UI
Includes DB Migration (follow approval process in SIP-59)
- Migration is atomic, supports rollback & is backwards-compatible
- Confirm DB migration upgrade and downgrade tested
- Runtime estimates and downtime expectations provided
Introduces new feature or API
Removes existing feature or API

Credit to

@Austin-X @Eyang0612 @PDOracle @MasahisaSekita

…le to be logged in

This reverts commit f220397.

…ask-AppBuilder 5.0.0 and marshmallow 4.x

korbit-ai · 2025-10-31T01:47:24Z

Based on your review schedule, I'll hold off on reviewing this PR until it's marked as ready for review. If you'd like me to take a look now, comment /korbit-review.

Your admin can change your review schedule in the Korbit Console

kevin-lann · 2025-11-03T14:04:45Z

/korbit-review

korbit-ai

I've completed my review and didn't find any issues.

Check out our docs on how you can make Korbit work best for you and your team.

Loving Korbit!? Share us on LinkedIn Reddit and X

korbit-ai

Review by Korbit AI

Korbit automatically attempts to detect when you fix issues in new commits.

Category	Issue	Status
	Redundant JSON serialization without caching ▹ view
	Sanitized data stored but not consumed ▹ view
	Missing import for Length validator ▹ view
	Module-level patch execution adds import overhead ▹ view
	Marshmallow patch applied at module level ▹ view
	Inefficient retry loop with redundant field processing ▹ view
	Unmanaged Monkey-Patching of External Library ▹ view
	Improper Logging Mechanism ▹ view
	Unreliable KeyError message parsing ▹ view
	Unsafe declared_fields modification ▹ view

Files scanned

File Path	Reviewed
superset/advanced_data_type/schemas.py	✅
superset/marshmallow_compatibility.py	✅
superset/themes/schemas.py	✅
superset/app.py	✅
superset/db_engine_specs/databend.py	✅
superset/reports/schemas.py	✅
superset/charts/schemas.py	✅

Explore our documentation to understand the languages and file types we support and the files we ignore.

Check out our docs on how you can make Korbit work best for you and your team.

Loving Korbit!? Share us on LinkedIn Reddit and X

superset/themes/schemas.py

superset/charts/schemas.py

superset/app.py

superset/marshmallow_compatibility.py

kevin-lann · 2025-11-15T01:01:29Z

@mistercrunch Hi, could you re-run workflows on this PR? Also lmk if you have any concerns/suggestions about the compatibility changes here. Much appreciated 🙏

rusackas · 2025-11-17T22:07:11Z

re-running workflows 🤞

rusackas · 2025-11-17T22:08:38Z

CC @dpgaspar @Antonio-RiveroMartnez

Copilot

Pull Request Overview

This PR upgrades marshmallow from version 3.x to 4.x to resolve compatibility issues with Flask-AppBuilder 5.0.0. The upgrade addresses a critical initialization error where Flask-AppBuilder auto-generates schema fields for SQLAlchemy relationships that don't pass marshmallow 4.x's stricter field validation.

Key Changes:

Introduces a compatibility layer that patches marshmallow's Schema._init_fields to dynamically add missing auto-generated fields as Raw fields
Updates all marshmallow schemas to follow 4.x conventions (validator signatures, field parameters, metadata usage)
Upgrades marshmallow to >=4.0.0 and marshmallow-sqlalchemy to 1.4.2

Reviewed Changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
superset/marshmallow_compatibility.py	New module that patches marshmallow to handle Flask-AppBuilder compatibility by dynamically adding missing fields
superset/app.py	Applies the marshmallow compatibility patch during application initialization
tests/unit_tests/test_marshmallow_compatibility.py	Comprehensive unit tests for the compatibility module
superset/themes/schemas.py	Migrates validators to accept `**kwargs` parameter and replaces `self.context` with `ContextVar`
superset/reports/schemas.py	Updates validator signatures and replaces `default` with `load_default`
superset/models/helpers.py	Updates validator signature to include `**kwargs`
superset/databases/schemas.py	Updates validator signature to include `**kwargs`
superset/charts/schemas.py	Migrates field parameters: `description` to `metadata`, `minLength` to `validate=Length()`, `min` to `validate=Range()`
superset/db_engine_specs/databend.py	Migrates `description` to `metadata` and `default` to `dump_default`
superset/advanced_data_type/schemas.py	Updates JSON schema with `dump_default` (needs correction)
pyproject.toml	Updates marshmallow dependency from `>=3.0, <4` to `>=4`
requirements/base.in	Updates marshmallow-sqlalchemy from `<1.4.1` to `>=1.4.2`
requirements/base.txt	Locks marshmallow at 4.0.0 and marshmallow-sqlalchemy at 1.4.2
requirements/development.txt	Locks marshmallow at 4.0.0 and marshmallow-sqlalchemy at 1.4.2

superset/themes/schemas.py

superset/marshmallow_compatibility.py

superset/themes/schemas.py

superset/advanced_data_type/schemas.py

…etc.)

…anup Feature/marshmallow upgrade cleanup

kevin-lann · 2025-11-24T20:39:59Z

@rusackas The tests/checks should be all fixed now (hopefully). Could you rerun once again?

codecov · 2025-11-28T19:51:45Z

Codecov Report

❌ Patch coverage is 77.27273% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.55%. Comparing base (1e4bc6e) to head (33c0814).
⚠️ Report is 479 commits behind head on master.

Files with missing lines	Patch %	Lines
superset/marshmallow_compatibility.py	78.26%	2 Missing and 3 partials ⚠️
superset/themes/schemas.py	70.00%	3 Missing ⚠️
superset/app.py	60.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #35920       +/-   ##
===========================================
+ Coverage        0   67.55%   +67.55%     
===========================================
  Files           0      637      +637     
  Lines           0    46847    +46847     
  Branches        0     5084     +5084     
===========================================
+ Hits            0    31646    +31646     
- Misses          0    13912    +13912     
- Partials        0     1289     +1289

Flag	Coverage Δ
hive	`43.77% <75.00%> (?)`
mysql	`67.10% <77.27%> (?)`
postgres	`67.15% <77.27%> (?)`
presto	`47.36% <75.00%> (?)`
python	`67.51% <77.27%> (?)`
sqlite	`66.77% <77.27%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…anup Fix ruff and mympy checks

kevin-lann · 2025-11-28T21:47:45Z

The above commit should fix the remaining checks 33c0814

Austin-X added 9 commits October 6, 2025 18:16

My Local Setup (Revert this commit later)

f220397

Upgrade to Marshmallow >= 4 with Superset able to be booted up and ab…

663922c

…le to be logged in

Revert "My Local Setup (Revert this commit later)"

7b4f545

This reverts commit f220397.

Reset feature flags back to their original status

c841166

Apply the compatibility patch to address incompatibilities between Fl…

bc755de

…ask-AppBuilder 5.0.0 and marshmallow 4.x

Revert initialization\__init__.py to its initial state

bf9fc91

Fix whitespace in superset\initialization\__init__.py

dd9186a

Add unit tests

2a4339a

Rename "marshmallow_fix" to "marshmallow_compatibility"

72fd0b6

pull-request-size bot added the size/L label Oct 31, 2025

Move patch fn to app.py

ca17fbb

kevin-lann marked this pull request as ready for review November 3, 2025 14:03

dosubot bot added the change:backend Requires changing the backend label Nov 3, 2025

empty

c0fe5d4

korbit-ai bot approved these changes Nov 3, 2025

View reviewed changes

korbit-ai bot suggested changes Nov 3, 2025

View reviewed changes

kevin-lann mentioned this pull request Nov 3, 2025

Make Superset compatible with marshmallow>=4.0.0 #33162

Open

3 tasks

rusackas requested a review from mistercrunch November 3, 2025 18:29

Fix validates decorator

fd7d995

rusackas requested a review from Copilot November 17, 2025 22:07

Copilot started reviewing on behalf of rusackas November 17, 2025 22:08 View session

Copilot finished reviewing on behalf of rusackas November 17, 2025 22:11

Copilot AI reviewed Nov 17, 2025

View reviewed changes

Eyang0612 added 2 commits November 22, 2025 22:06

Apply JSON Schema Change

84cddb3

Style: Add Typing + Remove Redundancy in Code Logic

3fcc002

kevin-lann and others added 3 commits November 23, 2025 19:21

Allow optional fields to load as None

e586e16

Apply some formatting changes (e.g. make all lines <= 88 chars long, …

4fe8938

…etc.)

Merge pull request #2 from kevin-lann/feature/marshmallow-upgrade-cle…

5ab4b7a

…anup Feature/marshmallow upgrade cleanup

dpgaspar self-requested a review November 24, 2025 20:44

kevin-lann added 2 commits November 28, 2025 16:40

Fix ruff and mympy checks

2e6a6d6

Merge pull request #3 from kevin-lann/feature/marshmallow-upgrade-cle…

33c0814

…anup Fix ruff and mympy checks

fix(marshmallow): add compatibility layer for Flask-AppBuilder with marshmallow 4.x #35920

Are you sure you want to change the base?

fix(marshmallow): add compatibility layer for Flask-AppBuilder with marshmallow 4.x #35920

Conversation

kevin-lann commented Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

SUMMARY

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

Credit to

Uh oh!

korbit-ai bot commented Oct 31, 2025

Uh oh!

kevin-lann commented Nov 3, 2025

Uh oh!

korbit-ai bot left a comment

Choose a reason for hiding this comment

Uh oh!

korbit-ai bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Review by Korbit AI

Korbit automatically attempts to detect when you fix issues in new commits.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kevin-lann commented Nov 15, 2025

Uh oh!

rusackas commented Nov 17, 2025

Uh oh!

rusackas commented Nov 17, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kevin-lann commented Nov 24, 2025

Uh oh!

codecov bot commented Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

kevin-lann commented Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kevin-lann commented Oct 31, 2025 •

edited

Loading

korbit-ai bot left a comment •

edited

Loading

codecov bot commented Nov 28, 2025 •

edited

Loading

kevin-lann commented Nov 28, 2025 •

edited

Loading