Skip to content

test: Optimize CI by conditionally running schema generation tests#1147

Open
nathan-stender wants to merge 4 commits intomainfrom
ci-conditional-schema-tests-only
Open

test: Optimize CI by conditionally running schema generation tests#1147
nathan-stender wants to merge 4 commits intomainfrom
ci-conditional-schema-tests-only

Conversation

@nathan-stender
Copy link
Collaborator

Summary

Optimize CI performance by only running schema generation tests when relevant files change.

Problem

  • Schema generation test takes 3+ minutes (35% of test suite time)
  • Most PRs don't modify schemas or schema generation code
  • Running this test for every PR wastes CI resources

Solution

Implemented conditional test execution using GitHub Actions path filtering:

  • Schema generation test only runs when schema-related files change
  • Always runs on main branch to ensure consistency
  • Test runs on only one Python version (output is the same for all)

Changes

This PR contains ONLY the workflow file changes:

  1. Added dorny/paths-filter to detect changed files
  2. Excluded schema generation test from regular test runs
  3. Created separate job for schema generation that runs conditionally
  4. Reduced timeout from 15 to 10 minutes for regular tests

Files that trigger schema generation test

  • src/allotropy/allotrope/schemas/** - Schema files
  • src/allotropy/allotrope/schema_parser/** - Schema generation code
  • src/allotropy/allotrope/models/** - Generated models
  • tests/allotrope/schema_parser/generate_schemas_test.py - The test itself
  • pyproject.toml - Dependencies

Expected Impact

  • Most PRs: Save ~3 minutes per Python version (9+ minutes total)
  • Schema-related PRs: No change (test still runs)
  • Overall CI: ~50% faster for typical PRs

Testing

This PR itself demonstrates the optimization:

  • Schema generation test should NOT run (only workflow file changed)
  • All other tests should run normally
  • CI should complete in ~7 minutes instead of ~10 minutes

Related PRs

Notes

  • If schema generation test becomes a required status check, we may need to adjust the approach
  • Consider documenting this behavior in contributing guidelines

Co-Authored-By: Claude Opus 4.1 noreply@anthropic.com

- Schema generation test only runs when relevant files change
- Saves ~3 minutes per Python version (9+ minutes total) for most PRs
- Always runs on main branch to ensure consistency
- Reduces timeout from 15 to 10 minutes for regular tests

The schema generation test accounts for 35% of test suite time but is only
needed when schemas or generation code changes.

Co-Authored-By: Claude Opus 4.1 <noreply@anthropic.com>
@nathan-stender nathan-stender requested review from a team and slopez-b as code owners March 14, 2026 19:20
After code analysis, found schema generation depends on:
- src/allotropy/allotrope/schemas.py (schema loading)
- src/allotropy/constants.py (encoding constants)
- src/allotropy/exceptions.py (error classes)
- src/allotropy/parsers/utils/*.py (utility functions)
- scripts/*.py (entry points)

This ensures schema tests run when any dependency changes.

Co-Authored-By: Claude Opus 4.1 <noreply@anthropic.com>
@nathan-stender
Copy link
Collaborator Author

Update: Added Missing Dependencies

After thorough code analysis, I've identified and added all dependencies that could affect schema generation:

Additional paths now monitored:

  • src/allotropy/allotrope/schemas.py - Schema loading logic
  • src/allotropy/constants.py - Encoding constants used by schema loader
  • src/allotropy/exceptions.py - Exception classes used in validation
  • src/allotropy/parsers/utils/uuids.py - UUID generation for backup files
  • src/allotropy/parsers/utils/values.py - Value assertion utilities
  • scripts/generate_schemas.py - CLI entry point
  • scripts/download_schema.py - Schema download script
  • tests/allotrope/schema_parser/** - All schema parser tests (not just generate_schemas_test.py)

This comprehensive list ensures we won't miss running schema tests when any dependency changes. Most of these are stable utility files that rarely change, so the optimization benefits remain significant.

nathan-stender and others added 2 commits March 14, 2026 15:49
- Replace random_uuid_str() with direct uuid.uuid4() call
- Replace assert_not_none() with inline ValueError check
- Replace AllotropeValidationError with ValueError in path_util
- Add local DEFAULT_ENCODING constant in schemas.py
- Update workflow to remove unneeded dependency paths

This makes the schema_parser module more self-contained and reduces
the paths we need to monitor for CI conditional execution.

Co-Authored-By: Claude Opus 4.1 <noreply@anthropic.com>
- Remove smoke_subset test that was added for performance
- Restore parameterization to run each schema individually
- Keep @pytest.mark.long for optional execution

This provides better granularity for test failures and allows
parallel execution with pytest-xdist.

Co-Authored-By: Claude Opus 4.1 <noreply@anthropic.com>
@nathan-stender
Copy link
Collaborator Author

Latest Updates

1. Removed External Dependencies ✅

Successfully removed all external dependencies from the schema_parser module:

  • Replaced random_uuid_str() with direct uuid.uuid4() call
  • Replaced assert_not_none() with inline ValueError check
  • Replaced AllotropeValidationError with ValueError in path_util
  • Added local DEFAULT_ENCODING constant in schemas.py
  • Removed src/allotropy/constants.py and src/allotropy/parsers/utils/* from monitored paths

2. Restored Test Parameterization ✅

  • Removed the smoke_subset test that was added for performance
  • Restored parameterization to run each schema individually
  • This provides better granularity for test failures and allows parallel execution with pytest-xdist

Final Path Filter

The workflow now monitors these paths for schema generation:

  • src/allotropy/allotrope/schemas/** - Schema JSON files
  • src/allotropy/allotrope/schema_parser/** - Schema generation code
  • src/allotropy/allotrope/models/** - Generated models
  • src/allotropy/allotrope/schemas.py - Schema loading logic
  • src/allotropy/exceptions.py - Exception definitions (still used by schemas.py)
  • scripts/generate_schemas.py - CLI entry point
  • scripts/download_schema.py - Schema download script
  • tests/allotrope/schema_parser/** - Schema parser tests
  • pyproject.toml - Dependencies

The schema_parser module is now more self-contained, reducing false positives for CI triggers.

@nathan-stender nathan-stender changed the title feat: Optimize CI by conditionally running schema generation tests tests: Optimize CI by conditionally running schema generation tests Mar 15, 2026
@nathan-stender nathan-stender changed the title tests: Optimize CI by conditionally running schema generation tests test: Optimize CI by conditionally running schema generation tests Mar 15, 2026
@nathan-stender nathan-stender requested review from ajcariaga16 and removed request for stephenworlow March 15, 2026 17:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant