Add Concourse a provider with compiler enhancements#527
Open
tristanpoland wants to merge 10 commits intogenesis-community:genesis-pipeline-refactorfrom
Open
Add Concourse a provider with compiler enhancements#527tristanpoland wants to merge 10 commits intogenesis-community:genesis-pipeline-refactorfrom
tristanpoland wants to merge 10 commits intogenesis-community:genesis-pipeline-refactorfrom
Conversation
Introduce two new CI provider implementations and documentation: add Concourse and GitHub Actions provider modules under lib/Genesis/CI/Compiler/Providers. Concourse provider (Concourse.pm) implements the CI trait and currently delegates parsing/generation to Genesis::CI::Legacy and adds deploy/graphviz/describe/accessors. GithubActions.pm is a self-contained provider that parses/validates ci.yml, normalizes git/vault settings, generates GitHub Actions workflow YAML (using YAML::PP), and writes workflows to .github/workflows/ (supports dry-run). Also add a comprehensive README for the new CI compiler architecture and update .gitignore to include /pkg.
Introduce a new CI compilation pipeline and wire it into the CLI. - Add Genesis::CI factory and trait interface plus a Compiler orchestrator (Parser, Validator, ScriptDiscovery, ASTBuilder, AST, PipelineProvider). - Provide provider discovery for 'concourse' and 'github-actions' and update provider modules to support the new pipeline. - Extend bin/genesis with --platform, --output-dir and --skip-vault options (also add platform flag to graph/describe commands) to select the new compiler flow, emit compiled artifacts, and allow local compilation. - Add debug fixtures (concourse pipeline YAML/minimal and AST JSON) and a test t/ci-compiler.t to exercise the compiler. - Update Commands/Pipelines and existing provider implementations to work with the new compiler path. This enables generation of a platform-agnostic AST and platform-specific artifacts, supports inspecting compiled output without deploying, and provides a path to migrate from the legacy single-file ci.yml format.
Introduce a complete .genesis/ci multi-file example for the Genesis CI compiler. Adds README documentation plus example pipeline.yml, integrations.yml, targets.yml, and provider-config overrides for Concourse and GitHub Actions. Includes scripts (deploy and smoke tests) and a scripts/manifest.yml demonstrating explicit script metadata, discovery, inputs/outputs, and environment requirements. This provides a ready-to-use reference for multi-region BOSH deployments, notifications, Vault/Git integration, and provider-specific compilation.
Add Genesis::CI::Compiler::PipelineDescriptor to translate the compiler's source AST into a fully-resolved generic pipeline (resource_types, resources, jobs, groups) that providers consume. Refactor Genesis::CI::Compiler::AST to separate a public "pipeline" layer from an internal "_source" representation, add pipeline accessors (resource_types, pipeline_resources, jobs, groups, graphviz, description), query helpers, and set_pipeline. Update Compiler to resolve the descriptor and attach the generic pipeline before loading providers; extend ASTBuilder to carry pipeline-related triggers/resources. Add design notes (Branch-pipeliens-new.md) and adjust Concourse provider/tests to use the new pipeline boundary. This centralizes pipeline construction and cleanly separates Genesis domain logic from provider serialization.
Rename CLI option usage to accept --provider (alias for --platform) and add a --debug-dir option to dump intermediate compiler artifacts. Normalize provider output in Genesis::CI::Compiler so provider generators that return raw strings are wrapped into a filename->content map using provider->output_files. Implement debug artifact dumping in Commands::Pipelines (requires JSON::PP): write parsed config, AST source, resolved pipeline JSON, graphviz DOT, human description, and provider output files. Update option validation to require --provider for output-dir/skip-vault/debug-dir. Add an end-to-end test t/ci-compile-e2e.t and accompanying t/repos/compile-test fixtures (pipeline, targets, integrations, and zip) to exercise the full compiler pipeline and both Concourse and GitHub Actions providers.
Removed mention of experimental support for GitHub Actions.
Member
Author
|
@dennisjbell https://github.com/tristanpoland/genesis/tree/providers/docs/ci These may make a good summary of the system so far for you |
Clarified comment regarding GitHub Actions provider maintenance status.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces comprehensive documentation for the Genesis CI pipeline system, targeting both operators and developers. The new docs clarify the architecture, data flow, AST structure, and the role of the PipelineDescriptor module, making it much easier to understand and contribute to the CI pipeline codebase. The changes are grouped as follows:
Documentation Structure and Audience
README.mdindocs/cithat explains the CI pipeline system, splits documentation for operators and developers, and links to relevant guides for each audience.Architecture and Code Flow
dev/architecture.mddetailing the dual code paths (legacy and compiler), module map (including a mermaid diagram), file locations, entry points, data flow, provider inheritance, and the Concourse legacy bridge mechanism.AST and PipelineDescriptor Design
dev/ast-and-descriptor.mddescribing the two-layer AST structure (source representation and generic pipeline), accessors and query methods, the role of PipelineDescriptor, resource and job generation, environment variable assembly, file path computation, group generation, workflow data extraction, and AST query methods.