GitHub Action - Workflow template for Python library

🤖 Test state:

Usage scenario / state

This is a GitHub Action workflow template for Python library project.

Overview | Workflow template usages

---

Overview

In development of Python library, it configures the mostly same CI/CD processes again and again. That's the reason why I consider and implement this project. This project has some workflow templates for reusing in GitHub Action CI/CD processes so that it could reach some benefits:

• Be more clear what thing to do in a job.
• Be more clear and simpler of entire CI/CD work flow.
• Be easier to read and configure at configurations of GitHub Action work flow.
• It has greatly improved the GitHub Action configuration to be more reusable in multiple different projects (git repositories).

Workflow template usages

The usage of each workflow template.

• rw_get_tests.yaml
• rw_run_test.yaml
• rw_poetry_run_test.yaml
• rw_run_test_with_multi_py_versions.yaml
• rw_poetry_run_test_with_multi_py_versions.yaml
• rw_organize_test_cov_reports.yaml
• rw_upload_test_cov_report.yaml
• rw_pre-building_test.yaml
• rw_checking_deployment_state.yaml
• rw_build_git-tag_and_create_github-release.yaml
• rw_push_pypi.yaml

---

rw_get_tests.yaml

• Description: Prepare the test items.
• Options:

option name	data type	optional or required	function content
shell_path	string	Optional, Default value is ./scripts/ci/get-all-tests.sh	The path shell script for getting the testing items.
shell_arg	string	Required	Input arguments of the shell script.
use_customized_shell	boolean	Optional, Default value is false	Whether it should use the customized shell script or not.

• Output:

  • all_test_items: All the test items it would run.

• How to use it?

  • Use default shell script (recommended)

    If we want to use default shell script to auto-scan all tests, it only needs to give a shell script argument which is the directory path of test code:

      prepare-testing-items_unit-test:
    #    name: Prepare all unit test items
        uses: Chisanan232/GitHub-Action-Template-Python/.github/workflows/rw_get_tests.yaml@master
        with:
          shell_arg: test/unit_test/

    And it would get all tests you need. And the keyword to get this workflow output result is all_test_items.

  • Use customized shell script

    Before use this workflow, it should prepare a shell script for getting the testing items.

      prepare-testing-items_unit-test:
    #    name: Prepare all unit test items
        uses: Chisanan232/GitHub-Action-Template-Python/.github/workflows/rw_get_tests.yaml@master
        with:
          shell_path: scripts/ci/get-unit-test-paths.sh
          shell_arg: unix
          use_customized_shell: true

    And we could get this workflow output result via keyword all_test_items.

---

rw_run_test.yaml

• Description: Run testing by specific type with all test items via PyTest and generate its testing coverage report (it would save reports by actions/upload-artifact).
• Options:

option name	data type	optional or required	function content
runtime_os	string	Optional, Default value is ubuntu-latest	The OS to use for runtime environment.
python_version	string	Optional, Default value is 3.11	The Python version to run the test in workflow.
test_type	string	Required	The testing type. In generally, it only has 2 options: unit-test and integration-test.
all_test_items_paths	string	Required	The target paths of test items under test.
setup_http_server	string	Optional, Default value is false	If it's true, it would set up and run HTTP server for testing.
http_server_host	string	Optional, Default value is 0.0.0.0	The host IPv4 address of HTTP server.
http_server_port	string	Optional, Default value is 12345	The port number of HTTP server.
http_server_app_module	string	Optional, Default value is app	The module path of HTTP server.
http_server_enter_point	string	Optional, Default value is app	The object about the web application.
debug_mode	boolean	Optional, Default value is false	For debug, so it's matrix would only has one os: ubuntu-22.04 & one python-version: 3.10.

• Output:

No, but it would save the testing coverage reports to provide after-process to organize and record.

Upload-Artifact name	description
coverage	The test coverage report which be generated by PyTest, and it's recorded after run test done. The file name format would be .coverage.<inputs.test type>.<matrix.runtime os>-<matrix.python version>

• How to use it?

  run_unit-test:
#    name: Run all unit test items
    needs: prepare-testing-items_unit-test
    uses: Chisanan232/GitHub-Action-Template-Python/.github/workflows/rw_run_test.yaml@master
    with:
      python_version: '3.10'
      test_type: unit-test
      all_test_items_paths: ${{needs.prepare-testing-items_unit-test.outputs.all_test_items}}
      setup_http_server: true
      http_server_host: 0.0.0.0
      http_server_port: 30303
      http_server_app_module: test._http_server.app
      http_server_enter_point: app

Please take a look of option all_test_items_paths. You could find that it get the input result of previous workflow prepare-testing-items_unit-test via below way:

${{needs.prepare-testing-items_unit-test.outputs.all_test_items}}

Character part needs.prepare-testing-items_unit-test means it want to get something context info from needed workflow prepare-testing-items_unit-test, and the context it wants to get is outputs. And be more essentially, what outputs it want to use? It's all_test_items. Do you discover this keyword is provided by previous workflow? That is all testing items.

NOTE:

It also has another reusable workflow names poetry_run_test_via_pytest.yaml. If your Python project manages by Poetry, it recommends you to use this one replaces rw_run_test.yaml. The usage and running details are mostly same as rw_run_test.yaml. But, workflow poetry_run_test_via_pytest.yaml only supports Python version 3.8 up.

---

rw_poetry_run_test.yaml

• Description: Same working with workflow rw_run_test.yaml, but this workflow would run test via Poetry.

• Difference between workflows rw_run_test.yaml and poetry_run_test_via_pytest.yaml:

Workflow	Running way
rw_run_test.yaml	Command lines like pip, python, etc
poetry_run_test_via_pytest.yaml	Use poetry feature or run command lines in it

And for the poetry reusable workflow, it has 3 parameters another doesn't have:

Workflow	Running way
test_working_directory	The working directory for test running.
install_dependency_with_group	Install the dependency by Poetry configuration with dependency group setting. This parameter receive the dependency group naming. (multiple values allowed)
install_dependency_without_group	Install the dependency by Poetry configuration without dependency group setting. This parameter receive the dependency group naming. (multiple values allowed)

---

rw_run_test_with_multi_py_versions.yaml

• Description: Almost same working with workflow rw_run_test.yaml, but it would run test with multiple Python versions with multiple runtime environment OS.

---

rw_poetry_run_test_with_multi_py_versions.yaml

• Description: Almost same working with workflow rw_poetry_run_test.yaml, but it would run test with multiple Python versions with multiple runtime environment OS.

• Difference between workflows rw_run_test_with_multi_py_version.yaml and rw_poetry_run_test_with_multi_py_versions.yaml

Workflow	Running way	Support Python version
rw_run_test_with_multi_py_version.yaml	Command lines like pip, python, etc	3.6 - 3.11
rw_poetry_run_test_with_multi_py_versions.yaml	Use poetry feature or run command lines in it	3.8 - 3.11

---

rw_organize_test_cov_reports.yaml

• Description: Organize all the testing coverage reports which be generated in different runtime OS with Python version. (it would save reports by actions/upload-artifact).
• Options:

option name	data type	optional or required	function content
test_type	string	Required	The testing type. In generally, it only has 2 options: unit-test and integration-test.

• Output:

No, but it would save the testing coverage reports to provide after-process to organize and record.

Upload-Artifact name	description
test_coverage_report	The handled test coverage report (.coverage file). It's file name format would be .coverage.<inputs.test type>.
test_coverage_xml_report	The handled test coverage report (.xml file). It's file name format would be coverage_<inputs.test type>.xml.

• How to use it?

  unit-test_codecov:
#    name: Organize and generate the testing report and upload it to Codecov
    needs: run_unit-test
    uses: Chisanan232/GitHub-Action-Template-Python/.github/workflows/rw_organize_test_cov_reports.yaml@master
    with:
      test_type: unit-test

It would upload the organized report via actions/upload-artifact. And it doesn't support customize options of actions/upload-artifact currently.

---

rw_upload_test_cov_report.yaml

• Description: Upload the testing coverage reports to Codecov.
• Options:

It has 2 different types option could use:

General option:

option name	data type	optional or required	function content
download_path	string	Optional. Default value is './'.	The path to download testing coverage reports via actions/download-artifact.
test_type	string	Required	The testing type. In generally, it only has 3 options: unit-test, integration-test and all-type.
upload-to-codecov	boolean	Optional. Default value is false.	If it's true, it would upload testing coverage report for Codecov (https://codecov.io).
codecov_flags	string	Optional. Default value is empty string.	The flags of the testing coverage report for Codecov. This option would be required if upload-to-codecov is true.
codecov_name	string	Optional. Default value is empty string.	The name of the testing coverage report for Codecov. This option would be required if upload-to-codecov is true.
upload-to-coveralls	boolean	Optional. Default value is false.	If it's true, it would upload testing coverage report for Coveralls (https://coveralls.io).
upload-to-codacy	boolean	Optional. Default value is false.	If it's true, it would upload testing coverage report for Codacy (https://app.codacy.com/).

Secret option:

option name	option is optional or required	function content
codecov_token	Optional. Default value is empty string.	The API token for uploading testing coverage report to Codecov. This option would be required if upload-to-codecov is true.
coveralls_token	Optional. Default value is empty string.	The API token for uploading testing coverage report to Coveralls. This option would be required if upload-to-coveralls is true.
codacy_token	Optional. Default value is empty string.	The API token for uploading testing coverage report to Codacy. This option would be required if upload-to-codacy is true.

• Output:

Nothing.

• How to use it?

  ⚠️ Before run this reusable workflow, please make sure testing coverage report is ready.
  

  ❔What format of test coverage report it could use? Different platform would need different format. But basically, it only accepts 2 types: .coverage & .xml.

  👀 This reusable work flow would check the input parameters first. The specific platform token shouldn't be empty where uploading flag is true.

  • Uploading test coverage report to Codecov (accepted report format: .xml)

    In Codecov case, it would need other 2 necessary options codecov_flags & codecov_name.

      codecov_finish:
    #    name: Organize and generate the testing report and upload it to Codecov
        needs: [unit-test_codecov]
        uses: Chisanan232/GitHub-Action-Template-Python/.github/workflows/rw_upload_test_cov_report.yaml@master
        secrets:
          codecov_token: ${{ secrets.CODECOV_TOKEN }}
        with:
          test_type: unit-test
          upload-to-codecov: true
          codecov_flags: unittests # required if upload-to-codecov is true
          codecov_name: smoothcrawler-cluster_github-actions_test # required if upload-to-codecov is true

    The badge it generates:

    

  • Uploading test coverage report to Coveralls (accepted report format: .coverage)

    In Coveralls case, the Python tool coveralls only accept .coverage type report so that it would do coverage process again (integrate all test types report into one report).

      codecov_finish:
    #    name: Organize and generate the testing report and upload it to Coveralls
        needs: [unit-test_codecov]
        uses: Chisanan232/GitHub-Action-Template-Python/.github/workflows/rw_upload_test_cov_report.yaml@master
        secrets:
          coveralls_token: ${{ secrets.COVERALLS_TOKEN }}
        with:
          test_type: unit-test
          upload-to-coveralls: true

    The badge it generates:

    

  • Uploading test coverage report to Codacy (accepted report format: .xml)

    In Codacy case, please use CODACY_PROJECT_TOKEN, not CODACY_API_TOKEN.

      codecov_finish:
    #    name: Organize and generate the testing report and upload it to Codacy
        needs: [unit-test_codecov]
        uses: Chisanan232/GitHub-Action-Template-Python/.github/workflows/rw_upload_test_cov_report.yaml@master
        secrets:
          codacy_token: ${{ secrets.CODACY_PROJECT_TOKEN }}
        with:
          test_type: unit-test
          upload-to-codacy: true

    The badge it generates:

    

---

rw_pre-building_test.yaml

• Description: Test package by simple way after running setup.py script to install Python package
• Options:

option name	data type	optional or required	function content
build-type	string	Optional, Default value is setuptools	The way CI should run the pre-build test.
python_package_name	string	Required	The Python package name.
test_shell	string	Optional, Default value is empty string	Run command line(s) for testing.
test_shell_in_python	string	Optional, Default value is empty string	Run Python code as script for testing.
test_python_script	string	Optional, Default value is empty string	Run a Python script for testing the Python package.

• Output:

No, nothing at all.

• How to use it?

  rw_pre-building_test:
#    name: Check about it could work finely by installing the Python package with setup.py file
    uses: ./.github/workflows/rw_pre-building_test.yaml
    with:
      build-type: setuptools
      python_package_name: Test-GitHub-Action-workflow
      test_shell_in_python: from test_gh_workflow import sample
      test_shell: echo 'Echo something for testing'
      test_python_script: ./scripts/ci/test/test_pgk_install.py

---

rw_checking_deployment_state.yaml

• Description: Before truly deployment, e.g., release and push Python package to PyPI, it would check the release relative information to judge whether it should release or not.
• Requirement: The target project must has one Python module named __pkg_info__.py. And the content of it should be like following:

__version__ = "0.2.3"

• How to trigger it?:

It would use the package info module to get the current Package version info. And it would also try to install the Python package from PyPI to get the latest version of it. Finally, it would use this 2 version info to compare whether the project current version is same as the version in PyPI or not. If it is, it would skip. But if it isn't, it would start to run deployment process to release.

Note

For the newborn project, it won't be released to PyPI before, it would set its software version as 0.0.0 at default.

• Trigger condition:

  1. Python file __pkg_info__.py has been updated.
  2. The property __version__ has been updated. (aka the software version value)
  3. The software version value is different with the version in PyPI.

• Options:

option name	data type	optional or required	function content
working-directory	string	Optional, Default value is ./	The working directory for this CI running.
library-name	string	Required	The target library name for checking the version info.
library-source-code-path	string	Required_	The source code path of target library to check.
library-default-version	string	Optional, Default value is 0.0.0	The default value of software version if it cannot get the software version info from installed library.

• Output:

Yes, it has running result output. It would output the updating state about whether the version has been updated or not. And the CI workflow after-process could use this output result to judge whether it should run the deployment process or not.

Upload-Artifact name	description
version_update_state	The version update state. It only has 2 states: VERSION UPDATE and NO VERSION UPDATE.

• How to use it?

  • cd.yaml usage case:

  name: CD
  
  on:
    # Run the deployment about publishing the Python source code to PyPI.
    push:
      branches:
        - "master"
      paths:
  #     This deployment workflow would only be triggered by file change of module *__pkg_info__* because it has the package version info.
  #      - ".github/workflows/cd.yaml"    # For test or emergency scenario only
        - "**/__pkg_info__.py"
  
  jobs:
    check_version-state:
  #    name: Check the version update state
      uses: ./.github/workflows/rw_checking_deployment_state.yaml
      with:
        library-name: your-python-package-name
        library-source-code-path: ./your_source_code_directory
  
    push_python_pkg_to_pypi:
  #    name: Check about it could work finely by installing the Python package with setup.py file
      needs: check_version-state
      if: ${{ needs.check_version-state.outputs.version_update_state == 'VERSION UPDATE' }}
      uses: ./.github/workflows/rw_push_pypi.yaml
      with:
        build-type: poetry
        release-type: ${{ needs.build_git-tag_and_create_github-release.outputs.python_release_version }}
        push-to-PyPI: official
      secrets:
        PyPI_user: ${{ secrets.PYPI_USERNAME }}
        PyPI_token: ${{ secrets.PYPI_PASSWORD }}

The badge it generates:

---

rw_build_git-tag_and_create_github-release.yaml

• Description: Build a git tag on a specific commit in every git branch. And create GitHub release if current git branch is 'master'.
• Options:

option name	data type	optional or required	function content
project_type	string	Required	Different project type would get the software version info in different way. Currently, it only has 2 options: python-package or github-action-reusable-workflow
debug_mode	boolean	Optional, Default value is false	It would run the tasks as log message, doesn't essentially run feature if this option is true.
project_name	string	Optional, Default value is empty string	The project name.
software_version_format	string	Optional, Default value is empty string	The format of software version.

• Output:

Yes, it has running result output. It would output the version which could provide after-process to verify what thing it should do, e.g., release or not.

Workflow output	description
python_release_version	Python project release version info. It only has 2 types value: Official-Release or Pre-Release.
github-action_reusable_workflow_release_version	Python project release version info.

• How to use it?

  • Python package usage case:

    rw_build_git-tag_and_create_github-release:
  #    name: Build git tag and GitHub release if it needs for Python package project
      needs: [coveralls_finish, codacy_finish]
      uses: ./.github/workflows/rw_build_git-tag_and_create_github-release.yaml
      with:
        project_type: python-package
        project_name: test_gh_workflow
        software_version_format: general-3
        debug_mode: true

  • GitHub Action reusable workflow usage case:

    rw_build_git-tag_and_create_github-release:
  #    name: Build git tag and GitHub release if it needs for GitHub Action reusable workflow project
      needs: [coveralls_finish, codacy_finish]
      uses: ./.github/workflows/rw_build_git-tag_and_create_github-release.yaml
      with:
        project_type: github-action-reusable-workflow
        debug_mode: true

The badge it generates:

---

rw_push_pypi.yaml

• Description: Compile source code and push the Python package to PyPI. (Official release the Python package)
• Options:

It has 2 different types option could use:

General option:

option name	data type	optional or required	function content
build-type	string	Optional, Default value is setuptools	Which way CI should to package the Python code. It has 2 type options: 'setuptools' or 'poetry'.
release-type	string	Required	The type of release processing. It has 2 type options: 'Official-Release' or 'Pre-Release'. It won't push the package to PyPI if it's 'Pre-Release'.
push-to-PyPI	string	Optional, Default value is empty string	Push Python package to official PyPI or test PyPI. It has 2 type options: 'official' or 'test'.

Secret option:

option name	option is optional or required	function content
PyPI_user	Required	The username of PyPI.
PyPI_token	Required	The password token of PyPI.

• Output:

No, nothing at all.

• How to use it?

  push_python_pkg_to_pypi:
#    name: Upload the Python package files which has been compiled to PyPi
    uses: ./.github/workflows/rw_push_pypi.yaml
    with:
      build-type: setuptools
      release-type: 'Official-Release'
      push-to-PyPI: test
    secrets:
      pypi_user: ${{ secrets.PYPI_USERNAME }}
      pypi_token: ${{ secrets.PYPI_PASSWORD }}