Skip to content

Run tests in bash scripts on all commands #555

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
bebatut wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Run tests in bash scripts on all commands #555

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ff4d4c8
Run tests in bash scripts on all commands
bebatut Jan 5, 2026
8355bdc
Uncomment workflow tests in CI
bebatut Jan 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 23 additions & 23 deletions .github/workflows/run_tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,26 +98,26 @@ jobs:
run: |
bash sources/bin/get_community_tutorials.sh test

# test-workflows:
# runs-on: ubuntu-latest
# # This job runs tests for workflows.
# # It performs the same checks as the other jobs, ensuring
# # quality for workflow scripts.
# if: github.event.pull_request.head.repo.full_name == github.repository || github.ref == 'refs/heads/dev'
# strategy:
# matrix:
# python-version: ['3.11']
# steps:
# - name: Checkout
# uses: actions/checkout@v6
# - uses: actions/setup-python@v6
# with:
# python-version: ${{ matrix.python-version }}
# - name: Install requirements
# run: python -m pip install -r requirements.txt
# - name: Workflow extraction
# run: |
# bash sources/bin/extract_all_workflows.sh test
# - name: Workflow filtering
# run: |
# bash sources/bin/get_community_workflows.sh test
test-workflows:
runs-on: ubuntu-latest
# This job runs tests for workflows.
# It performs the same checks as the other jobs, ensuring
# quality for workflow scripts.
if: github.event.pull_request.head.repo.full_name == github.repository || github.ref == 'refs/heads/dev'
strategy:
matrix:
python-version: ['3.11']
steps:
- name: Checkout
uses: actions/checkout@v6
- uses: actions/setup-python@v6
with:
python-version: ${{ matrix.python-version }}
- name: Install requirements
run: python -m pip install -r requirements.txt
- name: Workflow extraction
run: |
bash sources/bin/extract_all_workflows.sh test
- name: Workflow filtering
run: |
bash sources/bin/get_community_workflows.sh test
3 changes: 0 additions & 3 deletions sources/bin/extract_all_tools.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,8 +53,5 @@ else
--all-yml 'communities/all/resources/tools.yml' \
--all-workflows "communities/all/resources/workflows.json" \
--all-tutorials "communities/all/resources/tutorials.json"

# ln -s "./../../communities/all/resources/tools.yml" "./_data/tools.yml"

fi

26 changes: 12 additions & 14 deletions sources/bin/extract_all_tutorials.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ if [ ! -z $1 ]
then
python sources/bin/extract_gtn_tutorials.py \
extract \
--all "communities/all/resources/test_tutorials.json" \
--tools "communities/all/resources/test_tools.json" \
--all "communities/all/resources/tutorials.json" \
--tools "communities/all/resources/tools.json" \
--api $PLAUSIBLE_API_KEY \
--test
else
Expand All @@ -14,19 +14,17 @@ else
--all "communities/all/resources/tutorials.json" \
--tools "communities/all/resources/tools.json" \
--api $PLAUSIBLE_API_KEY
fi

python sources/bin/extract_gtn_tutorials.py \
filter \
--all "communities/all/resources/tutorials.json" \
--yml "communities/all/resources/tutorials.yml" \
--filtered "communities/all/resources/tutorials.tsv"
python sources/bin/extract_gtn_tutorials.py \
filter \
--all "communities/all/resources/tutorials.json" \
--yml "communities/all/resources/tutorials.yml" \
--filtered "communities/all/resources/tutorials.tsv"

# ln -s "./../../communities/all/resources/tutorials.yml" "./_data/tutorials.yml"

python sources/bin/create_interactive_table.py \
--input "communities/all/resources/tutorials.tsv" \
--template "sources/data/interactive_table_template.html" \
--output "communities/all/resources/tutorials.html"
fi
python sources/bin/create_interactive_table.py \
--input "communities/all/resources/tutorials.tsv" \
--template "sources/data/interactive_table_template.html" \
--output "communities/all/resources/tutorials.html"


2 changes: 0 additions & 2 deletions sources/bin/extract_all_workflows.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,4 @@ else
--all "communities/all/resources/workflows.json" \
--yml "communities/all/resources/workflows.yml" \
--tools "communities/all/resources/tools.json"

# ln -s "./../../communities/all/resources/workflows.yml" "./_data/workflows.yml"
fi
97 changes: 36 additions & 61 deletions sources/bin/get_community_tools.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,76 +3,51 @@
# stop on error
set -e

if [[ ! -z $1 && $1 == "test" ]]; then
python sources/bin/extract_galaxy_tools.py \
filter \
--all "communities/all/resources/test_tools.json" \
--categories "communities/microgalaxy/metadata/categories" \
--filtered "communities/microgalaxy/resources/tools_filtered_by_ts_categories.json" \
--status "communities/microgalaxy/metadata/tool_status.tsv"

python sources/bin/extract_galaxy_tools.py \
curate \
--filtered "communities/microgalaxy/resources/tools_filtered_by_ts_categories.json" \
--status "communities/microgalaxy/metadata/tool_status.tsv" \
--curated "communities/microgalaxy/resources/curated_tools.tsv" \
--wo-biotools "communities/microgalaxy/resources/curated_tools_wo_biotools.tsv" \
--w-biotools "communities/microgalaxy/resources/curated_tools_w_biotools.tsv"\
--yml "communities/microgalaxy/resources/curated_tools.yml"
for com_data_fp in communities/* ; do
if [[ -d "$com_data_fp" && ! -L "$com_data_fp" ]]; then
community=`basename "$com_data_fp"`

mkdir -p _data/communities/microgalaxy/
ln -sf ../../../communities/microgalaxy/resources/curated_tools.yml _data/communities/microgalaxy/curated_tools.yml

python sources/bin/create_wordcloud.py \
--input "communities/microgalaxy/resources/curated_tools.tsv" \
--name-col "Suite ID" \
--stat-col "Suite runs (last 5 years) on main servers" \
--wordcloud_mask "sources/data/usage_stats/wordcloud_mask.png" \
--output "communities/microgalaxy/resources/tools_wordcloud.png"
if [[ ! -z $1 && $1 == "test" && "$community" != "microgalaxy" ]]; then
continue
fi;

else
for com_data_fp in communities/* ; do
if [[ -d "$com_data_fp" && ! -L "$com_data_fp" ]]; then
community=`basename "$com_data_fp"`
if [[ "$community" != "all" && -f "communities/$community/metadata/categories" ]]; then
echo "$community";
mkdir -p "communities/$community/resources"

if [[ "$community" != "all" && -f "communities/$community/metadata/categories" ]]; then
echo "$community";
mkdir -p "communities/$community/resources"
python sources/bin/extract_galaxy_tools.py \
filter \
--all "communities/all/resources/tools.json" \
--categories "communities/$community/metadata/categories" \
--filtered "communities/$community/resources/tools_filtered_by_ts_categories.json" \
--status "communities/$community/metadata/tool_status.tsv"

if [[ -e "communities/$community/metadata/tool_status.tsv" ]]; then
python sources/bin/extract_galaxy_tools.py \
filter \
--all "communities/all/resources/tools.json" \
--categories "communities/$community/metadata/categories" \
curate \
--filtered "communities/$community/resources/tools_filtered_by_ts_categories.json" \
--status "communities/$community/metadata/tool_status.tsv"

if [[ -e "communities/$community/metadata/tool_status.tsv" ]]; then
python sources/bin/extract_galaxy_tools.py \
curate \
--filtered "communities/$community/resources/tools_filtered_by_ts_categories.json" \
--status "communities/$community/metadata/tool_status.tsv" \
--curated "communities/$community/resources/curated_tools.tsv" \
--wo-biotools "communities/$community/resources/curated_tools_wo_biotools.tsv" \
--w-biotools "communities/$community/resources/curated_tools_w_biotools.tsv"\
--yml "communities/$community/resources/curated_tools.yml"

if [[ -e "communities/$community/resources/curated_tools.yml" ]]; then
mkdir -p _data/communities/$community/
ln -sf ../../../communities/$community/resources/curated_tools.yml _data/communities/$community/curated_tools.yml
fi;
--status "communities/$community/metadata/tool_status.tsv" \
--curated "communities/$community/resources/curated_tools.tsv" \
--wo-biotools "communities/$community/resources/curated_tools_wo_biotools.tsv" \
--w-biotools "communities/$community/resources/curated_tools_w_biotools.tsv"\
--yml "communities/$community/resources/curated_tools.yml"

if [[ -e "communities/$community/resources/curated_tools.yml" ]]; then
mkdir -p _data/communities/$community/
ln -sf ../../../communities/$community/resources/curated_tools.yml _data/communities/$community/curated_tools.yml
fi;

if [[ -e "communities/$community/resources/curated_tools.tsv" ]]; then
python sources/bin/create_wordcloud.py \
--input "communities/$community/resources/curated_tools.tsv" \
--name-col "Suite ID" \
--stat-col "Suite runs (last 5 years) on main servers" \
--wordcloud_mask "sources/data/usage_stats/wordcloud_mask.png" \
--output "communities/$community/resources/tools_wordcloud.png"
fi;
if [[ -e "communities/$community/resources/curated_tools.tsv" ]]; then
python sources/bin/create_wordcloud.py \
--input "communities/$community/resources/curated_tools.tsv" \
--name-col "Suite ID" \
--stat-col "Suite runs (last 5 years) on main servers" \
--wordcloud_mask "sources/data/usage_stats/wordcloud_mask.png" \
--output "communities/$community/resources/tools_wordcloud.png"
fi;
fi;
fi;
done
fi
fi;
done


70 changes: 27 additions & 43 deletions sources/bin/get_community_workflows.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,54 +1,38 @@
#!/usr/bin/env bash

if [ ! -z $1 ]
then
python sources/bin/extract_galaxy_workflows.py \
filter \
--all "communities/all/resources/test_workflows.json" \
--filtered "communities/microgalaxy/resources/test_workflows.json" \
--tsv-filtered "communities/microgalaxy/resources/test_workflows.tsv" \
--tags "communities/microgalaxy/metadata/workflow_tags" \
--status "communities/microgalaxy/metadata/test_workflow_status.tsv"
for com_data_fp in communities/* ; do
if [[ -d "$com_data_fp" && ! -L "$com_data_fp" ]]; then
community=`basename "$com_data_fp"`

python sources/bin/extract_galaxy_workflows.py \
curate \
--filtered "communities/microgalaxy/resources/test_workflows.json" \
--status "communities/microgalaxy/metadata/test_workflow_status.tsv" \
--curated "communities/microgalaxy/resources/test_curated_workflows.tsv" \
--tsv-curated "communities/microgalaxy/resources/test_curated_workflows.tsv" \
--yml "communities/microgalaxy/resources/test_curated_workflows.yml"
else
for com_data_fp in communities/* ; do
if [[ -d "$com_data_fp" && ! -L "$com_data_fp" ]]; then
community=`basename "$com_data_fp"`
if [[ ! -z $1 && $1 == "test" && "$community" != "microgalaxy" ]]; then
continue
fi;



if [[ "$community" != "all" && -f "communities/$community/metadata/workflow_tags" ]]; then
echo "$community";
python sources/bin/extract_galaxy_workflows.py \
filter \
--all "communities/all/resources/workflows.json" \
--filtered "communities/$community/resources/tag_filtered_workflows.json" \
--tsv-filtered "communities/$community/resources/tag_filtered_workflows.tsv" \
--tags "communities/$community/metadata/workflow_tags" \
--status "communities/$community/metadata/workflow_status.tsv"

if [[ "$community" != "all" && -f "communities/$community/metadata/workflow_tags" ]]; then
echo "$community";
if [[ -f "communities/$community/metadata/workflow_status.tsv" ]]; then
python sources/bin/extract_galaxy_workflows.py \
filter \
--all "communities/all/resources/workflows.json" \
curate \
--filtered "communities/$community/resources/tag_filtered_workflows.json" \
--tsv-filtered "communities/$community/resources/tag_filtered_workflows.tsv" \
--tags "communities/$community/metadata/workflow_tags" \
--status "communities/$community/metadata/workflow_status.tsv"

if [[ -f "communities/$community/metadata/workflow_status.tsv" ]]; then
python sources/bin/extract_galaxy_workflows.py \
curate \
--filtered "communities/$community/resources/tag_filtered_workflows.json" \
--status "communities/$community/metadata/workflow_status.tsv" \
--curated "communities/$community/resources/curated_workflows.json" \
--tsv-curated "communities/$community/resources/curated_workflows.tsv" \
--yml "communities/$community/resources/curated_workflows.yml"
--status "communities/$community/metadata/workflow_status.tsv" \
--curated "communities/$community/resources/curated_workflows.json" \
--tsv-curated "communities/$community/resources/curated_workflows.tsv" \
--yml "communities/$community/resources/curated_workflows.yml"

if [[ -f "communities/$community/metadata/curated_workflows.yml" ]]; then
mkdir -p _data/communities/$community/
ln -sf ../../../communities/$community/resources/curated_workflows.yml _data/communities/$community/curated_workflows.yml
fi;
if [[ -f "communities/$community/metadata/curated_workflows.yml" ]]; then
mkdir -p _data/communities/$community/
ln -sf ../../../communities/$community/resources/curated_workflows.yml _data/communities/$community/curated_workflows.yml
fi;
fi;
fi;
done
fi
fi;
done