Pipestat messaging update suggestions

[nsheff]

There's some confusing language I think we should clean up.

here's a vanilla looper check:

 looper check
Looper version: 2.0.0a2
Command: check
Using looper config (.looper.yaml).
No config key in Project, or reading project from dict
Traceback (most recent call last):
  File "/home/ns5bc/.local/bin/looper", line 8, in <module>
    sys.exit(main_cli())
             ^^^^^^^^^^
  File "/home/ns5bc/.local/lib/python3.11/site-packages/looper/cli_pydantic.py", line 357, in main_cli
    main()
  File "/home/ns5bc/.local/lib/python3.11/site-packages/looper/cli_pydantic.py", line 353, in main
    return run_looper(args, parser, test_args=test_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ns5bc/.local/lib/python3.11/site-packages/looper/cli_pydantic.py", line 319, in run_looper
    raise PipestatConfigurationException("check")

Then I added this to the looper config:

pipestat:
  results_file_path: results.yaml

And now it says:

looper check
Looper version: 2.0.0a2
Command: check
Using looper config (.looper.yaml).
No config key in Project, or reading project from dict
Pipeline name mismatch detected. Pipeline interface: PEPATAC  Output schema: None  Defaulting to pipeline_interface value.
Initializing results file '/project/shefflab/brickyard/results_pipeline/tlgl_atac/PEPATAC/results.yaml'
File does not exist, but create_file is true. Creating...
     'PEPATAC' pipeline status summary     
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃  Status   ┃    Jobs count/total jobs    ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
└───────────┴─────────────────────────────┘

---

Some things here confusing me:

• Why does it say "No config key in Project, or reading project from dict". Can this be reworded? I'm not getting anything useful from this message.
• Looper documentation/tutorial does not actually specify how to configure: "Pipestat must be configured for command check". I think it's lumped in with description of results
• Is the pipeline name thing important? Can we explain what this means somewhere: "Pipeline name mismatch detected. Pipeline interface: PEPATAC Output schema: None Defaulting to pipeline_interface value."
• To what is "File does not exist, but create_file is true. Creating..." referring? To "results.yaml file"?
• Why is looper check creating results.yaml? Shouldn't that be at looper run?
• This is not finding the jobs. I do not know how to configure pipestat to actually find the jobs and check/report on them correctly. This is using PEPATAC and it is creating _failed flags.

[nsheff]

I guess the biggest issue here is this: I cannot figure out how to configure pipestat such that looper check works, and there appears to be little documentation on that. it just says "use looper check", but that doesn't work.

Do you have to configure it before running the pipeline? I ran the pipeline before configuring pipestat. It still created flag files. now how do I check them?

[donaldcampbelljr]

I've added a bit more clarity on using looper check in tandem with pipestat to the documentation.

"No config key in project" is being tracked here: pepkit/peppy#497

In the above example, looper check was creating results.yaml files because the check command creates a pipestatmanager object for each pipeline interface to retrieve the flag statuses based on a flag directory. I assume it was being passed a filepath for a file backend. If it is passed a path and the file backend does not exist at the given path, it will create one automatically.