Conversation
rohitpaulk
left a comment
There was a problem hiding this comment.
Let's add failure test cases
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
| [33m[tester::#DK5] [0m[94mRunning ./your_shell.sh[0m | ||
| [33m[your-program] [0m[0m$ sleep 100 &[0m | ||
| [33m[your-program] [0m[0m[1] 2375[0m | ||
| [33m[tester::#DK5] [0m[92m✓ Received entry for the started job[0m |
There was a problem hiding this comment.
"entry" here doesn't quite make sense - would just say "Output includes job number with PID." or something like that
| [33m[tester::#DK5] [0m[92m✓ Received entry for the started job[0m | ||
| [33m[your-program] [0m[0m$ jobs[0m | ||
| [33m[your-program] [0m[0m[1]+ Running sleep 100 &[0m | ||
| [33m[tester::#DK5] [0m[92m✓ Received 1 entry for the running jobs[0m |
There was a problem hiding this comment.
| �[33m[tester::#DK5] �[0m�[92m✓ Received 1 entry for the running jobs�[0m | |
| �[33m[tester::#DK5] �[0m�[92m✓ Received 1 entry for the running job�[0m |
(singular)
| [33m[your-program] [0m[0m$ jobs[0m | ||
| [33m[your-program] [0m[0m[1]- Running sleep 100 &[0m | ||
| [33m[your-program] [0m[0m[2]+ Running sleep 200 &[0m | ||
| [33m[tester::#DK5] [0m[92m✓ Received 2 entries for the running jobs[0m |
There was a problem hiding this comment.
I think we're overusing the "Received" word here, maybe the word "found" is shorter and more descriptive of what the assertion is doing? "Found 2 entries matching running jobs", or just "✔︎ 2 entries match running jobs".
| [33m[tester::#DK5] [0m[94mRunning ./your_program.sh[0m | ||
| [33m[your-program] [0m[0m$ sleep 100 &[0m | ||
| [33m[your-program] [0m[0m[1] 2549[0m | ||
| [33m[tester::#DK5] [0m[92m✓ Received entry for the started job[0m |
There was a problem hiding this comment.
Same note as above - let's not use the word entry here
| [33m[your-program] [0m[0m[1]- Running sleep 100 &[0m | ||
| [33m[tester::#DK5] [0m[91m^ Line does not match expected value.[0m | ||
| [33m[tester::#DK5] [0m[91m[32mExpected:[0m "[1]+ Running sleep 100 &"[0m | ||
| [33m[tester::#DK5] [0m[91m[31mReceived:[0m "[1]- Running sleep 100 &"[0m |
There was a problem hiding this comment.
This is fine for now, but I can imagine a lot of confusion stemming from the usage of spaces here - if the user users a single space between each but has another mistake, it might look like we're complaining about spaces when we're complaining about something else.
Can't think of a neat way to solve this, but I think one way would be to use LLMs. We could expose this via the stage harness and find a mechanism to do restricted prompts (i.e. we have some guarantees over what kind of output they can emit). Then we could generate error messages like "missing space between - and Running"
There was a problem hiding this comment.
Noted - Adding this to linear.
Note
Medium Risk
Adds new background-jobs stages and updates
jobsoutput matching rules (markers/format and trailing&), which can change pass/fail behavior for multiple stages and fixtures. CI test execution is also switched to a Docker-based runner, which may introduce environment-related test differences.Overview
Adds Background Jobs Stage #3 (
jd6) and Stage #4 (dk5) to validatejobsoutput for one and multiple running background jobs, including correct+/-markers and ordering.Tightens and refactors
JobsBuiltinResponseTestCaseexpectations (renamesExpectedOutputItemstoExpectedOutputEntries, requires the trailing&forRunningentries while still allowing a regex fallback) and updates fixtures/scenarios to cover incorrect marker and output-format failures.Updates stage registration (
tester_definition.go), course metadata (course_definition.yml), Makefile stage list, and switches the GitHub Actionstestjob to run tests via./docker_test/docker_test.shinstead of a native Go setup.Written by Cursor Bugbot for commit 21336ff. This will update automatically on new commits. Configure here.