Skip to content

fix: remove duplicate status:2 attempt writes in TreeSearchProbe#1655

Open
Christbowel wants to merge 1 commit intoNVIDIA:mainfrom
Christbowel:fix/wordnet-duplicate-status2
Open

fix: remove duplicate status:2 attempt writes in TreeSearchProbe#1655
Christbowel wants to merge 1 commit intoNVIDIA:mainfrom
Christbowel:fix/wordnet-duplicate-status2

Conversation

@Christbowel
Copy link
Copy Markdown

What this fixes

TreeSearchProbe.probe() was calling the detector internally and then
writing each attempt directly to the report file as status:2. The harness
(harnesses/base.py) independently writes the same attempts as status:2
after probe.probe() returns — this is the standard flow for all probe types.

The result: every attempt produced by TreeSearchProbe appeared once as
status:1 and twice as status:2 in the report, with identical UUIDs
and content. Verified on topic.WordnetControversial which showed 56/56
UUIDs affected.

The fix removes the 4 redundant lines from TreeSearchProbe.probe().
The detector call is kept, TreeSearchProbe needs the detector results
internally to make tree-search decisions (continue exploring or not).
Only the write is removed.

Fixes #1522

Verification

# Before fix: every UUID appears 3 times (1x status:1, 2x status:2)
python -m garak --target_type test.Blank --probes topic.WordnetControversial --generations 1

# After fix: every UUID appears exactly twice (1x status:1, 1x status:2)
python3 -c "
import json
from collections import defaultdict
counts = defaultdict(lambda: defaultdict(int))
with open('<latest_report>.report.jsonl') as f:
    for line in f:
        d = json.loads(line)
        if d.get('entry_type') == 'attempt':
            counts[d['uuid']][d['status']] += 1
dupes = {u: s for u, s in counts.items() if s.get(2, 0) > 1}
print(f'UUIDs with duplicate status:2: {len(dupes)}')  # Expected: 0
print(f'Total UUIDs: {len(counts)}')                   # Expected: 56
"

All probes inheriting from TreeSearchProbe are affected:
topic.WordnetControversial, topic.WordnetBlockedWords,
topic.WordnetAllowedWords.

TreeSearchProbe.probe() was writing each attempt to the report file
after running the detector internally. The harness also writes the
same attempt after probe.probe() returns, resulting in every attempt
appearing once as status:1 and twice as status:2 in the report.

Remove the redundant write from TreeSearchProbe — the harness already
handles status:2 reporting correctly for all probe types.

Fixes NVIDIA#1522

Signed-off-by: christbowel <0xdeadbeef@christbowel.com>
Copy link
Copy Markdown
Collaborator

@jmartin-tech jmartin-tech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this makes sense.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: WordnetControversial creating duplicate status:2 Attempts

2 participants