Add relevance metrics including pruned tokens to MS Marco ranking track #525
Conversation
kderusso
force-pushed
the
kderusso/msmarco-passage-ranking-iter-from-jim
branch
from
d33ff08 to
1574135
Compare
…e the recall against the original weighted terms query
…NDCG based on a smaller version of the queries used to test performance.
kderusso
force-pushed
the
kderusso/msmarco-passage-ranking-iter-from-jim
branch
from
105b19a to
ac79e54
Compare
| "track_total_hits": false | ||
| }, | ||
| { | ||
| "name": "pruned-text-expansion-search-maxwand-disabled", |
| "track_total_hits": false | ||
| }, | ||
| { | ||
| "name": "pruned-weighted-terms-recall-10-10", |
There was a problem hiding this comment.
This file was auto-formatted for correct json indentation. I have left comments starting at the two places where I made changes.
kderusso
changed the title
kderusso
requested review from
Mikep86,
afoucret,
demjened,
ioanatia,
jimczi and
saikatsarkar056
demjened
left a comment
demjened
left a comment
There was a problem hiding this comment.
LGTM with a few minor comments
Mikep86
left a comment
Mikep86
left a comment
There was a problem hiding this comment.
Looking good! Left a few minor comments.
| def generate_pruned_query(field, query_expansion, boost=1.0): | ||
| return { | ||
| "query": { | ||
| "weighted_tokens": { |
There was a problem hiding this comment.
I've tried running the modified track and noticed weighted_tokens query is not available in 8.11.x. Is this planned for 8.13.0? We need to apply the right versioning as per https://esrally.readthedocs.io/en/stable/track.html#custom-track-repositories. The most recent non-master branch in the track repo is 8.11. We need to bump up to 8.13 before merging this change.
There was a problem hiding this comment.
Correct, this is for 8.13.0.
gbanasiak
left a comment
gbanasiak
left a comment
There was a problem hiding this comment.
LGTM. Left small remark regarding _tools/requirements.txt file.
| pytrec_eval | ||
| numpy |
There was a problem hiding this comment.
Should the presence of this file be mentioned in README? Also, please align version pinning with dependencies section.
8 participants
Adds recall metrics for query pruning using the weighted tokens query and precomputed ELSER tokens.