Reference evaluation code?

[hassony2]

Hi @vis-nlp,

Thank you for the useful benchmark and insightful paper !
Could you point me to a reference implementation for the relaxed accuracy computation ?
In particular I am interested in the logic that allows to discriminate between numeric and non-numeric answer types (based on casting behavior ?)

Have a great day !

Yana

[AhmedMasryKU]

Hi @hassony2
Thanks for your interest in our work. We are currently updating the repo and adding all the missing codes.
Yeah, as far as I remember, we used the casting behavior to discriminate between the numeric and non-numeric answers for our ChartQA dataset.