-
Notifications
You must be signed in to change notification settings - Fork 12
Open
Description
First of all, thanks for putting up this comprehensive benchmarks on BioML agents.
When I prepared the dataset, all manual tasks (OpenProblem) are having empty leaderboard.csv files due to
@register_data_source("manual")
class ManualDataSource(DataSource):
...
def get_leaderboard(self, source_config: Dict[str, Any]) -> pd.DataFrame:
"""
Manual tasks don't have external leaderboards.
Args:
source_config: Configuration (not used for manual tasks)
Returns:
Empty DataFrame with expected leaderboard columns
"""
# Return empty leaderboard for manual tasks
return pd.DataFrame(columns=['teamName', 'score', 'submissionDate'])
I'm wondering if there's any instruction to to obtain the leaderboard info for manual tasks to reproduce the result in Table 4 of the preprint.
Metadata
Metadata
Assignees
Labels
No labels