Skip to content

Missing leaderboards for manual tasks #32

@Alexiland

Description

@Alexiland

First of all, thanks for putting up this comprehensive benchmarks on BioML agents.

When I prepared the dataset, all manual tasks (OpenProblem) are having empty leaderboard.csv files due to

@register_data_source("manual")
class ManualDataSource(DataSource):
    ...
    def get_leaderboard(self, source_config: Dict[str, Any]) -> pd.DataFrame:
        """
        Manual tasks don't have external leaderboards.
        
        Args:
            source_config: Configuration (not used for manual tasks)
            
        Returns:
            Empty DataFrame with expected leaderboard columns
        """
        # Return empty leaderboard for manual tasks
        return pd.DataFrame(columns=['teamName', 'score', 'submissionDate']) 

I'm wondering if there's any instruction to to obtain the leaderboard info for manual tasks to reproduce the result in Table 4 of the preprint.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions