Automated Integration Test Goldens Update from CI #5931
Conversation
…b7b-b36f-2afb9e3d06bd)
Summary of ChangesHello @datacommons-robot-author, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request automatically updates an integration test golden file. The changes reflect an evolution in the system's query processing and statistical variable matching capabilities, leading to revised expected outputs for a specific query. Reviewers should examine these updated golden values to ensure the new behavior is correct and intended. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request contains an automated update to a golden test file, reflecting a change in the NLP model's output. The main change is the removal of 'number of' from the query, which is a reasonable preprocessing step. However, I've noted a potential regression in how the query is split into semantic parts, which may affect the quality of results for similar queries. Please see the specific comment for details.
| { | ||
| "AggCosineScore": 0.8315, | ||
| "AggCosineScore": 0.8564, | ||
| "DelimBased": false, | ||
| "Parts": [ | ||
| { | ||
| "CosineScore": [ | ||
| 0.83118, | ||
| 0.831, | ||
| 0.82552, | ||
| 0.81955, | ||
| 0.81456, | ||
| 0.80907, | ||
| 0.80864, | ||
| 0.80744, | ||
| 0.8003, | ||
| 0.79858, | ||
| 0.78782 | ||
| 0.89767 | ||
| ], | ||
| "QueryPart": "number of poor hispanic", | ||
| "QueryPart": "poor", | ||
| "SV": [ | ||
| "Count_Person_BelowPovertyLevelInThePast12Months_HispanicOrLatino", | ||
| "Count_Person_HispanicOrLatino", | ||
| "Count_Person_Female_AbovePovertyLevelInThePast12Months_HispanicOrLatino", | ||
| "Count_Person_AbovePovertyLevelInThePast12Months_HispanicOrLatino", | ||
| "Count_Household_WithoutFoodStampsInThePast12Months_HispanicOrLatino", | ||
| "Count_Person_Male_AbovePovertyLevelInThePast12Months_BlackOrAfricanAmericanAlone", | ||
| "Count_Person_Male_BelowPovertyLevelInThePast12Months_HispanicOrLatino", | ||
| "Count_Person_Female_BelowPovertyLevelInThePast12Months_HispanicOrLatino", | ||
| "Count_Person_NoHealthInsurance_HispanicOrLatino", | ||
| "Count_Person_WithDisability_HispanicOrLatino", | ||
| "Count_Person_15OrMoreYears_Separated_HispanicOrLatino" | ||
| "dc/topic/Poverty" | ||
| ] | ||
| }, | ||
| { | ||
| "CosineScore": [ | ||
| 0.83183, | ||
| 0.80299 | ||
| 0.81515, | ||
| 0.7886, | ||
| 0.77756, | ||
| 0.77521 | ||
| ], | ||
| "QueryPart": "women phd", | ||
| "QueryPart": "hispanic women phd", | ||
| "SV": [ | ||
| "Count_Person_25OrMoreYears_EducationalAttainmentDoctorateDegree_Female", | ||
| "Count_Person_25OrMoreYears_Female_DoctorateDegree_AsFractionOf_Count_Person_25OrMoreYears_Female" | ||
| "dc/06f0jf8xvzw4f", | ||
| "Count_Person_Female_HispanicOrLatino", | ||
| "dc/3w039ndqy7qv1", | ||
| "dc/topic/HispanicOrLatinoFemalePopulationByAge" | ||
| ] | ||
| } | ||
| ] | ||
| }, | ||
| { | ||
| "AggCosineScore": 0.8259, | ||
| "AggCosineScore": 0.8118, | ||
| "DelimBased": false, | ||
| "Parts": [ | ||
| { | ||
| "CosineScore": [ | ||
| 0.83667, | ||
| 0.79727 | ||
| 0.79171, | ||
| 0.78522, | ||
| 0.77956, | ||
| 0.77731, | ||
| 0.77682, | ||
| 0.77496, | ||
| 0.76645, | ||
| 0.74689, | ||
| 0.74408, | ||
| 0.74334, | ||
| 0.74237 | ||
| ], | ||
| "QueryPart": "number of poor", | ||
| "QueryPart": "poor hispanic", | ||
| "SV": [ | ||
| "Count_Person_BelowPovertyLevelInThePast12Months_HispanicOrLatino", | ||
| "Count_Person_Female_AbovePovertyLevelInThePast12Months_HispanicOrLatino", | ||
| "Count_Person_Male_AbovePovertyLevelInThePast12Months_BlackOrAfricanAmericanAlone", | ||
| "Count_Person_Male_BelowPovertyLevelInThePast12Months_HispanicOrLatino", | ||
| "Count_Person_AbovePovertyLevelInThePast12Months_HispanicOrLatino", | ||
| "Count_Household_WithoutFoodStampsInThePast12Months_HispanicOrLatino", | ||
| "Count_Person_Female_BelowPovertyLevelInThePast12Months_HispanicOrLatino", | ||
| "dc/topic/Poverty", | ||
| "Count_Person_Rural_BelowPovertyLevelInThePast12Months" | ||
| "Count_Person_HispanicOrLatino", | ||
| "Count_Person_NoHealthInsurance_HispanicOrLatino", | ||
| "dc/topic/PovertyByRace" | ||
| ] | ||
| }, | ||
| { | ||
| "CosineScore": [ | ||
| 0.81515, | ||
| 0.7886, | ||
| 0.77756, | ||
| 0.77521 | ||
| 0.83183, | ||
| 0.80299 | ||
| ], | ||
| "QueryPart": "hispanic women phd", | ||
| "QueryPart": "women phd", | ||
| "SV": [ | ||
| "dc/06f0jf8xvzw4f", | ||
| "Count_Person_Female_HispanicOrLatino", | ||
| "dc/3w039ndqy7qv1", | ||
| "dc/topic/HispanicOrLatinoFemalePopulationByAge" | ||
| "Count_Person_25OrMoreYears_EducationalAttainmentDoctorateDegree_Female", | ||
| "Count_Person_25OrMoreYears_Female_DoctorateDegree_AsFractionOf_Count_Person_25OrMoreYears_Female" | ||
| ] | ||
| } | ||
| ] | ||
| }, |
There was a problem hiding this comment.
The updated model now ranks a query split of ("poor", "hispanic women phd") higher than ("poor hispanic", "women phd"). While the first part poor matches the general dc/topic/Poverty with a high score, this split breaks the semantic unit of 'poor hispanic'. The second candidate, which keeps 'poor hispanic' together, seems more appropriate for the user's query but receives a lower aggregate score. This might indicate a regression in the model's ability to handle multi-word concepts, potentially favoring more generic but less accurate interpretations.
1 participant
This pull request updates the golden files automatically via Cloud Build. Please review the changes carefully. Cloud Build Log