-
Notifications
You must be signed in to change notification settings - Fork 2
feat: [REL-10772] Implement Langchain provider for online evals #64
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
…nd update imports for clarity
…update async tracking for OpenAI metrics
| self._client.track( | ||
| "$ld:ai:agent:function:multiple", | ||
| context, | ||
| agent_count, | ||
| agent_count | ||
| ) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tracking the config ids that were pulled here would be a good facet to add imo
| all_variables = {} | ||
| if variables: | ||
| all_variables.update(variables) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit/suggestion: all_variables = variables or {}
| self._logger.warning('Judge configuration must include messages') | ||
| return None | ||
|
|
||
| if random.random() > sampling_rate: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this should be random.random() <= sampling_rate
A sampling rate of 0.1 should be 10% of the invocations, since the range of random() is 0-1, that leaves 90% of the results as greater than 0.1. So I think we want lte to capture 10%.
A sample rate of 1 should be 100%, so <= will always return true
Summary
This PR adds LangChain provider support to the LaunchDarkly AI Python SDK
The Python SDK follows a monorepo structure with packages split across:
This architecture allows users to install just what they need mirroring our TypeScript SDK's modular design.
Before:
After: