feat: complete refactor to remove traceloop and speakeasy #154

josh-hhai · 2025-08-28T02:47:17Z

Completely replaces the old sdk with datamodel-codegen for the openapi spec, and removes traceloop and implemente opentlemetry. Greatly increases testability. Delivers no sdk code change support for bring your own instrumnetor.

TODO: Need to rework github workflows to suit the new model.
TODO: Look at adding specific environment testing, i.e. aws lambda

Note

Adds comprehensive specification packages (SRD/specs/tasks/implementation) and validation analyses for the Workflow Engine, HoneyHive SDK Docs MCP (v1 and v2), and Documentation P0 fixes.

Specifications Added:
- workflow-engine-design:
  - Adds full design spec for phase-gated Workflow Engine.
- honeyhive-sdk-docs-mcp (v1):
  - Adds SRD, specs, tasks, implementation, and validation documents.
- honeyhive-sdk-docs-mcp-v2:
  - Adds production-hardened SRD/specs/tasks/implementation with concurrency safety, dependency pinning, and failure-mode testing.
- documentation-p0-fixes:
  - Adds SRD/specs/tasks/implementation for documentation restructuring and quality fixes.
Supporting Artifacts:
- Validation/improvements analyses, revision summaries, and processing metadata under supporting-docs/.
Scope:
- Documentation/spec additions only; no source code changes included.

^{Written by Cursor Bugbot for commit e20b584. This will update automatically on new commits. Configure here.}

dhruv-hhai · 2025-08-28T16:50:04Z

Check for backwards compatibility of new API client against the old. Refer to docs scripts for reference + the speakeasy SDK ref docs in main.

dhruv-hhai · 2025-08-28T16:52:06Z

Check for backwards compatibility on environment variable references + add support for experiment harness related env vars

HoneyHiveClient does not read standard environment variables

dhruv-hhai · 2025-08-28T16:53:01Z

Enable verbose flag on the HoneyHiveClient init for customer to debug API errors.

dhruv-hhai · 2025-08-28T16:55:10Z

Apply pydantic models on SDK caller params directly instead of inside the function.

dhruv-hhai · 2025-08-28T16:55:46Z

Add a SSL cert override option on the HoneyHiveClient init with the env var for httpx.

Add a SSL no verify flag on HoneyHiveClient

dhruv-hhai · 2025-08-28T16:57:10Z

Standardize python error handling middleware (context handler of some kind) and add that in all client wrapper classes.

dhruv-hhai · 2025-08-28T16:58:39Z

Add docstrings on SDK functions

dhruv-hhai · 2025-08-28T17:00:05Z

Investigate pydantic alternative

dhruv-hhai · 2025-08-28T17:01:05Z

Ensure there's an async method for each API call wrapper.

dhruv-hhai · 2025-08-28T17:02:25Z

Nit: Add argument builders for async callers

dhruv-hhai · 2025-08-28T17:04:23Z

Investigate an alternative to data model codegen to also include client codegen

dhruv-hhai · 2025-08-28T17:10:28Z

Drop the HoneyHiveLogger class / evaluate moving the logger repo into this repo

dhruv-hhai · 2025-08-28T17:11:25Z

Drop project from tracer init

dhruv-hhai · 2025-08-28T17:11:42Z

Drop unused imports

dhruv-hhai · 2025-08-28T17:13:04Z

Move tracer away from singleton. We want to support multiple sessions within the same runtime.

dhruv-hhai · 2025-08-28T17:13:22Z

Default session name should be the file name where the tracer is initialized.

dhruv-hhai · 2025-08-28T17:13:41Z

Check if TracerProvider is initialized before initializing. We should support not being the main provider if someone already has a tracer provider set.

dhruv-hhai · 2025-08-28T17:14:03Z

OTLP export is enabled by default

dhruv-hhai · 2025-08-28T17:15:19Z

Provide a flag to disable batching on span exporter + support simple span processor

Lambda mode flag to auto-set these configs

dhruv-hhai · 2025-08-28T17:17:36Z

Provide ability to set custom session_id via an argument on the tracer init.

dhruv-hhai · 2025-08-28T17:18:12Z

Auto-generate UUIDv4 for session_id even if session start fails

dhruv-hhai · 2025-08-28T17:18:49Z

Pick up session_id from baggage context if available by default

dhruv-hhai · 2025-08-28T17:19:40Z

Setup baggage context should also check if pre-existing baggage has the main association properties set

dhruv-hhai · 2025-08-28T17:23:48Z

The context manager sets span attributes as honeyhive.* it should also support traceloop.association.properties.*

dhruv-hhai · 2025-08-28T17:24:25Z

Centralize the enrich_session implementation to the tracer class. Don't do create_event do update_event.

dhruv-hhai · 2025-08-28T17:25:25Z

Enrich session should use the baggage to fetch the session_id, not the tracer (since we aren't on a singleton model)

dhruv-hhai · 2025-08-28T17:26:18Z

Drop configure_otlp_exporter

dhruv-hhai · 2025-08-28T17:29:04Z

Migrate evaluate from the old SDK. Look at the multi-threading patterns + tracer initializations to see how we want to support multiple sessions in one runtime.

Co-Authored-By: Dhruv <dhruv@honeyhive.ai>

…ion-support feat: add async function support to evaluate function

github-actions · 2025-12-10T22:47:11Z

📚 Documentation Preview Built

✅ Documentation preview is ready!

📦 Download Preview

Download documentation artifact

🔍 How to Review

Download the artifact from the link above
Extract the files
Open index.html in your browser

✅ Validation Status

API validation: ✅ Passed
Build process: ✅ Successful
Import tests: ✅ All imports working

Preview generated for PR #154

- Add typed models for MetricDetail, DatapointResult, DatapointMetric, MetricDatapoints - Update AggregatedMetrics to use details array instead of dynamic keys - Update print_table() to use typed MetricDetail objects - Update get_run_result() to parse datapoints into DatapointResult objects - Fix datapoint rendering to use typed attributes instead of hasattr checks This fixes the issue where the metrics table was printing without values because the SDK expected dynamic metric keys but the backend returns metrics in a 'details' array format per the OpenAPI spec. Co-Authored-By: Dhruv <dhruv@honeyhive.ai>

- Add pylint disable=no-member comments for Pydantic field access - Add unit tests for new typed models (MetricDetail, DatapointResult, etc.) - Add unit tests for print_table() with details array format - Add integration test to validate typed models against real API Co-Authored-By: Dhruv <dhruv@honeyhive.ai>

Co-Authored-By: Dhruv <dhruv@honeyhive.ai>

… AggregatedMetrics Co-Authored-By: Dhruv <dhruv@honeyhive.ai>

…dels Co-Authored-By: Dhruv <dhruv@honeyhive.ai>

…-table-printing fix: update metrics table printing to handle backend response format

github-actions · 2025-12-11T19:35:33Z

📚 Documentation Preview Built

✅ Documentation preview is ready!

📦 Download Preview

Download documentation artifact

🔍 How to Review

Download the artifact from the link above
Extract the files
Open index.html in your browser

✅ Validation Status

API validation: ✅ Passed
Build process: ✅ Successful
Import tests: ✅ All imports working

Preview generated for PR #154

- Add instrumentors parameter to evaluate() and run_experiment() functions - Accept list of instrumentor factory functions that create new instances per datapoint - Each datapoint gets its own tracer and instrumentor instance for proper trace routing - Instrumentors are initialized with tracer_provider before processing and uninstrumented after - Graceful error handling for instrumentor initialization and cleanup Co-Authored-By: Dhruv <dhruv@honeyhive.ai>

Co-Authored-By: Dhruv <dhruv@honeyhive.ai>

…rs-evaluate feat: add instrumentors support to evaluate function

github-actions · 2025-12-11T20:49:47Z

📚 Documentation Preview Built

✅ Documentation preview is ready!

📦 Download Preview

Download documentation artifact

🔍 How to Review

Download the artifact from the link above
Extract the files
Open index.html in your browser

✅ Validation Status

API validation: ✅ Passed
Build process: ✅ Successful
Import tests: ✅ All imports working

Preview generated for PR #154

github-actions · 2025-12-11T21:37:45Z

📚 Documentation Preview Built

✅ Documentation preview is ready!

📦 Download Preview

Download documentation artifact

🔍 How to Review

Download the artifact from the link above
Extract the files
Open index.html in your browser

✅ Validation Status

API validation: ✅ Passed
Build process: ✅ Successful
Import tests: ✅ All imports working

Preview generated for PR #154

github-actions · 2025-12-11T21:41:56Z

📚 Documentation Preview Built

✅ Documentation preview is ready!

📦 Download Preview

Download documentation artifact

🔍 How to Review

Download the artifact from the link above
Extract the files
Open index.html in your browser

✅ Validation Status

API validation: ✅ Passed
Build process: ✅ Successful
Import tests: ✅ All imports working

Preview generated for PR #154

OTLP Json exporter support

github-actions · 2025-12-11T22:54:32Z

📚 Documentation Preview Built

✅ Documentation preview is ready!

📦 Download Preview

Download documentation artifact

🔍 How to Review

Download the artifact from the link above
Extract the files
Open index.html in your browser

✅ Validation Status

API validation: ✅ Passed
Build process: ✅ Successful
Import tests: ✅ All imports working

Preview generated for PR #154

josh-hhai requested review from dhruv-hhai and sunnybak August 28, 2025 02:47

devin-ai-integration bot and others added 2 commits December 10, 2025 22:31

docs: add changelog, documentation, and tests for async function support

e894227

Co-Authored-By: Dhruv <dhruv@honeyhive.ai>

Merge pull request #159 from honeyhiveai/devin/1765405300-async-funct…

1cf21d7

…ion-support feat: add async function support to evaluate function

dhruv-hhai temporarily deployed to github-pages December 10, 2025 22:47 — with GitHub Actions Inactive

devin-ai-integration bot and others added 7 commits December 10, 2025 23:05

fix: add pylint disable comments for Pydantic field access in tests

31a2f74

Co-Authored-By: Dhruv <dhruv@honeyhive.ai>

style: apply black formatting to modified files

4a9d286

Co-Authored-By: Dhruv <dhruv@honeyhive.ai>

fix: support both legacy model_extra and new details array formats in…

7b85ad0

… AggregatedMetrics Co-Authored-By: Dhruv <dhruv@honeyhive.ai>

docs: add changelog entries and API reference for typed experiment mo…

f65c39f

…dels Co-Authored-By: Dhruv <dhruv@honeyhive.ai>

Merge pull request #160 from honeyhiveai/devin/1765407684-fix-metrics…

6d24cb5

…-table-printing fix: update metrics table printing to handle backend response format

dhruv-hhai temporarily deployed to github-pages December 11, 2025 19:35 — with GitHub Actions Inactive

devin-ai-integration bot and others added 5 commits December 11, 2025 19:43

chore: remove unused Union import

1c98ef7

Co-Authored-By: Dhruv <dhruv@honeyhive.ai>

docs: add ADK instrumentor example for evaluate function

760af4a

Co-Authored-By: Dhruv <dhruv@honeyhive.ai>

docs: add changelog, documentation, and tests for instrumentors feature

781fd62

Co-Authored-By: Dhruv <dhruv@honeyhive.ai>

Merge pull request #161 from honeyhiveai/devin/1765482008-instrumento…

ea166ae

…rs-evaluate feat: add instrumentors support to evaluate function

dhruv-hhai temporarily deployed to github-pages December 11, 2025 20:49 — with GitHub Actions Inactive

minor example fix

f754909

dhruv-hhai temporarily deployed to github-pages December 11, 2025 21:37 — with GitHub Actions Inactive

bump up to rc6

2caf95e

dhruv-hhai temporarily deployed to github-pages December 11, 2025 21:41 — with GitHub Actions Inactive

codehruv and others added 2 commits December 11, 2025 17:50

enable OTLP JSON exporter support

1eeafde

Merge pull request #162 from honeyhiveai/json-exporter-support

1b75b58

OTLP Json exporter support

dhruv-hhai deployed to github-pages December 11, 2025 22:54 — with GitHub Actions View deployment

feat: complete refactor to remove traceloop and speakeasy #154

Are you sure you want to change the base?

feat: complete refactor to remove traceloop and speakeasy #154

Uh oh!

Conversation

josh-hhai commented Aug 28, 2025 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dhruv-hhai commented Aug 28, 2025

Uh oh!

dhruv-hhai commented Aug 28, 2025

Uh oh!

dhruv-hhai commented Aug 28, 2025

Uh oh!

dhruv-hhai commented Aug 28, 2025

Uh oh!

dhruv-hhai commented Aug 28, 2025

Uh oh!

dhruv-hhai commented Aug 28, 2025

Uh oh!

dhruv-hhai commented Aug 28, 2025

Uh oh!

dhruv-hhai commented Aug 28, 2025

Uh oh!

dhruv-hhai commented Aug 28, 2025

Uh oh!

dhruv-hhai commented Aug 28, 2025

Uh oh!

dhruv-hhai commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dhruv-hhai commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dhruv-hhai commented Aug 28, 2025

Uh oh!

dhruv-hhai commented Aug 28, 2025

Uh oh!

dhruv-hhai commented Aug 28, 2025

Uh oh!

dhruv-hhai commented Aug 28, 2025

Uh oh!

dhruv-hhai commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dhruv-hhai commented Aug 28, 2025

Uh oh!

dhruv-hhai commented Aug 28, 2025

Uh oh!

dhruv-hhai commented Aug 28, 2025

Uh oh!

dhruv-hhai commented Aug 28, 2025

Uh oh!

dhruv-hhai commented Aug 28, 2025

Uh oh!

dhruv-hhai commented Aug 28, 2025

Uh oh!

dhruv-hhai commented Aug 28, 2025

Uh oh!

dhruv-hhai commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dhruv-hhai commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dhruv-hhai commented Aug 28, 2025

Uh oh!

dhruv-hhai commented Aug 28, 2025

Uh oh!

github-actions bot commented Dec 10, 2025

📚 Documentation Preview Built

📦 Download Preview

🔍 How to Review

✅ Validation Status

Uh oh!

github-actions bot commented Dec 11, 2025

📚 Documentation Preview Built

📦 Download Preview

🔍 How to Review

✅ Validation Status

josh-hhai commented Aug 28, 2025 •

edited by cursor bot

Loading

dhruv-hhai commented Aug 28, 2025 •

edited

Loading

dhruv-hhai commented Aug 28, 2025 •

edited

Loading

dhruv-hhai commented Aug 28, 2025 •

edited

Loading

dhruv-hhai commented Aug 28, 2025 •

edited

Loading

dhruv-hhai commented Aug 28, 2025 •

edited

Loading