Skip to content

Commit f7f37bb

Browse files
skylarmbskylarmb
committed
docs: add TODO.md documenting 37 test failures
Comprehensive documentation of remaining test failures after model generation and test infrastructure fixes. These appear to be pre-existing issues from API evolution without corresponding test updates. Categories: - 8 failures: Missing tracer_id property (tests expect .tracer_id, code has ._tracer_id) - 21 failures: trace decorator kwargs handling (unexpected keyword arguments) - 3 failures: Backwards compatibility imports (missing modules/functions) - 2 failures: UUIDType repr (FIXED in current work) - 1 failure: External dataset evaluation mocks - 2 failures: Baggage propagation key mismatches Includes investigation notes, example errors, and suggested fixes for each category. ✨ Created with OpenCode
1 parent f6c6199 commit f7f37bb

File tree

1 file changed

+255
-0
lines changed

1 file changed

+255
-0
lines changed

‎TODO.md‎

Lines changed: 255 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,255 @@
1+
# TODO - Test Failures to Fix
2+
3+
This document tracks the 37 test failures discovered after fixing the model generation and test infrastructure. These appear to be pre-existing issues where the codebase evolved but tests weren't updated to match the new APIs.
4+
5+
**Last Updated:** 2025-12-12
6+
**Test Command:** `make test` (runs `pytest tests/unit/ tests/tracer/ tests/compatibility/ -n auto`)
7+
**Total Failures:** 37 out of 3014 tests
8+
9+
---
10+
11+
## Category 1: Missing `tracer_id` Property (8 failures)
12+
13+
**Issue:** Tests expect `.tracer_id` as a public property, but the implementation only has `._tracer_id` (private attribute).
14+
15+
**Root Cause:** The `HoneyHiveTracer` class needs a public `@property` for `tracer_id` to expose the private `._tracer_id` attribute.
16+
17+
**Affected Tests:**
18+
- `tests/tracer/test_baggage_isolation.py::TestBaggagePropagationIntegration::test_multi_instance_no_interference`
19+
- `tests/tracer/test_baggage_isolation.py::TestTracerDiscoveryViaBaggage::test_discover_tracer_from_baggage`
20+
- `tests/tracer/test_baggage_isolation.py::TestBaggageIsolation::test_two_tracers_isolated_baggage`
21+
- `tests/tracer/test_baggage_isolation.py::TestTracerDiscoveryViaBaggage::test_discovery_with_evaluation_context`
22+
- `tests/tracer/test_multi_instance.py::TestMultiInstanceSafety::test_discovery_in_threads`
23+
- `tests/tracer/test_multi_instance.py::TestMultiInstanceSafety::test_registry_concurrent_access`
24+
- `tests/tracer/test_multi_instance.py::TestMultiInstanceIntegration::test_two_projects_same_process`
25+
- `tests/tracer/test_multi_instance.py::TestMultiInstanceSafety::test_no_cross_contamination`
26+
27+
**Example Error:**
28+
```
29+
AttributeError: 'HoneyHiveTracer' object has no attribute 'tracer_id'. Did you mean: '_tracer_id'?
30+
```
31+
32+
**Suggested Fix:**
33+
Add to `src/honeyhive/tracer/core/tracer.py` or `base.py`:
34+
```python
35+
@property
36+
def tracer_id(self) -> str:
37+
"""Public accessor for tracer ID."""
38+
return self._tracer_id
39+
```
40+
41+
---
42+
43+
## Category 2: `trace` Decorator Kwargs Handling (21 failures)
44+
45+
**Issue:** The `_create_tracing_params()` function rejects kwargs that tests are passing to the `@trace` decorator (e.g., `name=`, `key=`, arbitrary attributes).
46+
47+
**Root Cause:** The decorator API may have changed to be more strict about accepted parameters, but tests still use the old flexible kwargs approach.
48+
49+
**Affected Tests:**
50+
- `tests/tracer/test_trace.py::TestTraceDecorator::test_trace_basic`
51+
- `tests/tracer/test_trace.py::TestTraceDecorator::test_trace_with_attributes`
52+
- `tests/tracer/test_trace.py::TestTraceDecorator::test_trace_with_return_value`
53+
- `tests/tracer/test_trace.py::TestTraceDecorator::test_trace_with_exception`
54+
- `tests/tracer/test_trace.py::TestTraceDecorator::test_trace_with_complex_attributes`
55+
- `tests/tracer/test_trace.py::TestTraceDecorator::test_trace_error_recovery`
56+
- `tests/tracer/test_trace.py::TestTraceDecorator::test_trace_performance`
57+
- `tests/tracer/test_trace.py::TestTraceDecorator::test_trace_with_arguments`
58+
- `tests/tracer/test_trace.py::TestTraceDecorator::test_trace_with_none_attributes`
59+
- `tests/tracer/test_trace.py::TestTraceDecorator::test_trace_with_dynamic_attributes`
60+
- `tests/tracer/test_trace.py::TestTraceDecorator::test_trace_concurrent_usage`
61+
- `tests/tracer/test_trace.py::TestTraceDecorator::test_trace_with_async_function`
62+
- `tests/tracer/test_trace.py::TestTraceDecorator::test_trace_with_context_manager`
63+
- `tests/tracer/test_trace.py::TestTraceDecorator::test_trace_with_generator_function`
64+
- `tests/tracer/test_trace.py::TestTraceDecorator::test_trace_with_nested_calls`
65+
- `tests/tracer/test_trace.py::TestTraceDecorator::test_trace_with_keyword_arguments`
66+
- `tests/tracer/test_trace.py::TestTraceDecorator::test_trace_with_class_method`
67+
- `tests/tracer/test_trace.py::TestTraceDecorator::test_trace_memory_usage`
68+
- `tests/tracer/test_trace.py::TestTraceDecorator::test_trace_with_large_data`
69+
- `tests/tracer/test_trace.py::TestTraceDecorator::test_trace_with_empty_attributes`
70+
- `tests/tracer/test_trace.py::TestTraceDecorator::test_trace_with_static_method`
71+
72+
**Example Error:**
73+
```
74+
TypeError: _create_tracing_params() got an unexpected keyword argument 'name'
75+
TypeError: _create_tracing_params() got an unexpected keyword argument 'key'
76+
```
77+
78+
**Example Test Usage:**
79+
```python
80+
@trace(name="test-function", tracer=self.mock_tracer)
81+
@trace(event_name="test-function", key="value", tracer=self.mock_tracer)
82+
```
83+
84+
**Investigation Needed:**
85+
1. Check `src/honeyhive/tracer/instrumentation/decorators.py` to see what params are accepted
86+
2. Determine if the decorator API intentionally changed or if tests need updating
87+
3. Either:
88+
- Update `_create_tracing_params()` to accept/ignore arbitrary kwargs, OR
89+
- Update all test cases to use the new strict API
90+
91+
---
92+
93+
## Category 3: Backwards Compatibility - Missing Imports (3 failures)
94+
95+
**Issue:** Tests expect certain modules/functions to exist that are no longer available or moved.
96+
97+
### 3a. Missing `honeyhive.utils.config` module
98+
99+
**Affected Test:**
100+
- `tests/compatibility/test_backward_compatibility.py::TestBackwardCompatibility::test_environment_variable_compatibility`
101+
102+
**Example Error:**
103+
```
104+
ModuleNotFoundError: No module named 'honeyhive.utils.config'
105+
```
106+
107+
**Investigation Needed:**
108+
- Check if `honeyhive.utils.config` was removed or renamed
109+
- Verify if this is intentional API change or if module needs to be restored
110+
111+
### 3b. Missing `evaluate_batch` function
112+
113+
**Affected Test:**
114+
- `tests/compatibility/test_backward_compatibility.py::TestBackwardCompatibility::test_new_features_availability`
115+
116+
**Example Error:**
117+
```
118+
Failed: New features should be importable: cannot import name 'evaluate_batch' from 'honeyhive'
119+
```
120+
121+
**Investigation Needed:**
122+
- Check if `evaluate_batch` was removed or renamed
123+
- Verify if it should be exported from main `honeyhive` package
124+
125+
### 3c. Config access compatibility
126+
127+
**Affected Test:**
128+
- `tests/compatibility/test_backward_compatibility.py::TestBackwardCompatibility::test_config_access_compatibility`
129+
130+
**Example Error:**
131+
```
132+
AssertionError: assert (False or False)
133+
```
134+
135+
**Investigation Needed:**
136+
- Test is checking config access patterns
137+
- Likely related to missing `honeyhive.utils.config` module
138+
139+
---
140+
141+
## Category 4: Model/Data Issues (3 failures)
142+
143+
### 4a. UUIDType repr format (2 failures) - **FIXED IN CURRENT COMMIT**
144+
145+
**Status:** ✅ **RESOLVED** - Fixed by adding `__repr__` method in post-processing
146+
147+
**Affected Tests:**
148+
- `tests/unit/test_models_generated.py::TestGeneratedModels::test_uuid_type`
149+
- `tests/unit/test_models_integration.py::TestUUIDType::test_uuid_type_repr_method`
150+
151+
**Previous Error:**
152+
```
153+
assert "UUIDType(root=UUID('...'))" == 'UUIDType(...)'
154+
```
155+
156+
**Fix Applied:** Updated `scripts/generate_v0_models.py` to add both `__str__` and `__repr__` methods to `UUIDType` during post-processing.
157+
158+
### 4b. External dataset evaluation
159+
160+
**Affected Test:**
161+
- `tests/unit/test_experiments_core.py::TestEvaluate::test_evaluate_with_external_dataset`
162+
163+
**Example Error:**
164+
```
165+
AssertionError: expected call not found.
166+
```
167+
168+
**Investigation Needed:**
169+
- Mock expectations don't match actual calls
170+
- May be related to API changes in evaluation functions
171+
172+
---
173+
174+
## Category 5: Baggage Propagation Issues (2 failures)
175+
176+
### 5a. SAFE_PROPAGATION_KEYS mismatch
177+
178+
**Affected Test:**
179+
- `tests/tracer/test_baggage_isolation.py::TestSelectiveBaggagePropagation::test_safe_keys_constant_complete`
180+
181+
**Example Error:**
182+
```
183+
AssertionError: SAFE_PROPAGATION_KEYS mismatch.
184+
Expected: {'source', 'dataset_id', 'datapoint_id', 'run_id', 'project', 'honeyhive_tracer_id'}
185+
```
186+
187+
**Investigation Needed:**
188+
- Check if `SAFE_PROPAGATION_KEYS` constant changed
189+
- Verify if test expectations need updating
190+
191+
### 5b. Evaluate pattern simulation
192+
193+
**Affected Test:**
194+
- `tests/tracer/test_baggage_isolation.py::TestBaggagePropagationIntegration::test_evaluate_pattern_simulation`
195+
196+
**Example Error:**
197+
```
198+
AssertionError: assert 'honeyhive.metadata.datapoint' in {'honeyhive.project': 'test-project', ...}
199+
```
200+
201+
**Investigation Needed:**
202+
- Baggage key format or propagation logic may have changed
203+
- Test expectations may need updating to match new baggage schema
204+
205+
---
206+
207+
## Summary Statistics
208+
209+
| Category | Count | Priority | Status |
210+
|----------|-------|----------|--------|
211+
| Missing tracer_id property | 8 | High | To Do |
212+
| trace decorator kwargs | 21 | High | To Do |
213+
| Backwards compat imports | 3 | Medium | To Do |
214+
| Model/Data issues | 3 | Medium | 2 Fixed, 1 To Do |
215+
| Baggage propagation | 2 | Medium | To Do |
216+
| **Total** | **37** | - | **2 Fixed, 35 To Do** |
217+
218+
---
219+
220+
## Action Items
221+
222+
### Immediate (High Priority)
223+
1. [ ] Add `tracer_id` property to `HoneyHiveTracer` class
224+
2. [ ] Investigate and fix `trace` decorator kwargs handling
225+
- Determine intended API design
226+
- Update either implementation or tests accordingly
227+
228+
### Short Term (Medium Priority)
229+
3. [ ] Restore or document removal of `honeyhive.utils.config`
230+
4. [ ] Restore or document removal of `evaluate_batch`
231+
5. [ ] Fix baggage propagation key mismatches
232+
6. [ ] Fix external dataset evaluation mock expectations
233+
234+
### Verification
235+
- [ ] Run `make test` and verify all tests pass
236+
- [ ] Run `make test-all` (requires .env) for full integration test suite
237+
- [ ] Update this TODO.md as issues are resolved
238+
239+
---
240+
241+
## Notes
242+
243+
- These failures were discovered after fixing model generation (Pydantic v2 compatibility)
244+
- The UUIDType `__str__` and `__repr__` issues have been resolved
245+
- Most failures appear to be from API evolution without corresponding test updates
246+
- No CI changes needed - CI uses tox environments which handle integration tests separately
247+
248+
---
249+
250+
## Related Commits
251+
252+
- `f6c6199` - Fixed test infrastructure and import paths for Pydantic v2 compatibility
253+
- `cf2ca51` - Fixed formatting tool version mismatch and expanded make format scope
254+
- `08b0bd4` - Consolidated pip dependencies: removed requests, beautifulsoup4, pyyaml from Nix
255+
- `755133a` - feat(dev): add v0 model generation and fix environment isolation

0 commit comments

Comments
 (0)