Skip to content

[REVIEW ONLY DO NOT MERGE] Span counting + khan specific change#1

Draft
zaquestion wants to merge 6 commits intomainfrom
khan
Draft

[REVIEW ONLY DO NOT MERGE] Span counting + khan specific change#1
zaquestion wants to merge 6 commits intomainfrom
khan

Conversation

@zaquestion
Copy link
Collaborator

See commit descriptions for context, and main review point is 42f0b8b

The goal of this feature is to enrich traces leaving refinery with more
information about what happened in the trace and ease querying on that
information in honeycomb.

Scenario: Finding endpoints with lots of database interactions

Challenge:

In honeycomb (and most trace providers) querying is done on span events
themselves, and there's only a single layer of aggregation. You can
query for a count of events where db.statement is set and group that by
resolver name. You'll end up with the total number of database
interactions per endpoint. Useful, but you can't take the avg or 99th%
sum of database interactions and see what endpoints are having issues.
I'll note that it is possible to create a calculated field to count the
interactions, compute a sum of that, and group by `trace.trace_id` to
see what endpoints are the worst offenders. If you export this table,
you can throw it into a sheet and do a second aggregation to get the avg
of those sums. That said, it's really expensive to do these querys and
I've found them to be really slow. Thus the goal of these changes is to
make it dramatically easier to query and monitor these values in
widgets.

For example: I might setup a filter for spans
with `db.statement` to count the number of database interactions in a
trace. Then in honeycomb I can take the avg of a new
database_interactions attribute now set on root spans.
Extracts findSuitableRootSpan: prefers the trace's RootSpan, falls back
to the first non-annotation span (not a span event or link). Updates
computeCustomCounts to use it so custom counts land on the best
available span even when no root has arrived before the trace times out.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant