Skip to content

Use structured logging for FetchFulfillment errors#3153

Merged
jonathangreen merged 1 commit intomainfrom
chore/structured-fulfillment-error-logging
Mar 18, 2026
Merged

Use structured logging for FetchFulfillment errors#3153
jonathangreen merged 1 commit intomainfrom
chore/structured-fulfillment-error-logging

Conversation

@jonathangreen
Copy link
Member

@jonathangreen jonathangreen commented Mar 18, 2026

Description

Replace the unstructured error log message in FetchFulfillment with structured palace_-prefixed fields, making fulfillment failures much easier to query and aggregate in CloudWatch Logs Insights.

Previously, the upstream response body, status code, and URL were all embedded in a single log message string, requiring parse expressions to extract in CloudWatch. Now they are emitted as separate structured fields:

  • fulfillment_url — the upstream URL that failed
  • fulfillment_status_code — the HTTP status code from upstream
  • fulfillment_response — the response body, parsed as a JSON object when possible, raw string otherwise
  • fulfillment_headers — the full response headers as a dict

The human-readable log message still includes the URL and status code for quick scanning.

Motivation and Context

While investigating audiobook fulfillment failures reported via Crashlytics, we found that the existing log format made it difficult to query and aggregate errors in CloudWatch. Extracting the upstream response from the log message required fragile parse expressions, and the response body couldn't be inspected structurally when it was JSON (e.g., OPDS problem detail documents).

How Has This Been Tested?

  • Updated existing test for JSON error responses to verify structured fields are present on the log record
  • Added new test for non-JSON error responses (e.g., HTML error pages) to verify fallback to raw string
  • All tests pass, mypy clean

Checklist

  • I have updated the documentation accordingly.
  • All new and existing tests passed.

Replace the unstructured error message with palace_-prefixed extra
fields so fulfillment failures are easier to query in CloudWatch.
The response body is attempted as JSON first for deeper inspection,
falling back to the raw string. Response headers are also included.
@jonathangreen jonathangreen force-pushed the chore/structured-fulfillment-error-logging branch from b776db8 to c301e60 Compare March 18, 2026 19:16
@codecov
Copy link

codecov bot commented Mar 18, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.26%. Comparing base (5f39ae3) to head (c301e60).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3153      +/-   ##
==========================================
- Coverage   93.26%   93.26%   -0.01%     
==========================================
  Files         493      493              
  Lines       45581    45586       +5     
  Branches     6252     6252              
==========================================
+ Hits        42513    42517       +4     
- Misses       1982     1983       +1     
  Partials     1086     1086              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Contributor

@tdilauro tdilauro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🪵

@jonathangreen jonathangreen merged commit 948ba8f into main Mar 18, 2026
20 checks passed
@jonathangreen jonathangreen deleted the chore/structured-fulfillment-error-logging branch March 18, 2026 20:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants