Skip to content

Nfk/refactor tvl#6

Open
nofreekoolaid wants to merge 2 commits intomainfrom
nfk/refactor-tvl
Open

Nfk/refactor tvl#6
nofreekoolaid wants to merge 2 commits intomainfrom
nfk/refactor-tvl

Conversation

@nofreekoolaid
Copy link
Owner

Add TVL Extrapolation Feature and Enhanced Tooling

Summary

This PR introduces linear extrapolation for TVL data analysis, adds convenient make targets for quick queries, and includes a comprehensive test suite with 26 tests covering all functionality.

New Features

TVL Extrapolation

  • Linear Extrapolation at start/end dates: Uses the trend from the two nearest data points to estimate TVL values when data only exists on one side of a date
  • --no-extrapolate flag: Optional flag to disable extrapolation and only return dates that can be interpolated between two surrounding data points
  • Default behavior: Extrapolation is enabled by default for complete date range coverage

Make Targets for Easy Usage

  • make tvl: Get daily TVL data in CSV format
    make tvl PROTOCOL=euler START=2025-01-01 END=2025-01-15
  • make avgtvl: Get average TVL with interpolation/extrapolation
    make avgtvl PROTOCOL=euler START=2025-01-01 END=2025-01-15
  • Optional parameters: Support for OPTS='--no-extrapolate' or other flags

Technical Details

API Changes

  • Added extrapolate: bool = True parameter to:
    • get_tvl_dataset()
    • get_average_tvl()
  • Added _get_extrapolation_slope() helper function
  • Enhanced interpolation logic to support both interpolation and extrapolation

How Extrapolation Works

  1. Start of range: If data only exists after the start date, uses the two earliest data points to calculate a slope and extrapolates backward
  2. End of range: If data only exists before the end date, uses the two most recent data points to calculate a slope and extrapolates forward
  3. Middle gaps: Continues to use linear interpolation between surrounding points

Example Output

date,tvl,is_interpolated
2025-01-01,110398623.00,false
2025-01-02,111157719.00,false
2025-01-03,115676449.00,false
2025-01-04,124228188.00,false
2025-01-05,137182701.00,false

Backward Compatibility

Fully backward compatible

  • Default behavior now includes extrapolation (more features)
  • Original behavior available with --no-extrapolate flag
  • Existing scripts will work without modifications
  • All existing function signatures maintained with new optional parameters

Usage Examples

# Get daily TVL data (CSV)
make tvl PROTOCOL=euler START=2025-01-01 END=2025-01-15

# Get average TVL
make avgtvl PROTOCOL=euler START=2025-01-01 END=2025-01-15

# Disable extrapolation
make tvl PROTOCOL=aave START=2025-01-01 END=2025-01-31 OPTS='--no-extrapolate'

# Using CLI tool
uv run trr.py tvl euler 2025-01-01 2025-01-15 --format json
uv run trr.py tvl euler 2025-01-01 2025-01-15 --mean

Testing Instructions

  1. Run the test suite:

    make test
  2. Test with real data:

    make avgtvl PROTOCOL=euler START=2025-01-01 END=2025-01-15
  3. Run all CI checks:

    make cichecks

This commit represents a comprehensive refactor of the codebase with
significant improvements to code quality, developer experience, and
maintainability.

- Migrated from pip/venv to uv for faster dependency management
- Added pyproject.toml with proper project metadata and dependencies
- Added uv.lock for reproducible builds
- Added python-dotenv for environment variable management

- Created unified CLI entrypoint (trr.py) for all scripts
- All individual scripts now accessible via subcommands:
  - download, analyze, summary, tvl, deployments, scan, compare, tdp
- Improved user experience with consistent interface
- Maintains backward compatibility with individual script usage

- Enhanced avg_tvls.py to return full TVL dataset (not just mean)
- Added linear interpolation for missing data points
- Added is_interpolated flag to distinguish raw vs interpolated data
- Support for CSV (default) and JSON output formats
- Backward compatible --mean flag for average-only output
- Edge case handling: forward-fill at start, backward-fill at end

- Fixed all linting issues (ruff):
  - Replaced bare except clauses with specific Exception types
  - Fixed exception chaining (raise ... from err)
  - Updated type hints to modern Python syntax (list, dict, tuple)
  - Fixed unused variables and import ordering
- Applied consistent code formatting across all files
- All 33 tests passing

- Added Makefile with development commands:
  - make lint, make lint-fix, make format, make format-check
  - make test, make cichecks (runs all CI checks)
- Added ruff for linting and formatting
- Added comprehensive test suite for TVL functionality
- Fixed test infrastructure (mock patches, config loading)

- Added .env.example template for API keys
- Added .env to .gitignore
- All scripts now automatically load .env via python-dotenv
- Support for ETHERSCAN_API_KEY, ARBISCAN_API_KEY, ETH_RPC_URL

- Updated README.md with new uv-based setup instructions
- Added unified CLI usage examples
- Added development scripts documentation
- Added .env configuration instructions
- Updated all examples to use new CLI interface

- Fixed test_scanner.py to work with new module structure
- Updated all mock patches to use correct module paths
- Added test environment detection for config loading
- Fixed test expectations to match actual behavior

- 17 files modified (code quality, type hints, formatting)
- 6 new files (trr.py, pyproject.toml, Makefile, .env.example, test_avg_tvls.py, __init__.py)
- Updated .gitignore for .env files

None - all changes maintain backward compatibility.

- All 33 tests passing
- All linting checks passing
- All formatting checks passing
- make cichecks passes completely
- Add linear extrapolation for start/end dates using trend from nearest two points
- Add --no-extrapolate flag to disable extrapolation (only interpolate between points)
- Add make tvl and make avgtvl targets for easy TVL queries
- Add comprehensive test suite (26 tests) for all TVL functionality
- Update README with complete documentation and real examples
- Simplify test code with helper function and update to 2025 dates
- Add extrapolate parameter to get_tvl_dataset() and get_average_tvl()
- Document all features with examples using real Euler Finance data
- Add .ruff_cache/ to .gitignore
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant