gdl - Go Downloader

A fast, concurrent, and feature-rich file downloader library and CLI tool written in Go.

⚡ Quick Start

Install

# Go users (recommended - works on all platforms)
go install github.com/forest6511/gdl/cmd/gdl@latest

# macOS/Linux via Homebrew
brew install forest6511/tap/gdl

# Or download binaries from GitHub Releases
# https://github.com/forest6511/gdl/releases

Download Your First File

# Simple download - gdl automatically optimizes for file size
gdl https://example.com/file.zip

# With resume support (recommended for large files)
gdl --resume https://releases.ubuntu.com/22.04/ubuntu-22.04-desktop-amd64.iso

gdl automatically detects optimal concurrency, resumes interrupted downloads, and handles errors gracefully.

Next Steps: Scenario Guides | CLI Reference | API Reference

✨ Features

🚀 Optimized Performance: Smart defaults with adaptive concurrency based on file size
⚡ Bandwidth Control: Rate limiting with human-readable formats (1MB/s, 500k, etc.)
📊 Progress Tracking: Real-time progress with multiple display formats
🔄 Resume Support: Automatic resume of interrupted downloads
🌐 Protocol Support: HTTP/HTTPS with custom headers and proxy support
🛡️ Error Handling: Comprehensive error handling with smart retry logic
⚡ Cross-Platform: Works on Linux, macOS, Windows, and ARM (including Apple Silicon, Raspberry Pi, and ARM servers)
🔧 Dual Interface: Both library API and command-line tool
📱 User-Friendly: Interactive prompts and helpful error messages
🔌 Plugin System: Extensible plugin architecture for custom functionality
🎯 Event-Driven: Hook into download lifecycle events
📊 Performance Monitoring: Metrics collection and aggregation for production use
🔐 Security: Built-in security constraints and validation

📈 Performance

gdl uses intelligent optimization to provide the best download speeds:

Benchmark-Optimized Smart Defaults

Small files (<1MB): Lightweight mode - minimal overhead, 60-90% of curl speed ⚡
Small-medium files (1-10MB): 2 concurrent connections
Medium files (10-100MB): 4 concurrent connections
Large files (100MB-1GB): 8 concurrent connections
Very large files (>1GB): 16 concurrent connections

Real-World Performance

Based on benchmarks against curl and wget:

File Size	gdl (optimized)	gdl (baseline)	curl	wget
100KB	90% ⚡	30%	100%	90%
500KB	80% ⚡	20%	100%	80%
1MB	80%	80%	100%	130%
10MB	85% 🚀	70%	100%	90%
50MB	90% 🚀	60%	100%	85%

Performance as % of curl speed (baseline = 100%). Higher is better.

Technical Optimizations

Zero-Copy I/O: Linux sendfile for files >10MB reduces CPU by 20-30%
Buffer Pooling: 4-tier memory pool (8KB/64KB/1MB/4MB) reduces allocations by 50-90%
Advanced Connection Pool: DNS caching, TLS session resumption, CDN optimization
Lightweight Mode: Minimal overhead HTTP client for files <1MB (3-6x faster)
Optimized HTTP Client: Enhanced connection pooling and HTTP/2 support
Adaptive Chunk Sizing: Dynamic buffer sizes (8KB-1MB) based on file size
Memory Efficiency: 50-90% less memory usage with advanced pooling
CI Performance Testing: Automated regression detection with 10% threshold

📦 Installation

As a CLI tool

Go Install

go install github.com/forest6511/gdl/cmd/gdl@latest

Homebrew (macOS/Linux)

brew tap forest6511/tap
brew install forest6511/tap/gdl

Note: Use the full tap name forest6511/tap/gdl to avoid conflicts with the GNOME gdl package.

Binary Downloads

Download pre-built binaries from GitHub Releases

As a library

go get github.com/forest6511/gdl

🚀 Quick Start

CLI Usage

# Simple download (uses smart defaults)
gdl https://example.com/file.zip

# Override smart defaults with custom settings
gdl --concurrent 8 --chunk-size 2MB -o myfile.zip https://example.com/file.zip

# With bandwidth limiting
gdl --max-rate 1MB/s https://example.com/large-file.zip
gdl --max-rate 500k https://example.com/file.zip  # Smart concurrency still applies

# With custom headers and resume
gdl -H "Authorization: Bearer token" --resume https://example.com/file.zip

# Using plugins
gdl plugin install oauth2-auth
gdl plugin list
gdl --plugin oauth2-auth https://secure-api.example.com/file.zip

Library Usage

package main

import (
    "bytes"
    "context"
    "fmt"
    "github.com/forest6511/gdl"
)

func main() {
    // Simple download using Download function
    stats, err := gdl.Download(context.Background(), 
        "https://example.com/file.zip", "file.zip")
    if err != nil {
        panic(err)
    }
    fmt.Printf("Downloaded %d bytes in %v\n", stats.BytesDownloaded, stats.Duration)
    
    // Download with progress callback and bandwidth limiting using DownloadWithOptions
    options := &gdl.Options{
        MaxConcurrency: 4,
        MaxRate: 1024 * 1024, // 1MB/s rate limit
        ProgressCallback: func(p gdl.Progress) {
            fmt.Printf("Progress: %.1f%% Speed: %.2f MB/s\n", 
                p.Percentage, float64(p.Speed)/1024/1024)
        },
    }
    
    stats, err = gdl.DownloadWithOptions(context.Background(),
        "https://example.com/file.zip", "file.zip", options)
    if err == nil {
        fmt.Printf("Download completed successfully! Average speed: %.2f MB/s\n", 
            float64(stats.AverageSpeed)/1024/1024)
    }
    
    // Download to memory using DownloadToMemory
    data, stats, err := gdl.DownloadToMemory(context.Background(),
        "https://example.com/small-file.txt")
    if err == nil {
        fmt.Printf("Downloaded %d bytes to memory in %v\n", len(data), stats.Duration)
    }
    
    // Download to any io.Writer using DownloadToWriter
    var buffer bytes.Buffer
    stats, err = gdl.DownloadToWriter(context.Background(),
        "https://example.com/data.json", &buffer)
    if err == nil {
        fmt.Printf("Downloaded to buffer: %d bytes\n", stats.BytesDownloaded)
    }
        
    // Resume a partial download using DownloadWithResume
    stats, err = gdl.DownloadWithResume(context.Background(),
        "https://example.com/large-file.zip", "large-file.zip")
    if err == nil && stats.Resumed {
        fmt.Printf("Successfully resumed download: %d bytes\n", stats.BytesDownloaded)
    }
        
    // Get file information without downloading using GetFileInfo
    fileInfo, err := gdl.GetFileInfo(context.Background(),
        "https://example.com/file.zip")
    if err == nil {
        fmt.Printf("File size: %d bytes\n", fileInfo.Size)
    }
    
    // Using the extensible Downloader with plugins
    downloader := gdl.NewDownloader()
    
    // Register custom protocol handler
    err = downloader.RegisterProtocol(customProtocolHandler)
    
    // Use middleware
    downloader.UseMiddleware(rateLimitingMiddleware)
    
    // Register event listeners
    downloader.On(events.EventDownloadStarted, func(event events.Event) {
        fmt.Printf("Download started: %s\n", event.Data["url"])
    })
    
    // Download with plugins and middleware
    stats, err = downloader.Download(context.Background(),
        "https://example.com/file.zip", "file.zip", options)
    if err == nil {
        fmt.Printf("Plugin-enhanced download completed: %d bytes\n", stats.BytesDownloaded)
    }
}

📚 Complete API Documentation

📋 API Reference - Complete library API documentation
📁 Directory Structure - Complete project organization
🔧 Maintenance Guide - Development and maintenance procedures
🚨 Error Handling - Error types and handling strategies
🔌 Plugin Development - Plugin development guide
🎨 Plugin Gallery - Example plugins and extensions
🚀 Extending Guide - Extension points and customization
📜 Compatibility Policy - Versioning and API stability
📦 Go Package Docs - Generated Go documentation

📋 Complete CLI Reference

⚙️ CLI Reference - Comprehensive CLI usage guide

📖 Examples

Complete working examples are available in the examples/ directory:

Basic Download - Simple download operations
Concurrent Downloads - Parallel download optimization
Progress Tracking - Real-time progress monitoring
Resume Support - Interrupt and resume downloads
Error Handling - Robust error recovery
Production Usage - Production-ready patterns with monitoring
CLI Examples - Command-line usage patterns
Integration Tests - Feature verification
Plugin Examples - Custom plugin development
Extension Examples - System extension patterns

Running Examples

# Core functionality examples
cd examples/01_basic_download && go run main.go
cd examples/02_concurrent_download && go run main.go
cd examples/03_progress_tracking && go run main.go
cd examples/04_resume_functionality && go run main.go
cd examples/05_error_handling && go run main.go
cd examples/06_production_usage && go run main.go

# Interface examples
cd examples/cli
chmod +x *.sh
./basic_cli_examples.sh
./advanced_cli_examples.sh

# Integration demo
cd examples/integration
go run feature_demo.go

# Plugin examples
cd examples/plugins/auth/oauth2
go build -buildmode=plugin -o oauth2.so
cd examples/plugins/storage/s3
go build -buildmode=plugin -o s3.so

🔄 Feature Parity Matrix

Feature	CLI	Library	Description
Basic download	✅	✅	Simple URL to file download
Custom destination	✅	✅	Specify output filename/path
Overwrite existing	✅	✅	Force overwrite of existing files
Create directories	✅	✅	Auto-create parent directories
Concurrent downloads	✅	✅	Multiple simultaneous connections
Custom chunk size	✅	✅	Configurable download chunks
Bandwidth throttling	✅	✅	Rate limiting with human-readable formats
Single-threaded mode	✅	✅	Disable concurrent downloads
Resume downloads	✅	✅	Continue interrupted downloads
Retry on failure	✅	✅	Automatic retry with backoff
Custom retry settings	✅	✅	Configure retry attempts/delays
Custom headers	✅	✅	Add custom HTTP headers
Custom User-Agent	✅	✅	Set custom User-Agent string
Proxy support	✅	✅	HTTP proxy configuration
SSL verification control	✅	✅	Skip SSL certificate verification
Redirect handling	✅	✅	Follow HTTP redirects
Timeout configuration	✅	✅	Set request/download timeouts
Progress display	✅	❌	Visual progress bars
Progress callbacks	❌	✅	Programmatic progress updates
Multiple progress formats	✅	❌	Simple/detailed/JSON progress
Quiet mode	✅	✅	Suppress output
Verbose mode	✅	✅	Detailed logging
Download to memory	❌	✅	Download directly to memory
Download to writer	❌	✅	Download to any io.Writer
File info retrieval	❌	✅	Get file metadata without download
Error handling	✅	✅	Robust error handling and recovery
Comprehensive errors	✅	✅	Detailed error information
Error suggestions	✅	❌	User-friendly error suggestions
Multilingual messages	✅	❌	Localized error messages
Interactive prompts	✅	❌	User confirmation prompts
Disk space checking	✅	❌	Pre-download space verification
Network diagnostics	✅	❌	Network connectivity testing
Signal handling	✅	❌	Graceful shutdown on signals
Plugin system	✅	✅	Extensible plugin architecture
Custom protocols	✅	✅	Plugin-based protocol handlers
Middleware support	❌	✅	Request/response processing
Event system	❌	✅	Download lifecycle events
Custom storage	❌	✅	Pluggable storage backends
Performance monitoring	❌	✅	Metrics collection and aggregation

Legend

✅ Fully supported - Feature is available and fully functional
❌ Not applicable - Feature doesn't make sense in this context

Key Concepts: Resume vs. Retry

The terms "Resume" and "Retry" sound similar but handle different situations. Understanding the difference is key to using gdl effectively.

	Retry	Resume
Purpose	Automatic recovery from temporary errors	Manual continuation after an intentional stop
Trigger	Network errors, server errors	Existence of an incomplete file
Control	Number of attempts (`RetryAttempts`)	Enabled/Disabled (`EnableResume`)
Result	`stats.Retries` (count)	`stats.Resumed` (true/false)

Scenario: Combining Resume and Retry

Day 1: You start downloading a 10GB file. It gets to 5GB, and you stop the program (e.g., with Ctrl+C). (-> Interruption)
Day 2: You run the same command again with resume enabled. gdl detects the incomplete 5GB file and starts downloading from that point. (-> This is a Resumed download)
During the download of the remaining 5GB, your network connection briefly drops, causing a timeout error.
gdl automatically waits a moment and re-attempts the failed request. (-> This is a Retry)
The download then completes successfully.

In this case, the final DownloadStats would be Resumed: true and Retries: 1.

🏗️ Architecture

For the complete project structure, see Directory Structure.

Key Components

Core Engine (internal/core): Main download orchestration
Concurrency Manager (internal/concurrent): Parallel download coordination
Rate Limiter (pkg/ratelimit): Bandwidth throttling and rate control
Resume Engine (internal/resume): Download resumption and partial file handling
Progress System (pkg/progress): Real-time progress tracking
Error Framework (pkg/errors): Comprehensive error handling
Network Layer (internal/network): HTTP client and diagnostics
Storage Layer (internal/storage): File system operations
Plugin System (pkg/plugin): Extensible plugin architecture
Event System (pkg/events): Download lifecycle events
Middleware Layer (pkg/middleware): Request/response processing
Protocol Registry (pkg/protocols): Custom protocol handlers
Monitoring System (pkg/monitoring): Performance metrics and analytics
CLI Interface (cmd/gdl): Command-line tool implementation

Plugin Architecture

┌─────────────────────────────────────────────────────────────┐
│                       gdl Core                              │
├─────────────────────────────────────────────────────────────┤
│                 Plugin Manager                              │
├─────────────────────────────────────────────────────────────┤
│  Auth Plugins  │ Protocol Plugins │ Storage Plugins         │
├────────────────┼──────────────────┼─────────────────────────┤
│ Transform      │ Hook Plugins     │ Custom Plugins          │
│ Plugins        │                  │                         │
└─────────────────────────────────────────────────────────────┘

Plugin Types:

Authentication Plugins: OAuth2, API keys, custom auth schemes
Protocol Plugins: FTP, S3, custom protocols
Storage Plugins: Cloud storage, databases, custom backends
Transform Plugins: Compression, encryption, format conversion
Hook Plugins: Pre/post processing, logging, analytics

🧪 Testing

The project includes comprehensive testing:

Quick Development Testing

# Run all tests (basic development)
go test ./...

# Run tests with race detection (recommended)
go test -race ./...

# Run with coverage
go test -coverprofile=coverage.out ./...

# Run benchmarks
go test -bench=. ./...

# Run performance benchmarks with optimization comparisons
go test -bench=BenchmarkDownloadWith -benchmem ./internal/core/

# Check for performance regressions
./scripts/performance_check.sh

Pre-Push Validation (Recommended)

# Complete local CI validation (recommended before pushing)
./scripts/local-ci-check.sh

# OR use Makefile targets
make pre-push           # Format + all CI checks
make ci-check          # All CI checks without formatting

Cross-Platform Testing

# Test all platforms locally (requires: brew install act)
make test-ci-all       # Ubuntu + Windows + macOS
make test-ci-ubuntu    # Ubuntu only
make test-ci-windows   # Windows only  
make test-ci-macos     # macOS only

# Quick cross-compilation check
make test-cross-compile

When to Use What

Purpose	Command	Use Case
Development	`go test ./...`	Quick feedback during coding
Safety Check	`go test -race ./...`	Detect race conditions
Before Push	`./scripts/local-ci-check.sh`	Full CI validation
Cross-Platform	`make test-ci-all`	Test Windows/macOS compatibility
Coverage	`go test -coverprofile=...`	Coverage analysis

Test Coverage

Unit tests: All packages have >90% coverage
Integration tests: Real HTTP download scenarios
CLI tests: Command-line interface functionality
Benchmark tests: Performance regression detection
Race detection: Concurrent safety verification

🤝 Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines.

Development Setup

# Clone the repository
git clone https://github.com/forest6511/gdl.git
cd gdl

# Install dependencies
go mod download

# Install golangci-lint (if not already installed)
go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest

# Install act for local CI testing (optional but recommended)
brew install act

# Verify everything works (CI equivalent check)
make ci-check

# Run tests
go test ./...

# Build CLI
go build -o gdl ./cmd/gdl/

Pre-commit Checks

Before committing and pushing changes, always run these checks locally to avoid CI failures:

# Run lint checks (essential before commit/push)
golangci-lint run

# Run tests with race detection
go test -race ./...

# Run tests with coverage
go test -coverprofile=coverage.out ./...

🔄 CI Compatibility

To prevent "works locally but fails in CI" issues, use these CI-equivalent commands:

# RECOMMENDED: Run complete pre-push validation
make pre-push        # Formats code AND runs all CI checks

# Alternative: Run CI checks without formatting
make ci-check        # All CI checks locally

# Cross-platform testing with act (requires: brew install act)
make test-ci-all     # Test Ubuntu, Windows, macOS locally
make test-ci-ubuntu  # Test Ubuntu CI locally
make test-ci-windows # Test Windows CI locally  
make test-ci-macos   # Test macOS CI locally

# Fix formatting issues automatically
make fix-and-commit  # Auto-fix formatting and create commit if needed

# Quick cross-compilation check
make test-cross-compile  # Fast Windows/macOS build verification

# Individual commands
make ci-format       # Format code to CI standards
make ci-vet          # go vet (excluding examples)
make ci-test-core    # core library tests with race detection

⚠️ Important: If the pre-commit hook reformats your code, it will stop the commit. Add the changes and commit again:

git add .
git commit

A pre-commit hook is automatically installed that runs CI-equivalent checks on every commit, preventing most CI failures.

Important: Always run make ci-check locally before pushing to ensure all checks pass. This prevents CI pipeline failures and maintains code quality standards.

🔧 Developer Tools

Git Hooks

# Setup commit message validation and pre-commit checks
./scripts/setup-git-hooks.sh

Release Management

# Prepare a new release
./scripts/prepare-release.sh v0.10.0
# Edit CHANGELOG.md with release notes
./scripts/prepare-release.sh --release v0.10.0

Local Testing

# Test GitHub Actions locally with act
act push -j quick-checks          # Fast validation
act push -W .github/workflows/main.yml --dryrun  # Full CI dry run

📚 Documentation

Contributing Guide - Development guidelines and workflow
Release Setup - Release management and distribution
Local Testing - GitHub Actions testing with act
API Reference - Library API documentation
CLI Reference - Command-line usage
Plugin Development - Plugin system guide
Maintenance - Development and maintenance procedures

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

Go community for excellent libraries and tools
Contributors who helped improve this project
Users who provided feedback and bug reports

Made with ❤️ in Go

Name		Name	Last commit message	Last commit date
Latest commit History 55 Commits
.github		.github
.vscode		.vscode
cmd/gdl		cmd/gdl
docs		docs
examples		examples
internal		internal
pkg		pkg
scripts		scripts
.actrc		.actrc
.editorconfig		.editorconfig
.gitignore		.gitignore
.gitmessage		.gitmessage
.goreleaser.yml		.goreleaser.yml
CHANGELOG.md		CHANGELOG.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
gdl.go		gdl.go
gdl_integration_test.go		gdl_integration_test.go
gdl_test.go		gdl_test.go
go.mod		go.mod
go.sum		go.sum

License

forest6511/gdl

Folders and files

Latest commit

History

Repository files navigation

gdl - Go Downloader

⚡ Quick Start

Install

Download Your First File

✨ Features

📈 Performance

Benchmark-Optimized Smart Defaults

Real-World Performance

Technical Optimizations

📦 Installation

As a CLI tool

Go Install

Homebrew (macOS/Linux)

Binary Downloads

As a library

🚀 Quick Start

CLI Usage

Library Usage

📚 Complete API Documentation

📋 Complete CLI Reference

📖 Examples

Running Examples

🔄 Feature Parity Matrix

Legend

Key Concepts: Resume vs. Retry

Scenario: Combining Resume and Retry

🏗️ Architecture

Key Components

Plugin Architecture

🧪 Testing

Quick Development Testing

Pre-Push Validation (Recommended)

Cross-Platform Testing

When to Use What

Test Coverage

🤝 Contributing

Development Setup

Pre-commit Checks

🔄 CI Compatibility

🔧 Developer Tools

Git Hooks

Release Management

Local Testing

📚 Documentation

📄 License

🙏 Acknowledgments

About

Topics

Resources

License

Contributing

Uh oh!

Stars

Watchers

Forks

Releases 12

Packages 0

Uh oh!

Contributors 2

Uh oh!

Languages

Packages