|
1 | 1 | # FireRequests 🔥 |
2 | 2 |
|
3 | | -[](https://badge.fury.io/py/firerequests) |
4 | | -[](https://opensource.org/licenses/Apache-2.0) |
| 3 | +[](https://pypi.org/project/firerequests/) |
| 4 | +[](https://github.com/rishiraj/firerequests/blob/main/LICENSE) |
| 5 | +[](https://pypi.org/project/firerequests/) |
5 | 6 |
|
6 | | -FireRequests is a cutting-edge, high-performance asynchronous HTTP client library designed for blazing-fast file transfers. Leveraging advanced concurrency paradigms and innovative networking techniques, FireRequests achieves up to 5x real-world speed improvements over traditional synchronous methods. |
| 7 | +**FireRequests** is a high-performance, asynchronous HTTP client library for Python, engineered to accelerate your file transfers. By harnessing advanced concepts like semaphores, exponential backoff with jitter, concurrency, and fault tolerance, FireRequests can achieve up to a **5x real-world speedup** in file downloads and uploads compared to traditional synchronous methods. |
7 | 8 |
|
8 | | -## Key Features |
| 9 | +## Features 🚀 |
9 | 10 |
|
10 | | -- **Asynchronous Architecture**: Utilizes `asyncio` for non-blocking I/O operations, maximizing throughput and minimizing latency. |
11 | | -- **Concurrent Chunk Processing**: Implements parallel downloading and uploading of file chunks for optimal resource utilization. |
12 | | -- **Adaptive Exponential Backoff**: Incorporates a sophisticated retry mechanism with jitter for robust error handling and network resilience. |
13 | | -- **Semaphore-based Concurrency Control**: Employs fine-grained concurrency management to prevent resource exhaustion and ensure system stability. |
14 | | -- **Progress Visualization**: Integrates `tqdm` for real-time progress tracking, enhancing user experience and operational visibility. |
15 | | -- **Flexible API**: Supports both high-level convenience methods and low-level customization options for advanced use cases. |
| 11 | +- **Asynchronous I/O**: Non-blocking network and file operations using `asyncio`, `aiohttp`, and `aiofiles`, boosting throughput for I/O-bound tasks. |
| 12 | +- **Concurrent Transfers**: Uses `asyncio.Semaphore` to limit simultaneous tasks, optimizing performance by managing system resources effectively. |
| 13 | +- **Fault Tolerance**: Retries failed tasks with exponentially increasing wait times, adding random jitter to prevent network congestion. |
| 14 | +- **Chunked Processing**: Files are split into configurable chunks for parallel processing, significantly accelerating uploads/downloads. |
| 15 | +- **Compatibility**: Supports environments like Jupyter through `nest_asyncio`, enabling reusable `asyncio` loops for both batch and interactive Jupyter use. |
16 | 16 |
|
17 | | -## Installation |
| 17 | +## Installation 📦 |
| 18 | + |
| 19 | +Install FireRequests using pip: |
18 | 20 |
|
19 | 21 | ```bash |
20 | 22 | pip install firerequests |
21 | 23 | ``` |
22 | 24 |
|
23 | | -## Usage Examples |
| 25 | +## Quick Start 🏁 |
24 | 26 |
|
25 | | -### High-Speed File Download |
| 27 | +Accelerate your downloads with just a few lines of code: |
26 | 28 |
|
27 | 29 | ```python |
28 | 30 | from firerequests import FireRequests |
29 | 31 |
|
30 | | -fr = FireRequests() |
31 | | -url = "https://example.com/large-file.iso" |
32 | | -filename = "large-file.iso" |
| 32 | +url = "https://example.com/largefile.iso" |
| 33 | +filename = "largefile.iso" |
33 | 34 |
|
34 | | -# Asynchronous download with optimized parameters |
35 | | -fr.run_download(url, filename, max_files=10, chunk_size=2 * 1024 * 1024) |
| 35 | +fr = FireRequests() |
| 36 | +fr.download(url, filename) |
36 | 37 | ``` |
37 | 38 |
|
38 | | -### Accelerated File Upload |
| 39 | +## Real-World Speed Test 🏎️ |
39 | 40 |
|
40 | | -```python |
41 | | -from firerequests import FireRequests |
| 41 | +FireRequests delivers significant performance improvements over traditional download methods. Below is the result of a real-world speed test: |
42 | 42 |
|
43 | | -fr = FireRequests() |
44 | | -file_path = "large-file.iso" |
45 | | -parts_urls = ["https://example.com/upload/part1", "https://example.com/upload/part2", ...] |
| 43 | +```plaintext |
| 44 | +Normal Download 🐌: 100%|██████████| 3.42G/3.42G [06:16<00:00, 9.08MB/s] |
| 45 | +Downloading on 🔥: 100%|██████████| 3.42G/3.42G [01:15<00:00, 45.2MB/s] |
46 | 46 |
|
47 | | -# Parallel multi-part upload |
48 | | -fr.run_upload(file_path, parts_urls, chunk_size=5 * 1024 * 1024, max_files=8) |
| 47 | +🐌 Download Time: 376.77 seconds |
| 48 | +🔥 Download Time: 75.75 seconds |
49 | 49 | ``` |
50 | 50 |
|
51 | | -### Performance Comparison |
| 51 | +## Advanced Usage ⚙️ |
| 52 | + |
| 53 | +### Downloading Files |
52 | 54 |
|
53 | 55 | ```python |
54 | | -fr = FireRequests() |
55 | | -url = "https://mirror.example.com/large-dataset.zip" |
56 | | -filename = "large-dataset.zip" |
| 56 | +from firerequests import FireRequests |
57 | 57 |
|
58 | | -# Benchmark FireRequests against traditional methods |
59 | | -fr.compare_speed(url, filename) |
| 58 | +url = "https://example.com/largefile.iso" |
| 59 | +filename = "largefile.iso" |
| 60 | + |
| 61 | +fr = FireRequests() |
| 62 | +fr.download(url, filename, max_files=10, chunk_size=2 * 1024 * 1024) |
60 | 63 | ``` |
61 | 64 |
|
62 | | -## Advanced Usage |
| 65 | +- **`url`**: The URL of the file to download. |
| 66 | +- **`filename`**: The local filename to save the downloaded file. |
| 67 | +- **`max_files`**: The maximum number of concurrent chunk downloads. |
| 68 | +- **`chunk_size`**: The size of each chunk in bytes. |
63 | 69 |
|
64 | | -### Custom Callback Integration |
| 70 | +### Uploading Files |
65 | 71 |
|
66 | 72 | ```python |
67 | | -import asyncio |
68 | 73 | from firerequests import FireRequests |
69 | 74 |
|
70 | | -async def progress_callback(bytes_transferred): |
71 | | - print(f"Transferred: {bytes_transferred / 1024 / 1024:.2f} MB") |
| 75 | +file_path = "largefile.iso" |
| 76 | +parts_urls = ["https://example.com/upload_part1", "https://example.com/upload_part2", ...] |
72 | 77 |
|
73 | 78 | fr = FireRequests() |
74 | | -url = "https://example.com/massive-file.tar.gz" |
75 | | -filename = "massive-file.tar.gz" |
76 | | - |
77 | | -asyncio.run(fr.download_file( |
78 | | - url, filename, max_files=12, chunk_size=4 * 1024 * 1024, |
79 | | - callback=progress_callback |
80 | | -)) |
| 79 | +fr.upload(file_path, parts_urls, chunk_size=2 * 1024 * 1024, max_files=10) |
81 | 80 | ``` |
82 | 81 |
|
83 | | -## Performance Metrics |
84 | | - |
85 | | -In real-world tests, FireRequests demonstrated exceptional performance gains: |
86 | | - |
87 | | -- **Traditional Download**: 376.77 seconds |
88 | | -- **FireRequests Download**: 75.75 seconds |
89 | | -- **Speed Improvement**: 4.98x faster |
| 82 | +### Comparing Download Speed |
90 | 83 |
|
91 | | -These results showcase the significant efficiency enhancements achievable through FireRequests' advanced asynchronous architecture and optimized networking strategies. |
| 84 | +```python |
| 85 | +from firerequests import FireRequests |
92 | 86 |
|
93 | | -## Contributing |
| 87 | +url = "https://example.com/largefile.iso" |
| 88 | +filename = "largefile.iso" |
94 | 89 |
|
95 | | -We welcome contributions! Please see our [Contributing Guidelines](CONTRIBUTING.md) for more details. |
| 90 | +fr = FireRequests() |
| 91 | +fr.compare_speed(url, filename) |
| 92 | +``` |
96 | 93 |
|
97 | | -## License |
| 94 | +## License 📄 |
98 | 95 |
|
99 | | -FireRequests is released under the Apache License 2.0. See the [LICENSE](LICENSE) file for more details. |
| 96 | +This project is licensed under the Apache License 2.0 - see the [LICENSE](https://github.com/rishiraj/firerequests/blob/main/LICENSE) file for details. |
0 commit comments