PHP XML Sitemap Generator (Library + CLI)

A professional, production-ready PHP sitemap generator by IProDev (Hemn Chawroka) — supports concurrency, robots.txt, gzip compression, sitemap index files, and comprehensive error handling.

✨ Features

• 🚀 High Performance: Concurrent HTTP requests using Guzzle
• 🤖 Robots.txt Support: Respects robots.txt rules (including wildcards)
• 📦 Gzip Compression: Automatic .gz file generation
• 📊 Sitemap Index: Automatic index file creation for large sites
• 🛡️ Error Handling: Comprehensive error handling and validation
• 📝 Logging: PSR-3 compatible logging support
• 🎯 Canonical URLs: Automatic canonical URL detection
• 🧪 Well Tested: Comprehensive unit tests with PHPUnit
• 🐳 Docker Support: Ready-to-use Docker configuration
• 💻 CLI Tool: Professional command-line interface with progress reporting

📋 Requirements

• PHP >= 8.0
• Composer
• Extensions: curl, xml, mbstring, zlib

📥 Installation

composer require iprodev/sitemap-generator-pro

🚀 CLI Usage

Basic Usage

php bin/sitemap --url=https://www.example.com

Advanced Usage

php bin/sitemap \
  --url=https://www.iprodev.com \
  --out=./sitemaps \
  --concurrency=20 \
  --max-pages=10000 \
  --max-depth=5 \
  --public-base=https://www.iprodev.com \
  --verbose

CLI Options

Option	Required	Default	Description
--url	✅ Yes	-	Starting URL to crawl
--out	No	./output	Output directory for sitemap files
--concurrency	No	10	Number of concurrent HTTP requests (1-100)
--max-pages	No	50000	Maximum number of pages to crawl
--max-depth	No	5	Maximum link depth to follow
--public-base	No	-	Public base URL for sitemap index
--verbose, -v	No	false	Enable verbose output
--help, -h	No	-	Show help message

CLI Output Example

======================================================================
  PHP XML Sitemap Generator
======================================================================
Configuration:
  URL:         https://www.example.com
  Domain:      www.example.com
  Output:      ./output
  Concurrency: 20
  Max Pages:   10000
  Max Depth:   5
======================================================================

[0.50s] [info] Initializing crawler...
[0.75s] [info] Fetching robots.txt...
[1.20s] [info] Starting crawl...
[45.30s] [info] Crawl completed {"duration":"45.3s","pages":1523}

======================================================================
  ✅ Success!
======================================================================
Generated Files:
  • sitemap-1.xml.gz (125.4 KB)
  • sitemap-index.xml (892 B)

Statistics:
  • Total Pages:    1523
  • Total Time:     46.2s
  • Crawl Speed:    33.0 pages/sec
  • Memory Used:    45.8 MB
  • Output Dir:     ./output
======================================================================

💻 Programmatic Usage

Basic Example

use IProDev\Sitemap\Fetcher;
use IProDev\Sitemap\Crawler;
use IProDev\Sitemap\SitemapWriter;
use IProDev\Sitemap\RobotsTxt;

// Initialize fetcher
$fetcher = new Fetcher(['concurrency' => 10]);

// Load robots.txt
$robots = RobotsTxt::fromUrl('https://www.example.com', $fetcher);

// Create crawler
$crawler = new Crawler($fetcher, $robots);

// Crawl website
$pages = $crawler->crawl('https://www.example.com', 10000, 5);

// Write sitemap files
$files = SitemapWriter::write(
    $pages, 
    __DIR__ . '/sitemaps', 
    50000, 
    'https://www.example.com'
);

echo "Generated " . count($files) . " files\n";

Advanced Example with Logging

use IProDev\Sitemap\Fetcher;
use IProDev\Sitemap\Crawler;
use IProDev\Sitemap\SitemapWriter;
use IProDev\Sitemap\RobotsTxt;
use Monolog\Logger;
use Monolog\Handler\StreamHandler;

// Create logger
$logger = new Logger('sitemap');
$logger->pushHandler(new StreamHandler('sitemap.log', Logger::INFO));

// Initialize with logger
$fetcher = new Fetcher([
    'concurrency' => 20,
    'timeout' => 15,
], $logger);

$robots = RobotsTxt::fromUrl('https://www.example.com', $fetcher);
$crawler = new Crawler($fetcher, $robots, $logger);

// Crawl with error handling
try {
    $pages = $crawler->crawl('https://www.example.com', 10000, 5);
    $files = SitemapWriter::write($pages, './sitemaps', 50000, 'https://www.example.com');
    
    // Get statistics
    $stats = $crawler->getStats();
    echo "Processed: {$stats['processed']} pages\n";
    echo "Unique URLs: {$stats['unique_urls']}\n";
    
} catch (\InvalidArgumentException $e) {
    echo "Configuration error: {$e->getMessage()}\n";
} catch (\RuntimeException $e) {
    echo "Runtime error: {$e->getMessage()}\n";
}

Custom Fetcher Configuration

$fetcher = new Fetcher([
    'concurrency' => 20,
    'timeout' => 15,
    'connect_timeout' => 10,
    'headers' => [
        'User-Agent' => 'MyBot/1.0',
    ],
    'verify' => true, // SSL verification
], $logger);

🧪 Testing

Run unit tests:

composer install
vendor/bin/phpunit

Run with coverage:

vendor/bin/phpunit --coverage-html coverage

Code style check:

vendor/bin/phpcs --standard=PSR12 src/ tests/

🐳 Docker Usage

Build the Docker image:

docker build -t sitemap-generator-pro .

Run the container:

docker run --rm \
  -v $(pwd)/sitemaps:/app/output \
  sitemap-generator-pro \
  --url=https://www.iprodev.com \
  --out=/app/output \
  --concurrency=20 \
  --max-pages=10000 \
  --public-base=https://www.iprodev.com \
  --verbose

📚 API Documentation

Fetcher

// Constructor
new Fetcher(array $options = [], ?LoggerInterface $logger = null)

// Fetch multiple URLs concurrently
fetchMany(array $urls, callable $onFulfilled, ?callable $onRejected = null): void

// Fetch single URL
get(string $url): ResponseInterface

// Get concurrency setting
getConcurrency(): int

Crawler

// Constructor
new Crawler(Fetcher $fetcher, RobotsTxt $robots, ?LoggerInterface $logger = null)

// Crawl website
crawl(string $startUrl, int $maxPages = 10000, int $maxDepth = 5): array

// Get crawl statistics
getStats(): array

SitemapWriter

// Write sitemap files
static write(
    array $pages, 
    string $outPath, 
    int $maxPerFile = 50000, 
    ?string $publicBase = null
): array

Parser

// Extract links from HTML
static extractLinks(string $html, string $baseUrl): array

// Resolve relative URL
static resolveUrl(string $href, string $base): ?string

// Get canonical URL
static getCanonical(string $html, string $baseUrl): ?string

// Get meta robots directives
static getMetaRobots(string $html): array

RobotsTxt

// Load from URL
static fromUrl(string $baseUrl, Fetcher $fetcher): RobotsTxt

// Check if URL is allowed
isAllowed(string $url): bool

// Get disallow rules
getDisallows(): array

// Get allow rules
getAllows(): array

Utils

static normalizeUrl(string $url): string
static formatBytes(int $bytes, int $precision = 2): string
static formatDuration(float $seconds): string
static isValidUrl(string $url): bool
static getDomain(string $url): ?string
static calculateProgress(int $current, int $total): float
static progressBar(int $current, int $total, int $width = 50): string
static getMemoryUsage(): string
static getPeakMemoryUsage(): string
static cleanUrl(string $url, bool $removeQuery = false): string

🔧 Configuration Best Practices

For Small Sites (< 1,000 pages)

--concurrency=5 --max-pages=1000 --max-depth=10

For Medium Sites (1,000 - 10,000 pages)

--concurrency=10 --max-pages=10000 --max-depth=5

For Large Sites (> 10,000 pages)

--concurrency=20 --max-pages=50000 --max-depth=3

🛡️ Error Handling

The library includes comprehensive error handling:

• Invalid URLs: Validates all URLs before processing
• Network Errors: Gracefully handles timeouts and connection failures
• Memory Management: Efficient memory usage for large sites
• File System Errors: Proper validation and error messages
• Robots.txt Parsing: Handles malformed robots.txt files

📝 Generated Files

The generator creates the following files:

• sitemap-1.xml - First sitemap file
• sitemap-1.xml.gz - Compressed version
• sitemap-2.xml.gz - Additional files if needed
• sitemap-index.xml - Index file listing all sitemaps

🔒 Security Considerations

• Path traversal prevention
• URL validation and sanitization
• Safe XML generation with proper escaping
• Robots.txt respect
• Meta robots tag support
• SSL certificate verification

📊 Performance Tips

1. Increase Concurrency: For faster crawling (up to 100)
2. Reduce Max Depth: Focus on important pages
3. Use Memory: Ensure adequate memory for large sites
4. Network: Fast and stable internet connection recommended
5. Robots.txt: Proper robots.txt reduces unnecessary requests

🤝 Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Write tests for new features
4. Follow PSR-12 coding standards
5. Submit a pull request

📄 License

MIT License - see LICENSE.md for details

🙏 Credits

Created by iprodev - https://github.com/iprodev

📞 Support

• Issues: GitHub Issues
• Discussions: GitHub Discussions

---

Made with ❤️ by iprodev