Skip to content

API.md

Latest commit

 

History

History
273 lines (217 loc) · 6.95 KB

API.md

File metadata and controls

273 lines (217 loc) · 6.95 KB

API Documentation

Complete API reference for the AI SEO Optimizer service.

Base URL

https://{api-id}.execute-api.{region}.amazonaws.com/Prod

Endpoints

POST /analyze

Analyzes a website for llms.txt presence, validates existing files, or generates new ones.

Request

Headers:

Content-Type: application/json

Body:

{
  "url": "string (required)",
  "options": {
    "max_pages": "number (optional, default: 20)",
    "timeout": "number (optional, default: 10)"
  }
}

Parameters:

Parameter Type Required Default Description
url string Yes - The website URL to analyze (with or without https://)
options.max_pages number No 20 Maximum number of pages to crawl when generating llms.txt
options.timeout number No 10 Request timeout in seconds

Response

Success Response (200 OK):

When llms.txt exists:

{
  "url": "https://example.com",
  "llms_txt_exists": true,
  "llms_txt_url": "https://example.com/llms.txt",
  "validation": {
    "valid": true,
    "score": 85,
    "issues": [],
    "warnings": [
      "Some H2 sections may not have properly formatted link lists"
    ],
    "sections_found": [
      "Documentation",
      "Examples",
      "Optional"
    ],
    "links_count": 5,
    "has_optional_section": true
  },
  "current_llms_txt": "# Example Site\n> Description...",
  "recommendations": [
    "Consider adding more documentation links",
    "Ensure all linked pages have .md versions"
  ]
}

When llms.txt does not exist:

{
  "url": "https://example.com",
  "llms_txt_exists": false,
  "llms_txt_url": "https://example.com/llms.txt",
  "generated_llms_txt": "# Example Site\n> A great website...\n\n## Documentation\n- [Guide](https://example.com/guide)",
  "site_info": {
    "title": "Example Site",
    "description": "A great website for examples",
    "documentation_links_found": 3,
    "navigation_links_found": 8
  },
  "recommendations": [
    "Create an llms.txt file at the root of your website",
    "Include clear descriptions for each linked resource",
    "Ensure all linked pages have .md versions for better LLM consumption"
  ]
}

Error Responses:

400 Bad Request - Missing or invalid parameters:

{
  "error": "Missing required parameter: url"
}

500 Internal Server Error - Server-side error:

{
  "error": "Failed to crawl website: Connection timeout",
  "type": "RequestException"
}

Response Fields

Common Fields

Field Type Description
url string The normalized URL that was analyzed
llms_txt_exists boolean Whether llms.txt file was found
llms_txt_url string The full URL where llms.txt should be located
recommendations array List of actionable recommendations

Validation Fields (when llms.txt exists)

Field Type Description
validation.valid boolean Whether the llms.txt file is valid according to spec
validation.score number Quality score from 0-100
validation.issues array Critical issues that make the file invalid
validation.warnings array Non-critical warnings for improvement
validation.sections_found array List of H2 section names found
validation.links_count number Number of markdown links found
validation.has_optional_section boolean Whether an "Optional" section exists
current_llms_txt string The current content of the llms.txt file

Generation Fields (when llms.txt doesn't exist)

Field Type Description
generated_llms_txt string The generated llms.txt content
site_info.title string Extracted site title
site_info.description string Extracted site description
site_info.documentation_links_found number Number of documentation links found
site_info.navigation_links_found number Number of navigation links found

Validation Scoring

The validation score is calculated as follows:

  • Start: 100 points
  • Missing H1 title: -30 points (critical)
  • Missing blockquote: -10 points
  • Each structural issue: -5 points
  • Each warning: -2 to -3 points
  • Minimum: 0 points

Score Interpretation:

  • 90-100: Excellent - Fully compliant with best practices
  • 70-89: Good - Minor improvements recommended
  • 50-69: Fair - Several issues to address
  • Below 50: Poor - Major improvements needed

Examples

Example 1: Check FastHTML (has llms.txt)

Request:

curl -X POST https://your-api-url/analyze \
  -H "Content-Type: application/json" \
  -d '{"url": "https://fastht.ml"}'

Response:

{
  "url": "https://fastht.ml",
  "llms_txt_exists": true,
  "validation": {
    "valid": true,
    "score": 95,
    "issues": [],
    "warnings": []
  }
}

Example 2: Generate for Example.com

Request:

curl -X POST https://your-api-url/analyze \
  -H "Content-Type: application/json" \
  -d '{"url": "example.com"}'

Response:

{
  "url": "https://example.com",
  "llms_txt_exists": false,
  "generated_llms_txt": "# Example Domain\n> This domain is for use in illustrative examples...",
  "site_info": {
    "title": "Example Domain",
    "description": "This domain is for use in illustrative examples in documents."
  }
}

Example 3: With Custom Options

Request:

curl -X POST https://your-api-url/analyze \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "options": {
      "max_pages": 30,
      "timeout": 15
    }
  }'

Rate Limiting

Currently, there are no built-in rate limits. However, AWS Lambda has the following limits:

  • Concurrent executions: 1000 (default, can be increased)
  • Function timeout: 30 seconds (configurable up to 15 minutes)
  • Payload size: 6 MB (synchronous), 256 KB (asynchronous)

Best Practices

  1. URL Format: Include the protocol (https://) for faster processing
  2. Timeout: Increase timeout for large websites with many pages
  3. Max Pages: Reduce max_pages for faster responses if you only need basic info
  4. Caching: Consider caching results for frequently checked URLs
  5. Error Handling: Always handle both 4xx and 5xx errors in your client

CORS

The API includes CORS headers to allow cross-origin requests:

Access-Control-Allow-Origin: *

For production use, consider restricting this to specific domains.

Authentication

Currently, the API is open. For production use, consider adding:

  • API Keys via API Gateway
  • AWS IAM authentication
  • Custom authorizers
  • Rate limiting per API key

Monitoring

Monitor your API usage via:

  • CloudWatch Logs: Function execution logs
  • CloudWatch Metrics: Invocation count, duration, errors
  • X-Ray: Distributed tracing (if enabled)

Support

For issues or questions about the API, please refer to the main README.md.