Prompt Brewery

Try Prompt Brewery: promptbrewery.inworld.ai

Disclaimer

Prompt Brewery is a static analysis tool for prompt templates, specifically designed for chat and prompt templates used with Large Language Models (LLMs).

With 50+ analysis rules tested across 800+ real-world and synthetic templates, Prompt Brewery helps identify potential issues, structural problems, and optimization opportunities in your templates.

Prompt Engineering Challenges

Building successful AI applications hinges on meticulous prompt engineering.

Jinja has emerged as the de facto standard for templating these prompts, used in many prominent models on Hugging Face from leading providers.

However, Jinja's flexibility comes at a cost: it's notoriously difficult to debug, test, and maintain.

• Unpredictable failures: Without explicit typing, proper exception handling, or unit testing tools—combined with multiple rendering engines without cross-compatibility—you never know on which input your template will crash.
• Production debugging: A simple syntax error might only surface with specific, unpredictable user inputs, leaving you debugging broken templates in production.

Approach

Drawing from our experience powering 100M+ daily sessions in gaming, media, and apps like social platforms and learning tools, we built this static analysis tool to bring reliability and predictability to prompt engineering.

We use static analysis and basic fuzzing techniques to create comprehensive test suites for LLM templates.

Static analysis cannot guarantee all cases, so we tested this tool on:

• 400+ real templates we collected (different use-cases, LLM evaluation templates, open-sourced templates)
• Another 400+ generated synthetic tests to cover edge cases

However, if your template doesn't work, please report it as an issue so that we can add support for such cases.

Installation

Local Installation

pip install -r requirements.txt

Docker Installation (Recommended)

For a containerized setup that doesn't require local Python installation:

# Quick start with web interface
docker-compose up --build

# Then open http://localhost:7860 in your browser

Usage

Command Line Interface

For quick analysis of template files, use the CLI tool:

# Basic usage
python pbrewery_cli.py template.jinja

# Output in JSON format
python pbrewery_cli.py template.jinja --json

# Save report to file
python pbrewery_cli.py template.jinja --output report.json

The CLI provides the same analysis as the Python API but with convenient command-line options for file processing and output formatting.

Web Interface

Run the Gradio-based tool on your local machine:

python pbrewery_ui.py

Then open http://127.0.0.1:7860 in your browser.

Python API

If you're familiar with Python, you can run it in your tests or anywhere else:

import prompt_brewery as pb

# Assuming you have your template code in a tpl variable
report = pb.static_template_analysis(tpl)

# Now you have report['errors'], report['warnings'], report['info']
for error in report['errors']:
    print(f"ERROR: {error}")

for warning in report['warnings']:
    print(f"WARNING: {warning}")

for info in report['info']:
    print(f"INFO: {info}")

Analysis Categories

🔴 Errors: Critical issues that prevent template rendering or indicate serious problems

🟡 Warnings: Potential issues that should be reviewed, such as:

• Caching inefficiencies
• Redundant content
• Unbalanced tags
• Formatting inconsistencies

📘 Info: General insights about template structure:

• Template statistics
• Chat format detection
• Tool usage patterns
• Required functions and methods

Example Templates

The demo_templates/ folder contains example templates including:

• Consumer applications (workout recommendations, travel planning, recipe creation)
• Chat message formats (Qwen and Gemma)
• Tool usage and function calling
• Character prompts and roleplay scenarios
• Various prompt engineering patterns

Contributing

This tool is designed to help prompt engineers and LLM developers create better, more maintainable templates. Contributions and feedback are welcome!

Please see our contributing guidelines:

1. Fork the repository
2. Create a feature branch: git checkout -b feature/amazing-feature
3. Commit changes: git commit -m 'Add amazing feature'
4. Push to branch: git push origin feature/amazing-feature
5. Open a Pull Request

Support

• Bug Reports: GitHub Issues
• General Questions: For general inquiries and support, please email us