Project Title: UNQUERY (Text-to-SQL Model Comparison with Explanation and Evaluation Using RAGAS)

This project demonstrates a Text-to-SQL model comparison application using Streamlit, enhanced with Unsloth for optimized model inference and RAGAS (Retrieval-Augmented Generation Accuracy Scoring) for evaluation. The app allows users to compare the performance of different models in generating SQL queries from natural language inputs and provides explanations for the generated SQL queries.

Text-to-SQL Comparison

Prerequisites

Python Version

• Python 3.8 or higher

Required Python Libraries

• streamlit
• pandas
• torch
• transformers
• unsloth

GPU Support

• Recommended for faster inference

Installation

Clone this Repository

git clone https://github.com/your_username/text-to-sql-comparison.git
cd text-to-sql-comparison

Install Dependencies

pip install -r requirements.txt

Model and Data Placement

• Place your pre-trained and fine-tuned models in the models/ directory.
• Ensure that your validation data (validation_data.csv) is placed in the appropriate directory.

Usage

Run the Streamlit App

streamlit run explanation.py

• Open the app in your browser at http://localhost:8501.

Steps to Use the App

1. Select a model (e.g., Base Mistral, Fine-Tuned Llama3).
2. Choose a domain (e.g., Artificial Intelligence, Aerospace).
3. Select an SQL complexity level (e.g., Basic SQL, Aggregation).
4. Pick a sample query from the filtered list.
5. Click "Run Prediction" to generate the SQL query and its explanation.

Key Components

1. Unsloth Integration

• Unsloth optimizes model inference using 4-bit quantization and efficient GPU utilization, reducing runtime latency.

2. Prediction and Explanation

• The predict_sql() function supports two modes:
  • Prediction: Generates an SQL query from a natural language prompt.
  • Explanation: Explains how the generated SQL query answers the original question.

3. RAGAS Evaluation

• The evaluation pipeline (Pipeline.ipynb) uses RAGAS metrics to assess model performance:
  • Semantic Equivalence: Measures how semantically similar the predicted query is to the ground truth.
  • Exact Match Accuracy: Checks if the predicted query matches the ground truth exactly.
  • Syntax Correctness: Validates if the predicted query is syntactically correct.

Example Workflow

1. Select "Fine-Tuned Mistral" as the model.

2. Choose "Artificial Intelligence" as the domain.

3. Select "Basic SQL" as the complexity level.

4. Pick a sample query:

   What is the average explainability score of creative AI applications in 'Europe' and 'North America'?
   

5. The app will display:

   • Input Prompt
   • Ground Truth SQL Query
   • Predicted SQL Query
   • Explanation of the Predicted SQL Query

6. Run RAGAS evaluation using Pipeline.ipynb to analyze model performance.

Evaluation Results

• The evaluation notebook (Pipeline.ipynb) provides detailed metrics for each model using RAGAS:
  • Semantic Equivalence Score: Measures how semantically accurate predictions are compared to ground truth.
  • Exact Match Accuracy: Validates if predictions exactly match ground truth queries.
  • Syntax Correctness Score: Checks if predictions are syntactically valid.
• Results are visualized through plots comparing base vs fine-tuned models across these metrics.

Future Improvements

• Add more domains and query complexities to enhance testing diversity.
• Integrate additional evaluation metrics like BLEU or ROUGE scores.
• Incorporate schema-aware generation techniques for improved accuracy.

License

This project is licensed under the MIT License.

Acknowledgments

Special thanks to:

• Streamlit for enabling rapid UI development.
• Hugging Face Transformers for providing pre-trained models.
• Unsloth for efficient inference optimization.
• RAGAS for robust evaluation metrics tailored to text-to-SQL tasks.

---

Feel free to reach out with any questions or suggestions! 😊