Epic: Order Accuracy Personalisation with Agentic AI

[jcork-intel]

⚠️ Assumption Requiring Validation with @FarhaanMohideen:
This epic assumes personalisation is an extension layer on top of the existing packing station pipeline, adding multi-agent orchestration to incorporate customer loyalty data, preferences, and allergy information into the order validation workflow. The underlying CV + VLM pipeline from the base Order Accuracy implementation remains intact.

Key question: Is this intended for QSR packing stations only, or should it also apply to the dine-in pipeline? This draft assumes QSR packing station context.

Overview

We build these workloads to demonstrate real-world scenarios and enable performance measurements of those workloads on different hardware platforms. They are released as open source so that partners can look at the code, the metrics instrumentation, and run the metrics themselves to replicate results or test their own components (models, pipelines, business logic) running on the solution. This epic extends the Order Accuracy pipeline with agentic AI capabilities to validate orders against customer-specific requirements including loyalty program preferences, dietary restrictions, and allergy information.

Use Case Description

As a QSR operations manager, I want an AI-powered order validation system that not only verifies items are correctly packed but also validates that customer-specific requirements from our loyalty program are met - such as "no onions" preferences, nut allergy accommodations, or dietary restrictions like vegetarian/halal - so that I can reduce personalization errors that lead to customer complaints, health risks, and lost loyalty.

The system should integrate with customer loyalty/CRM systems to retrieve personalization requirements, then use a multi-agent approach to:

1. Validate base order accuracy (existing CV + VLM pipeline)
2. Cross-reference detected items against customer-specific restrictions and preferences
3. Flag potential conflicts (e.g., "Customer has peanut allergy but order contains satay sauce")
4. Verify preference compliance where visually detectable (e.g., "no pickles" on a burger)

Technical Assumptions

Hardware Configuration (Packing Station)

• Validation Platform Specifications:
  • Intel-based processing unit (integrated or external)
  • Camera positioned for packing area coverage (existing setup)
  • Integration capabilities with existing POS and loyalty/CRM systems
  • Local storage for model artifacts, customer preference caching
• Network Connectivity:
  • Primary operation without cloud dependencies for AI inference
  • Required: Integration with loyalty/CRM system for customer data retrieval
  • Optional: Cloud connectivity for model updates

Real-World Latency Considerations

This pipeline extends the existing packing station workflow. The base Order Accuracy pipeline operates within a ~1 second latency window to match continuous packing motion. The personalisation layer adds agentic reasoning overhead.

Operational Assumption: Total end-to-end validation (base pipeline + personalisation layer) should complete within approximately 1.5 seconds on modern Intel processors (MTL, LNL, ARL, PTL) to maintain compatibility with packing workflow. Customer profile data should be available before validation begins (e.g., retrieved when order enters queue).

Software Components

• Base Order Accuracy Pipeline: Existing CV + VLM pipeline for item detection and classification (prerequisite)
• Customer Profile Data: Customer preferences, allergies, and dietary restrictions associated with orders (mock data provided for demo; production implementations would integrate with loyalty/CRM systems)
• Menu Item-to-Ingredient Mapping: Restaurant-specific data mapping menu items to their ingredients (mock data provided for demo; enables the system to understand "Spicy Chicken Wrap contains peanut sauce")
• Agentic Reasoning Layer: Multi-agent or single-agent approach that reasons about detected items against customer requirements, leveraging LLM knowledge for common allergen relationships (e.g., LLM knows "peanut sauce contains peanuts")
• Alert Generation: Prioritized alerts distinguishing critical issues (allergies) from preferences

Agentic Reasoning Capabilities

The personalisation layer requires the following reasoning capabilities (implementation approach - single agent vs multi-agent - left to engineering team):

1. Customer Context Retrieval: Access customer profile data (allergies, dietary restrictions, preferences) associated with the current order

2. Menu-to-Ingredient Resolution: Map detected menu items to their ingredients using restaurant-specific data (e.g., "Spicy Chicken Wrap" → contains peanut sauce, chicken, lettuce)

3. Allergen Reasoning: Leverage LLM's inherent knowledge to identify allergen content in ingredients (e.g., LLM knows "peanut sauce contains peanuts" without needing explicit RAG)

4. Compliance Validation: Cross-reference detected items and their ingredients against customer requirements to identify conflicts

5. Alert Prioritization: Generate prioritized alerts - CRITICAL for allergy conflicts, HIGH for dietary violations, MEDIUM/LOW for preference issues

Performance Metrics

• Base Pipeline Performance (inherited):
  • CV detection latency
  • VLM classification latency
  • End-to-end item validation time
• Personalisation Layer Performance:
  • Agentic reasoning overhead (time added by personalisation layer)
  • Total end-to-end latency (base + personalisation)
• System-wide Metrics:
  • Orders validated per minute throughput
  • Resource utilization (CPU/GPU/NPU) across all components

AI Pipeline Components

Illustrative examples of AI capabilities needed for this type of solution

• Base Order Accuracy Pipeline: Existing CV + VLM pipeline for item detection and classification
• Customer Profile Retrieval: Access to customer-specific requirements (allergies, dietary restrictions, preferences) - mock data for demo
• Menu Knowledge: Restaurant-specific menu item to ingredient mapping - mock data for demo, potentially via RAG
• Agentic Reasoning: LLM-based reasoning about ingredient-allergen relationships and compliance validation
• Alert Generation: Prioritized output distinguishing critical safety issues from preference issues
• Performance Measurement Instrumentation: Metrics collection across base pipeline and personalisation layer

Data Output Requirements

The system will generate structured data outputs for each order validation. Illustrative examples:

Customer Profile (Pre-fetched)

{
  "profile_id": "unique_identifier",
  "loyalty_id": "LOYALTY-12345",
  "order_id": "ORD-2025-001234",
  "retrieved_at": "ISO8601_timestamp",
  "customer_requirements": {
    "allergies": [
      {"allergen": "peanuts", "severity": "severe"},
      {"allergen": "tree_nuts", "severity": "moderate"}
    ],
    "dietary_restrictions": ["halal"],
    "preferences": [
      {"preference": "no_onions", "applies_to": "all_items"},
      {"preference": "extra_pickles", "applies_to": "burgers"}
    ]
  },
  "retrieval_metrics": {
    "lookup_time_ms": 45,
    "cache_hit": false
  }
}

Personalisation Validation Output

{
  "validation_id": "unique_identifier",
  "timestamp": "ISO8601_timestamp",
  "order_id": "ORD-2025-001234",
  "loyalty_id": "LOYALTY-12345",
  "base_validation": {
    "items_detected": ["Chicken Burger", "Large Fries", "Chocolate Shake"],
    "items_expected": ["Chicken Burger", "Large Fries", "Chocolate Shake"],
    "base_accuracy": 1.0
  },
  "personalisation_validation": {
    "allergy_check": {
      "status": "PASS",
      "checked_allergens": ["peanuts", "tree_nuts"],
      "conflicts_found": []
    },
    "dietary_check": {
      "status": "PASS",
      "restriction": "halal",
      "items_validated": ["Chicken Burger"],
      "notes": ["Chicken certified halal per ingredient database"]
    },
    "preference_check": {
      "status": "REQUIRES_VISUAL_VERIFICATION",
      "preferences_checked": [
        {
          "preference": "no_onions",
          "item": "Chicken Burger",
          "visually_verifiable": true,
          "verification_result": "PASS",
          "confidence": 0.87,
          "note": "No onions visible on burger"
        },
        {
          "preference": "extra_pickles",
          "item": "Chicken Burger", 
          "visually_verifiable": true,
          "verification_result": "UNCERTAIN",
          "confidence": 0.62,
          "note": "Pickles detected but quantity difficult to assess"
        }
      ]
    }
  },
  "alerts": [
    {
      "priority": "LOW",
      "type": "preference_uncertain",
      "message": "Unable to confirm 'extra pickles' on Chicken Burger - visual verification inconclusive",
      "action": "Staff to verify pickle quantity"
    }
  ],
  "overall_status": "PASS_WITH_NOTES",
  "processing_metrics": {
    "base_pipeline_time_ms": 850,
    "personalisation_layer_time_ms": 320,
    "total_validation_time_ms": 1170,
    "agents_invoked": 4
  }
}

Allergy Alert Example (Critical)

{
  "validation_id": "unique_identifier",
  "timestamp": "ISO8601_timestamp",
  "order_id": "ORD-2025-005678",
  "alerts": [
    {
      "priority": "CRITICAL",
      "type": "allergy_conflict",
      "message": "ALLERGY ALERT: Satay Dipping Sauce contains PEANUTS - Customer has severe peanut allergy",
      "item": "Satay Dipping Sauce",
      "allergen": "peanuts",
      "customer_severity": "severe",
      "action": "REMOVE ITEM IMMEDIATELY - Do not serve"
    }
  ],
  "overall_status": "FAIL_CRITICAL",
  "requires_immediate_action": true
}

Performance Metrics (Use Existing Performance Tools)

• System Resource Metrics:
  • CPU/GPU/NPU Utilization across base pipeline and agentic layer
  • Memory footprint for models and data
  • Power consumption patterns
  • Orders per minute throughput per hardware configuration (with and without agentic layer enabled)

Acceptance Criteria

1. Extension of Base Pipeline: System builds on existing Order Accuracy CV + VLM pipeline, adding personalisation as a layer that can be enabled or disabled

2. Feature Flag Enablement: Agentic personalisation capabilities can be enabled via command-line flag (e.g., make run --agentic or similar), allowing benchmarking with and without the personalisation layer

3. Mock Data for Demo: System includes sample customer profile data and menu-to-ingredient mappings that allow running the demo out of the box without external integrations

4. Agentic AI Architecture: System demonstrates agentic reasoning pattern for cross-referencing detected items against customer requirements

5. Allergy Conflict Detection: System identifies potential allergen conflicts by reasoning about menu items, their ingredients, and customer allergy profiles

6. Dietary Restriction Validation: System validates detected items against dietary restrictions (vegetarian, vegan, halal, kosher, etc.) where data is available

7. Preference Verification: System attempts visual verification of customization preferences (no onions, extra pickles, etc.) where the VLM can reasonably detect the modification

8. Alert Prioritization: System produces prioritized alerts distinguishing CRITICAL (allergy), HIGH (dietary), and MEDIUM/LOW (preference) issues

9. Latency-Aware Benchmarking: Performance measurements include end-to-end latency with personalisation layer enabled, contextualized against the ~1.5 second operational target on modern Intel processors (MTL, LNL, ARL, PTL)

10. Edge Processing Optimization: All AI processing (base pipeline + agentic layer) runs locally on Intel hardware

11. Graceful Degradation: System continues base order validation if customer profile data is unavailable, with appropriate logging

12. Performance Benchmarking Integration: Code instrumented to enable performance measurements on different Intel hardware platforms using existing performance tools framework

13. Customization Documentation: Documentation explains how to run the demo out of the box, AND how partners can customize for their specific scenarios (their menu items, ingredient mappings, VLM prompts, etc.)

14. Rapid Deployment Capability: Repository designed for rapid onboarding - clone, build, deploy within 15 minutes including model artifacts and sample data

Business Logic Assumed (and intentionally omitted)

1. Production Loyalty/CRM Integration: Actual API contracts with loyalty providers (Toast, Square, proprietary systems)
2. Production Ingredient Database: Ongoing menu-to-ingredient mapping maintenance as menus change
3. Staff Alert Workflows: How staff respond to alerts, escalation procedures, service recovery
4. Customer Communication: Notifying customers of substitutions or issues
5. Liability and Compliance: Legal requirements around allergen disclosure and validation
6. False Positive Handling: Procedures when system incorrectly flags an allergy conflict
7. Audit Trail Requirements: Regulatory logging of allergen validation events
8. Multi-Language Support: Allergen alerts in staff's preferred language

Technical Requirements

• Base Pipeline Compatibility: Clean integration with existing Order Accuracy CV + VLM pipeline without modification to base functionality
• Mock Data Architecture: Sample customer profiles and menu-to-ingredient mappings structured for easy partner customization
• Local Processing Architecture: Complete AI inference stack (base + agentic layer) running on Intel platforms without cloud dependencies
• Hardware Configuration Support: Scalable deployment across various Intel hardware specifications (MTL, LNL, PTL) with performance characterization

Notes

• Agentic AI Demonstration: This epic showcases agentic AI patterns for Intel's retail portfolio, demonstrating LLM-based reasoning for real-world compliance validation
• LLM as Knowledge Base: Common allergen relationships (peanut sauce → peanuts) leverage the LLM's inherent knowledge rather than requiring separate RAG infrastructure
• Menu-Specific RAG: Restaurant-specific menu item to ingredient mappings may use RAG or similar retrieval, as this data is not in LLM training
• Safety-Critical Alerting: Allergy detection is treated as safety-critical functionality with clear prioritization to ensure staff attention
• Preference vs. Requirement: System distinguishes between customer preferences (nice-to-have) and requirements (allergies, dietary restrictions)
• Visual Verification Limits: Some preferences cannot be visually verified (e.g., "no salt") - system acknowledges these limitations
• Performance Benchmarking Priority: Primary goal is measuring agentic AI workload performance on Intel hardware platforms
• Complementary to Base Pipeline: This epic extends rather than replaces the existing Order Accuracy pipeline

This epic represents an extension of the Order Accuracy portfolio to demonstrate agentic AI patterns for customer-centric order personalisation, showcasing Intel edge computing capabilities for LLM-based reasoning in real-time retail environments.