Plan Fallback Scenarios When LLM Module Can't Generate Response

[nuwangeek]

Description

Define and implement comprehensive fallback strategies for scenarios where the LLM Module cannot generate a response due to errors, timeouts, or validation failures.

Fallback Chain

Service Workflow → Context Workflow → RAG Workflow → OOD Response

Failure Scenarios

Layer 1: Service Workflow Failures

• No Intent Detected → Fallback to Context Workflow
• Service Validation Failed → Fallback to Context Workflow
• Service Call Timeout → Return SERVICE_TIMEOUT_ERROR_MESSAGES
• Service Execution Error → Return SERVICE_EXECUTION_ERROR_MESSAGES
• Output Guardrails Blocked → Fallback to Context Workflow (alternative: return error)

Layer 2: Context Workflow Failures

• Not a Greeting AND No Context Match → Fallback to RAG Workflow
• LLM Parsing Failed → Fallback to RAG Workflow
• Empty Conversation History (and not greeting) → Fallback to RAG Workflow
• Output Guardrails Blocked → Fallback to RAG Workflow (alternative: return error)

Layer 3: RAG Workflow Failures

• No Relevant Chunks Found → Fallback to OOD Response
• Low Confidence Score → Fallback to OOD Response
• Output Guardrails Blocked → Return OUTPUT_GUARDRAIL_VIOLATION_MESSAGE

Layer 4: OOD Response

• Final Fallback → Return OUT_OF_SCOPE_MESSAGE