Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Document/content/1.2_Principles_of_AI_Testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ ISO/IEC 23053 \[4\] structures the ML-based AI system lifecycle into a series of

1. **Planning & Scoping:** In this phase, you establish clear business objectives, success metrics, and ML use cases while identifying key stakeholders, regulatory requirements, and the organization’s risk tolerance.
2. **Data Preparation:** In this phase, you gather and document raw data sources, conduct profiling and quality checks through preprocessing pipelines, and implement versioning and lineage tracking for full data traceability.
3. **Model Development & Training:** In this phase, you choose appropriate algorithms and architectures, train models on curated datasets with feature engineering, and record experiments, including the parameters that govern the learning process (i.e hyperparameters) and performance metrics in a model registry.
3. **Model Development & Training:** In this phase, you choose appropriate algorithms and architectures, train models on curated datasets with feature engineering, and record experiments, including the parameters that govern the learning process (i.e. hyperparameters) and performance metrics in a model registry.
4. **Validation & Evaluation:** in this phase, you test models using reserved and adversarial datasets, perform fairness, robustness, and security evaluations, and ensure they meet functional, ethical, and regulatory standards.
5. **Deployment & Integration:** in this phase, you are preparing and bundling your trained AI model into a deployable artifact for either service (i.e. wrap the model in a microservice or API) or edge deployment (i.e. convert and optimize the model for resource-constrained devices such as IoT gateways or mobile phones) automate build-test-release workflows via CI/CD, and verify infrastructure security measures
6. **Operation & Maintenance:** in this phase while the AI product is in production environment, you will continuously monitor performance, data drift, and audit logs, triggering alerts on anomalies or compliance breaches, while periodically retraining models with fresh data, re-validating security, privacy, and fairness controls, and updating documentation, training, and policies as needed.
Expand Down
12 changes: 6 additions & 6 deletions Document/content/2.0_Threat_Modeling_for_AI_Systems.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ Choose a methodology that best aligns with your organization’s objectives, sys
- **Privacy vs. Security Focus:** If data confidentiality and compliance are paramount, incorporate a privacy-centric method (LINDDUN) alongside your core security approach. When adversarial robustness is the top concern, ensure your chosen framework includes or can easily integrate adversarial test case design (MITRE ATLAS or custom AI-STRIDE extensions).
- **Agentic AI Threat Modeling:** Use MAESTRO (Note (a)) when you need to model risks in systems where AI agents interact with users, tools, other agents, or their environment—contexts where most real-world AI failures and security issues emerge.
- **LLM Powered Threat Modeling:** Large Language Models, or LLMs, can be used streamline the threat modeling process by automating several steps that are traditionally manual and time-consuming. LLM-augmented threat modeling, as taught in this training [25], uses large language models to accelerate and enhance each stage of the threat-modeling process—automatically generating threats, mitigations, and control recommendations directly from system descriptions—whether that’s text-based documentation, architecture diagrams, or even code.
- **Tools & Process Fit:** Pick a methodology compatible with your existing SDLC, threat-modeling tools and reporting dashboards. PASTA’s stages work well in risk-management platforms and can be LLM-powwered with LLM Threat Modeling Prompt Templates (Note (b)); STRIDE maps easily to both manual threat-modeling tools like ThreatDragon as well as LLM powered threat modeling tools like STRIDEGPT.
- **Tools & Process Fit:** Pick a methodology compatible with your existing SDLC, threat-modeling tools and reporting dashboards. PASTA’s stages work well in risk-management platforms and can be LLM-powered with LLM Threat Modeling Prompt Templates (Note (b)); STRIDE maps easily to both manual threat-modeling tools like ThreatDragon as well as LLM powered threat modeling tools like STRIDEGPT.

Note (a): MAESTRO It does not replace STRIDE, PASTA, or other traditional frameworks; instead, it complements them by adding AI-specific threat classes, multi-agent context, and full-lifecycle security considerations.
Note (b): You can use specially engineered prompt templates to augment your threat-modeling process with LLMs. Several examples of STRIDE and PASTA LLM Threat Modeling Prompt Templates are available in reference [26]. These templates provide reusable, structured prompts that guide Large Language Models to perform threat-modeling tasks with consistency and accuracy.
Expand Down Expand Up @@ -58,7 +58,7 @@ Alternatively, if STRIDE is adopted directly as the primary threat modeling appr
In PASTA’s seven‐stage process, we’ll enhance the Threat Analysis phase by incorporating MITRE ATLAS’s database of AI‐specific adversarial tactics, such as evasion, poisoning, model extraction, and inference attacks, into our threat mapping. This integration ensures our risk‐centric model aligns with business priorities and technical scope, while directly informing a targeted suite of offensive AI tests against the most critical attack vectors. AI-specific adversarial tactics such as evasion, poisoning, and model extraction are prime targets for specialized AI security assessments like red teaming. This focus is formally captured in the OWASP AI Red Teaming Framework [14], which defines how to simulate and evaluate these attack vectors against AI systems.
Effective threat modeling begins by scoping the analysis around the critical assets you must protect. To do this, you first decompose the system’s architecture into its essential components, services, data stores, interfaces, and supporting infrastructure. You then map out how these pieces interact by drawing data flow diagrams that trace information end-to-end, highlight entry and exit points, and establish trust boundaries. By visualizing where data is stored, processed, and transmitted, you can pinpoint the exact assets at risk and systematically identify potential threats and vulnerabilities against each component and boundary. This structured approach ensures your threat model remains focused, comprehensive, and aligned with the organization’s security priorities.
These scoping and decomposition activities, identifying critical assets, breaking the system into core components, and using data flow diagrams to map end-to-end interactions and trust boundaries are foundational steps shared by many threat-modeling methodologies, from STRIDE to PASTA and beyond, ensuring a consistent, thorough approach to identifying and prioritizing risks.
By focusing on the SAIF-aligned layers, Application, Data, Model, and Infrastructure, we intentionally keep our threat analysis at a high architectural level. This ensures broad coverage of AI-specific risks without delving into every sub-component of the system.
By focusing on the SAIF-aligned layers, Application, Data, Model, and Infrastructure, we intentionally keep our threat analysis at a high architectural level. This ensures broad coverage of AI-specific risks without delving into every subcomponent of the system.
In this AI threat model, we map threats, including AI-specific threats across the application, data, model, and infrastructure layers to ensure comprehensive coverage. Threat mitigations are defined as testable requirements, with validation activities documented in this guide. The goal is to provide a complete set of tests to assess the AI system’s security posture against the identified threats (Note).
Note: It’s important to note that the OWASP AI Testing Guide is scoped to post-deployment security assessments and does not cover the broader MLOps lifecycle. For teams seeking guidance on embedding adversarial robustness tests earlier during data preparation, model training, and CI/CD pipelines, we recommend the white paper in ref [16] Securing AI/ML Systems in the Age of Information Warfare which provides an excellent deep dive into adversarial testing techniques within the AI/ML development process as well as ref[17] John Sotiroupulos book.

Expand All @@ -75,7 +75,7 @@ Note: While RAG (Retrieval-Augmented Generation) isn’t explicitly defined in t
The application layer encompasses the application and any associated agents or plugins. It interfaces with users for input and output and with the AI model for processing and response. Agents and plugins extend functionality but also introduce additional transitive risks that must be managed.
The “Application” refers to the product, service, or feature that leverages an AI model to deliver functionality. Applications may be user-facing, such as a customer service chatbot, or service-oriented, where internal systems interact with the model to support upstream processes.
The “Agent/plugin” refers to a service, application, or supplementary model invoked by an AI application or model to perform a specific task, often referred to as ‘tool use.’ Because agents or plugins can access external data or initiate requests to other models, each invocation introduces additional transitive risks, potentially compounding the existing risks inherent in the AI development process.
The application layer can be decomposed in the following sub-components:
The application layer can be decomposed in the following subcomponents:
- **The User (SAIF #1):** this is the person or system initiating requests and receiving responses.
- **The User Input (SAIF #2):** these are inputs (queries, commands) submitted by the user.
- **The User Output (SAIF #3):** These are output such as answers to user actions that are returned by the application to the user.
Expand All @@ -87,15 +87,15 @@ The application layer can be decomposed in the following sub-components:

### Model Layer
The Model layer covers the core AI or ML components themselves, the logic, parameters, and runtime that transform inputs into outputs. It sits between the application (and any agents/plugins) and the underlying infrastructure or data. Because this layer embodies the “black box” of AI, it demands careful handling of inputs, outputs, and inference operations to prevent poisoning, leakage, or misuse.
The model layer can be decomposed in the following sub-components:
The model layer can be decomposed in the following subcomponents:
- **The Input Handling (SAIF #7) (Note):** whose purpose is to validate and sanitize all data, prompts, or feature vectors before they reach the model to prevent injection attacks, data poisoning, or malformed inputs that could lead to unintended behavior. The input handling comprises three key functions: an Input Validator to clean or reject bad data, Authentication & Authorization to allow only authorized callers, and a Rate Limiter to prevent denial-of-service or brute-force attacks.
- **The Output Handling (SAIF #8) (Note):** whose purpose is to filter, redact, or post-process model outputs to ensure they do not expose sensitive training data, violate privacy, or produce harmful content. It includes an Output Filter to detect and block harmful or disallowed content, Sanitization & Redaction to remove sensitive or private information, and a Response Validator to confirm outputs meet format and business rules before delivery.
- **The Model Usage (SAIF #9):** whose purpose is to execute the model against approved inputs in a controlled, auditable environment, ensuring that inference logic cannot be tampered with or subverted at runtime. It includes: the Inference Engine for loading weights and computing outputs, Policy Enforcement to apply guardrails (e.g., token limits, safe decoding), and an Audit Logger to record inputs, model versions, and outputs for traceability.
- Note: The two dotted lines from SAIF #5 (Agents/Plugins) to SAIF #7 (Input Handling) and SAIF #8 (Output Handling) reflect how plugins can dynamically alter prompts or post-process model outputs.

### Infrastructure Layer
The infrastructure layer provides the foundational compute, networking, storage, and orchestration services that host and connect all other AI system components. It ensures resources are provisioned, isolated, and managed securely, supporting everything from data processing and model training to inference and monitoring.
The infrastructure layer can be decomposed in the following sub-components (Note):
The infrastructure layer can be decomposed in the following subcomponents (Note):
- **Model Storage Infrastructure (SAIF #10):** This component safeguards the storage and retrieval of model artifacts, such as weight files, configuration data, and versioned metadata, ensuring they remain confidential, intact, and available. An artifact repository maintains versioning and enforces encryption at rest, while an integrity verifier computes and checks cryptographic hashes (e.g., SHA-256) on each upload and download to detect tampering. A key management service issues and rotates encryption keys under least-privilege policies, preventing unauthorized decryption of stored models.
- **Model Serving Infrastructure (SAIF #11):** This component provides the runtime environment in which models execute inference requests. It isolates the model execution process from other workloads, enforces resource quotas and rate limits, and ensures that only properly formatted inputs reach the model. Health-monitoring mechanisms detect failures or performance degradations, and automatic scaling or load-balancing ensures uninterrupted availability under varying demand.
- **Model Evaluation (SAIF #12):** This component measures model performance, fairness, and robustness before and after deployment. A validation suite runs the model against reserved test sets—including adversarial or edge-case inputs—and collects metrics on accuracy, bias, and error rates. Drift-detection tools compare new outputs to historical baselines to flag significant deviations, and reporting dashboards surface any regressions or policy violations for corrective action.
Expand All @@ -105,7 +105,7 @@ The infrastructure layer can be decomposed in the following sub-components (Note

### Data Layer
The Data layer underpins every AI system by supplying the raw and processed information that models consume. It encompasses the entire lifecycle of data, from initial collection and ingestion through transformation, storage, and provisioning for training or inference and ensures that data remains accurate, trustworthy, and compliant with privacy and security policies. Robust controls in this layer protect against poisoning, leakage, and unauthorized access, forming the foundation for reliable, responsible AI outcomes.
The data layer can be decomposed in the following sub-components:
The data layer can be decomposed in the following subcomponents:
- **Training Data (SAIF #16):** Training data consists of curated, labeled examples used to teach the model how to recognize patterns and make predictions. In a secure AI pipeline, organizations establish strict provenance and versioning for training datasets to guarantee integrity: every record’s origin, modification history, and access events are logged and auditable. By enforcing encryption-at-rest and role-based permissions on training repositories, the system prevents unauthorized tampering; any illicit change to the training corpus would corrupt the model’s learning process and open the door to adversarial manipulation.
- **Data Filtering and Processing (SAIF #17):** Before feeding raw inputs into model pipelines, data undergoes rigorous filtering and processing steps. This includes schema validation, anomaly detection to strip out corrupt or malicious entries, and privacy-preserving transformations like anonymization or pseudonymization. Secure processing frameworks execute these tasks in isolated environments, with reproducible pipelines that record every transformation applied. By embedding fine-grained access controls and change-tracking at each stage, the system ensures that only vetted, sanitized data influences the model, mitigating risks from both accidental errors and deliberate data-poisoning attacks.
- **Data Sources (SAIF #18) (note):** An AI system’s data may originate from internal operational databases, user-generated inputs, IoT sensors, or third-party providers. Internal sources are governed by organizational policies and monitored for access anomalies.
Expand Down
Loading