Post

AWS Paper - Methodology for incident response on Gen-AI workloads

Posted
By Grace L
16 min read

ref:

  • https://aws.amazon.com/blogs/security/methodology-for-incident-response-on-generative-ai-workloads/

Methodology for incident response on Gen-AI workloads

The AWS Customer Incident Response Team (CIRT) has developed a methodology to investigate security incidents involving Gen-AI-based applications.

To respond to security events related to a Gen-AI workload, you should still follow the guidance and principles outlined in the AWS Security Incident Response Guide

Methodology for incident response on Gen-AI workloads

  • seven elements to consider when triaging and responding to a security event on a Gen-AI workload.

Components of a Gen-AI workload

Screenshot 2024-10-03 at 09.16.26

Generative AI applications include the following five components:

  • An organization: owns or is responsible for infrastructure, Gen-AI applications, and the organization’s private data.

  • Infrastructure: isn’t specifically related to the Gen-AI application itself, include databases, backend servers, and websites.

  • Generative AI applications, which include the following:

    • Foundation models – AI models with a large number of parameters and trained on a massive amount of diverse data.
    • Custom models – models that are fine-tuned or trained on an organization’s specific data and use cases, tailored to their unique requirements.
    • Guardrails – mechanisms or constraints to help make sure that the Gen-AI application operates within desired boundaries. Examples include content filtering, safety constraints, or ethical guidelines.
    • Agents – workflows that enable Gen-AI applications to perform multistep tasks across company systems and data sources.
    • Knowledge bases – repositories of domain-specific knowledge, rules, or data that the Gen-AI application can access and use.
    • Training data – data used to train, fine-tune, or augment the Gen-AI application’s models, including data for techniques such as retrieval augmented generation (RAG)

      Note : Training data is distinct from an organization’s private data. A Gen-AI application might not have direct access to private data, although this is configured in some environments.

    • Plugins – additional software components or extensions that you can integrate with the Gen-AI application to provide specialized functionalities or access to external services or data sources.

  • Private data refers to the customer’s privately stored, confidential data that the Gen-AI resources or applications aren’t intended to interact with during normal operation.

  • Users are the identities that can interact with or access the Gen-AI application. They can be human or non-human (such as machines).

Prepare for incident response on Gen-AI workloads

You should prepare for a security event across three domains: people, process, and technology.

Preparation for a security event that’s related to a Gen-AI workload should include the following:

Important: Logs can contain sensitive information. To help protect this information, you should

Methodology for incident response on Gen-AI workloads

After the preparation items, use the Methodology for incident response on Gen-AI workloads for active response, to rapidly triage an active security event involving a Gen-AI application.

  • The methodology has seven elements, each element describes a method by which the components can interact with another component or a method by which a component can be modified.

    • Access

  • Consideration of these elements will help guide the actions during the Operations phase of a security incident, which includes detection, analysis, containment, eradication, and recovery phases.

Access

  • Determine the designed or intended access patterns for the organization that hosts the components of the Gen-AI application, and look for deviations or anomalies from those patterns. Consider whether the application is accessible externally or internally because that will impact the analysis.

  • determine whether an organization has access to their AWS account.
  • identify anomalous and potential unauthorized access to the environment,
    • use Amazon GuardDuty.
    • If the application is accessible externally, the threat actor might not be able to access the AWS environment directly and thus GuardDuty won’t detect it. The way that you’ve set up authentication to the application will drive how you detect and analyze unauthorized access.
  • If evidence of unauthorized access to the AWS account or associated infrastructure exists

Infrastructure changes

  • Review the supporting infrastructure, such as servers, databases, serverless computing instances, and internal or external websites, to determine if it was accessed or changed.

  • To investigate infrastructure changes, you can analyze CloudTrail logs for modifications of in-scope resources, or analyze other operating system logs or database access logs.

Analyze the infrastructure changes of an application

  • the control plane and data plane.
    • example, imagine that Amazon API Gateway was used for authentication to the downstream components of the Gen-AI application and that other ancillary resources were interacting with the application.
  • have additional logging to be turned on to review changes made on the operating system of the resource.

AI changes

Unauthorized changes can include, but are not limited to, system prompts, application code, guardrails, and model availability.

  • Investigate whether users have accessed components of the Gen-AI application and whether they made changes to those components.

  • Look for signs of unauthorized activities, such as the creation or deletion of custom models, modification of model availability, tampering or deletion of Gen-AI logging capabilities, tampering with the application code, and removal or modification of Gen-AI guardrails.

Data store changes

Typically, you use and access a data store and knowledge base through model invocation.

  • Determine the designed or intended data access patterns, whether users accessed the data stores of the Gen-AI application, and whether they made changes to these data stores.

  • look for the addition or modification of agents to a Gen-AI application.

    • if an unauthorized user gains access to the environment, they can create, change, or delete the data sources and knowledge bases that the Gen-AI applications integrate with.
    • This could cause data or model exfiltration or destruction, as well as data poisoning, and could create a denial-of-service condition for the model.

Invocation

  • Analyze invocations of Gen-AI models, including the strings and file inputs, for threats, such as prompt injection or malware. You can use the OWASP Top 10 for LLM as a starting point to understand invocation related threats, and you can use invocation logs to analyze prompts for suspicious patterns, keywords, or structures that might indicate a prompt injection attempt.

  • The logs also capture the model’s outputs and responses, enabling behavioral analysis to help identify uncharacteristic or unsafe model behavior indicative of a prompt injection. You can use the timestamps in the logs for temporal analysis to help detect coordinated prompt injection attempts over time and collect information about the user or system that initiated the model invocation, helping to identify the source of potential exploits.

Amazon Bedrock uses specific APIs to register model invocation.

  • When a model in Amazon Bedrock is invoked, CloudTrail logs it.

  • However, to determine the prompts that were sent to the Gen-AI model and the output response that was received from it, you must have configured model invocation logging.

    • crucial because they can reveal important information, such as whether a threat actor tried to get the model to divulge information from the data stores or release data that the model was trained or fine-tuned on.

    • For example, the logs could reveal if a threat actor attempted to prompt the model with carefully crafted inputs that were designed to extract sensitive data, bypass security controls, or generate content that violates the policies.
    • Using the logs, learn whether the model was used to generate misinformation, spam, or other malicious outputs that could be used in a security event.

Note : For services such as Amazon Bedrock, invocation logging is disabled by default. recommend enable data events and model invocation logging for Gen-AI services, where available. However, the organization might not want to capture and store invocation logs for privacy and legal reasons. One common concern is users entering sensitive data as input, which widens the scope of assets to protect. This is a business decision that should be taken into consideration.

Private data

  • Determine whether the in-scope Gen-AI application was designed to have access to private or confidential data. Then look for unauthorized access to, or tampering with, that data.

From an architectural standpoint, Gen-AI applications shouldn’t have direct access to an organization’s private data.

  • should classify data used to train a Gen-AI application or for RAG use as data store data and segregate it from private data, unless the Gen-AI application uses the private data (for example, in the case where a Gen-AI application is tasked to answer questions about medical records for a patient).

  • One way to help make sure that an organization’s private data is segregated from Gen-AI applications is to use a separate account and to authenticate and authorize access as necessary to adhere to the principle of least privilege.

Agency

  • Agency refers to the ability of applications to make changes to an organization’s resources or take actions on a user’s behalf.

    • For example, a Gen-AI application might be configured to generate content that is then used to send an email, invoking another resource or function to do so.

  • determine whether the Gen-AI application has the ability to invoke other functions. Then, investigate whether unauthorized changes were made or if the Gen-AI application invoked unauthorized functions.

  • Excessive agency for an LLM refers to an AI system that has too much autonomy or decision-making power, leading to unintended and potentially harmful consequences. This can happen when an LLM is deployed with insufficient oversight, constraints, or alignment with human values, resulting in the model making choices that diverge from what most humans would consider beneficial or ethical.

The following table lists some questions to help you address the seven elements of the methodology. Use the answers to guide the response.

TopicQuestions to address
AccessDo you still have access to the computing environment?
Is there continued evidence of unauthorized access to the organization?
Infrastructure changesWere supporting infrastructure resources accessed or changed?
AI changesWere the AI models, code, or resources accessed or changed?
Data store changesWere the data stores, knowledge bases, agents, plugins, or training data accessed or tampered with?
InvocationWhat data, strings, or files were sent as input to the model?
What prompts were sent?
What responses were produced?
Private dataWhat private or confidential data do Gen-AI resources have access to?
Was private data changed or tampered with?
AgencyCan the Gen-AI application resources be used to start computing services in an organization, or do the Gen-AI resources have the authority to make changes?
Were unauthorized changes made?

Example incident

An example security event where an unauthorized user compromises a Gen-AI application that’s hosted on AWS by using credentials that were exposed on a public code repository.

Access

Infrastructure changes

  • some common names for control plane events in CloudTrail for this element:

    • ec2:RunInstances
    • ec2:StartInstances
    • ec2:TerminateInstances
    • ecs:CreateCluster
    • cloudformation:CreateStack
    • rds:DeleteDBInstance
    • rds:ModifyDBClusterSnapshotAttribute

AI changes

Unauthorized changes can include, but are not limited to, system prompts, application code, guardrails, and model availability.

  • Internal user access to the Gen-AI resources that AWS hosts are logged in CloudTrail

  • event sources:

    • bedrock.amazonaws.com
    • sagemaker.amazonaws.com
    • qbusiness.amazonaws.com
    • q.amazonaws.com

  • event names that would represent Gen-AI resource log tampering:

    • bedrock:PutModelInvocationLoggingConfiguration
    • bedrock:DeleteModelInvocationLoggingConfiguration

  • event names that would represent access to the AI/ML model service configuration:

    • bedrock:GetFoundationModelAvailability
    • bedrock:ListProvisionedModelThroughputs
    • bedrock:ListCustomModels
    • bedrock:ListFoundationModels
    • bedrock:ListProvisionedModelThroughput
    • bedrock:GetGuardrail
    • bedrock:DeleteGuardrail

In our example scenario

  • the unauthorized user has gained access to the AWS account.

  • Now imagine that the compromised user has a policy attached that grants them full access to all resources. With this access, the unauthorized user can enumerate each component of Amazon Bedrock and identify the knowledge base and guardrails that are part of the application.

  • The unauthorized user then requests model access to other foundation models (FMs) within Amazon Bedrock and removes existing guardrails.

  • The access to other foundation models could indicate that the unauthorized user intends to use the Gen-AI application for their own purposes, and the removal of guardrails minimizes filtering or output checks by the model.

  • AWS recommends that you implement fine-grained access controls by using  IAM policies and resource-based policies  to restrict access to only the necessary Amazon Bedrock resources, AWS Lambda functions, and other components that the application requires.

  • Also, you should enforce the use of MFA for IAM users, roles, and service accounts with access to critical components such as Amazon Bedrock and other components of the Gen-AI application.

Data store changes

  • event names that would represent changes to AI/ML data sources:

    • bedrock:CreateDataSource
    • bedrock:GetKnowledgeBase
    • bedrock:DeleteKnowledgeBase
    • bedrock:CreateAgent
    • bedrock:DeleteAgent
    • bedrock:InvokeAgent
    • bedrock:Retrieve

    • bedrock:RetrieveAndGenerate

    • For the full list of possible actions, see the Amazon Bedrock API Reference.

In this scenario, we have established that the unauthorized user has full access to the Gen-AI application and that some enumeration took place.

  • The unauthorized user then identified the S3 bucket that was the knowledge base for the Gen-AI application and uploaded inaccurate data, which corrupted the LLM.
  • For examples of this vulnerability, see the section LLM03 Training Data Poisoning in the OWASP TOP 10 for LLM Applications.

Invocation

In our example scenario, imagine that model invocation wasn’t enabled, the incident responder

  • couldn’t collect invocation logs to see the model input or output data for unauthorized invocations.
  • wouldn’t be able to determine the prompts and subsequent responses from the LLM.
  • couldn’t see the full request data, response data, and metadata associated with invocation calls.

Event names in model invocation logs that would represent model invocation logging in Amazon Bedrock include:

  • bedrock:InvokeModel
  • bedrock:InvokeModelWithResponseStream
  • bedrock:Converse
  • bedrock:ConverseStream

sample log entry for Amazon Bedrock model invocation logging:

Figure 2: sample model invocation log including prompt and response

Private data

From an architectural standpoint, Gen-AI applications shouldn’t have direct access to an organization’s private data.

  • should classify data used to train a Gen-AI application or for RAG use as data store data and segregate it from private data, unless the Gen-AI application uses the private data (for example, in the case where a Gen-AI application is tasked to answer questions about medical records for a patient).

  • One way to help make sure that an organization’s private data is segregated from Gen-AI applications is to use a separate account and to authenticate and authorize access as necessary to adhere to the principle of least privilege.

Agency

In our example scenario

  • the Gen-AI application has excessive permissions to services that aren’t required by the application.

  • Imagine that the application code was running with an execution role with full access to Amazon Simple Email Service (Amazon SES). This could allow for the unauthorized user to send spam emails on the users’ behalf in response to a prompt.

  • You could help prevent this by limiting permission and functionality of the Gen-AI application plugins and agents.
  • During an investigation, while analyzing the logs, both the sourceIPAddress and the userAgent fields will be associated with the Gen-AI application (for example, sagemaker.amazonaws.com, bedrock.amazonaws.com, or q.amazonaws.com ).
    • Some examples of services that might commonly be called or invoked by other services are Lambda, Amazon SNS, and Amazon SES.
51AI, Paper
LLM
This post is licensed under CC BY 4.0 by the author.

Comments powered by Disqus.