github.com/gemaraproj/gemara@v1.3.0

test/test-data/good-aigf.yaml raw

  1metadata:
  2  id: FINOS-AIR
  3  type: GuidanceCatalog
  4  gemara-version: "1.1.0"
  5  description: >
  6    A comprehensive collection of risks and mitigations that support
  7    on-boarding, development of, and running Generative AI solutions.
  8  author:
  9    id: finos
 10    name: FINOS
 11    type: Human
 12  version: 0.1.0
 13  mapping-references:
 14    - id: AIR-PRIN
 15      title: AI Governance Framework Principles
 16      version: 0.1.0
 17      url: "https://aigf.finos.org/principles"
 18      description: Core principles underpinning the FINOS AI Governance Framework
 19    - id: AIR-VEC
 20      title: AI Governance Framework Risk Vectors
 21      version: 0.1.0
 22      url: "https://aigf.finos.org/risks"
 23      description: AIGF risks expressed as Gemara vectors
 24title: AI Governance Framework
 25type: Framework
 26front-matter: |
 27  AI, especially Generative AI, is reshaping financial services, enhancing products, client interactions, and productivity. However, challenges like hallucinations and model unpredictability make safe deployment complex. Rapid advancements require flexible governance.
 28  Financial institutions are eager to adopt AI but face regulatory hurdles. Existing frameworks may not address AI's unique risks, necessitating an adaptive governance model for safe and compliant integration.
 29  The following framework has been developed by FINOS (Fintech Open Source Foundation) members, providing a comprehensive catalogue of risks and associated mitigations. We suggest using our heuristic risk identification framework to determine which risks are most relevant for a given use case.
 30groups:
 31  - id: DET
 32    title: Detective
 33    description: Detection and Continuous Improvement
 34  - id: PREV
 35    title: Preventive
 36    description: Prevention and Risk Mitigation
 37guidelines:
 38  - id: AIR-PREV-002
 39    group: PREV
 40    title: Data Filtering From External Knowledge Bases
 41    objective: >
 42      This control addresses the critical need to sanitize, filter, and appropriately 
 43      manage sensitive information when AI systems ingest data from internal knowledge 
 44      sources such as wikis, document management systems, databases, or collaboration 
 45      platforms (e.g., Confluence, SharePoint, internal websites). The primary objective 
 46      is to prevent the inadvertent exposure, leakage, or manipulation of confidential 
 47      organizational knowledge when this data is processed by AI models, converted into 
 48      embeddings for vector databases, or used in Retrieval Augmented Generation (RAG) systems.
 49      Given that many AI applications, particularly RAG systems, rely on internal knowledge bases to 
 50      provide contextually relevant and organization-specific responses, ensuring that sensitive 
 51      information within these sources is appropriately handled is paramount for maintaining 
 52      data confidentiality and preventing unauthorized access.
 53    rationale:
 54      importance: >
 55        This control is particularly important given the evolving nature of AI technologies 
 56        and the sophisticated ways they interact with and process large volumes of organizational 
 57        information. A proactive approach to data sanitization helps maintain confidentiality, 
 58        integrity, and compliance while enabling the organization to benefit from AI capabilities.
 59      goals:
 60        - "Prevention of Data Leakage: Significantly reduces the risk of sensitive organizational information being inadvertently exposed through AI system outputs or stored in less secure external services."
 61        - "Regulatory Compliance: Helps meet requirements under data protection regulations (e.g., GDPR, CCPA, GLBA) that mandate the protection of personal and sensitive business information."
 62        - "Intellectual Property Protection: Safeguards valuable trade secrets, strategic information, and proprietary data from unauthorized disclosure or competitive exposure."
 63        - "Reduced Attack Surface: By controlling the information that enters AI operational environments, organizations minimize the potential impact of AI-specific attacks like prompt injection or data extraction attempts."
 64        - "Enhanced Trust and Confidence: Builds stakeholder confidence in AI systems by demonstrating rigorous data protection practices."
 65        - "Compliance with Internal Data Governance: Supports adherence to internal data classification and handling policies within AI contexts."
 66        - "Mitigation of Insider Risk: Reduces the risk of sensitive information being accessed by unauthorized internal users through AI interfaces."
 67    see-also:
 68      - AIR-DET-001
 69      - AIR-PREV-006
 70      - AIR-DET-016
 71    statements:
 72      - id: AIR-PREV-002.1
 73        title: Rigorous Data Cleansing and Anonymization at Ingestion
 74        text: "Identify and remove or appropriately anonymize sensitive details to ensure that data fed into the AI system is free from information that could pose a security or privacy risk if inadvertently exposed."
 75        recommendations:
 76          - >
 77            Pre-Processing Review and Cleansing: Before any information from
 78            internal knowledge sources is ingested by an AI system, it must
 79            undergo a thorough review and cleansing process to identify and
 80            remove or appropriately anonymize sensitive details.
 81          - >
 82            Categories of Data to Target for Filtering: Personally
 83            Identifiable Information (PII), Proprietary Business Information,
 84            Sensitive Internal Operational Data, Confidential Customer Data,
 85            and Regulatory or Compliance-Sensitive Information.
 86          - >
 87            Filtering and Anonymization Methods: Data Masking, Redaction,
 88            Generalization, Tokenization, and Synthetic Data Generation.
 89      - id: AIR-PREV-002.2
 90        title: Segregation for Highly Sensitive Data
 91        text: ""
 92        recommendations:
 93          - >
 94            Isolated AI Systems for Critical Data: For datasets containing
 95            exceptionally sensitive information that cannot be adequately
 96            protected through standard cleansing, implement separate, isolated
 97            AI systems with stricter access controls, enhanced encryption,
 98            and limited network connectivity.
 99          - >
100            Access Domain-Based Segregation: Segment data and AI system access
101            based on clearly defined access domains that mirror the
102            organization's existing data classification and access control
103            structures.
104      - id: AIR-PREV-002.3
105        title: Filtering AI System Outputs (Secondary Defense)
106        text: ""
107        recommendations:
108          - >
109            Response Filtering and Validation: Responses generated by the AI
110            system should be monitored and filtered before being presented to
111            users, acting as a safety net to detect sensitive data that might
112            have bypassed initial input cleansing.
113          - >
114            Contextual Output Analysis: Implement intelligent filtering that
115            considers the context of the user's query and their authorization
116            level to determine what information should be included in the
117            response.
118      - id: AIR-PREV-002.4
119        title: Integration with Source System Access Controls
120        text: ""
121        recommendations:
122          - "Respect Original Permissions: Design the AI system to respect and replicate the original access control permissions from source systems."
123          - "Dynamic Source Querying: For real-time RAG systems, consider querying source systems dynamically while respecting user permissions, rather than pre-processing all data indiscriminately."
124      - id: AIR-PREV-002.5
125        title: Monitoring and Continuous Improvement
126        text: >
127          Periodically audit the effectiveness of data filtering processes
128          by sampling processed data and checking for any sensitive
129          information that may have been missed.
130        recommendations:
131          - "Regular Review of Filtering Effectiveness: Periodically audit the effectiveness of data filtering processes by sampling processed data."
132          - "Feedback Loop Integration: Establish mechanisms for users and reviewers to report instances where sensitive information may have been inappropriately exposed."
133          - "Threat Intelligence Integration: Stay informed about new types of data leakage vectors and attack techniques that might affect AI systems."
134    principles:
135      - reference-id: AIR-PRIN
136        entries:
137          - reference-id: AIR-PRIN-001
138            remarks: Filtering and anonymization applied before data enters AI pipelines
139          - reference-id: AIR-PRIN-002
140            remarks: Filtering strategies respect source data sensitivity and access controls
141          - reference-id: AIR-PRIN-003
142            remarks: Only necessary data included in AI systems, de-identified where possible
143          - reference-id: AIR-PRIN-004
144            remarks: Multiple filtering layers at ingestion, processing, and output
145          - reference-id: AIR-PRIN-005
146            remarks: Audit trails document what filtering has been applied and why
147    vectors:
148      - reference-id: AIR-VEC
149        entries:
150          - reference-id: AIR-RC-001-01
151          - reference-id: AIR-RC-001-02
152          - reference-id: AIR-RC-001-03
153          - reference-id: AIR-RC-001-04
154          - reference-id: AIR-RC-001-05
155          - reference-id: AIR-SEC-009-01
156          - reference-id: AIR-SEC-009-02
157          - reference-id: AIR-SEC-009-03
158          - reference-id: AIR-SEC-009-04
159    state: Active
160
161  - id: AIR-PREV-003
162    group: PREV
163    title: User/App/Model Firewalling
164    objective: >
165      User/App/Model Firewalling encompasses the set of security controls
166      applied at the boundaries between users, applications, AI models,
167      and supporting data stores such as RAG databases. When internal
168      company information is used to enrich a RAG database, especially if
169      this involves processing by external services, this data and the
170      external communication pathways must be carefully managed and
171      secured. Any proprietary or sensitive information sent to an external
172      service for such processing requires rigorous filtering before
173      transmission to prevent data leakage.
174    rationale:
175      importance: >
176        Implementing comprehensive user/app/model firewalling provides
177        critical security benefits including attack prevention, data
178        protection, service availability, reputation protection, and
179        compliance support. Firewalling blocks prompt injection attacks
180        and malicious inputs before they reach AI models, prevents
181        sensitive information leakage through AI outputs or RAG processing,
182        and helps meet regulatory requirements for data handling and
183        system security.
184      goals:
185        - "RAG Data Ingestion: Filter sensitive or private data before transmitting internal information to external services for embedding creation"
186        - "User Input to AI Model: Detect and block malicious or abusive user inputs such as Prompt Injection attacks"
187        - "AI Model Output: Detect excessively long responses, format deviations, evasion patterns, data leakage, and inappropriate content"
188    see-also:
189      - AIR-PREV-017
190      - AIR-PREV-008
191      - AIR-DET-015
192    statements:
193      - id: AIR-PREV-003.1
194        title: RAG Data Ingestion Filtering
195        text: ""
196        recommendations:
197          - >
198            RAG Database Security: While it's often more practical to
199            pre-process and filter data for RAG systems before sending it
200            for external embedding creation, organizations might also
201            consider in-line filters for real-time checks. Once internal
202            information is converted into embeddings and stored in vector
203            databases, the data becomes largely opaque to traditional
204            security tools.
205          - >
206            Filtering Efficacy: Static filters based on regular expressions
207            or keyword blocklists are effective for well-defined patterns
208            but less effective at identifying nuanced issues such as
209            generic private information or subtle Prompt Injection attacks.
210          - >
211            Streaming Outputs: Streaming responses improve user experience
212            but implementing output filtering can be challenging. An
213            approach is to stream while performing on-the-fly detection,
214            cancelling output if an issue is found.
215      - id: AIR-PREV-003.2
216        title: Remediation Techniques
217        text: ""
218        recommendations:
219          - >
220            Basic Filters: Simple static checks using blocklists and
221            regular expressions can detect rudimentary attacks or policy
222            violations.
223          - >
224            System Prompts (Caution Advised): While system prompts can
225            instruct an LLM on what to avoid, they are generally not a
226            robust security control. Attackers can often bypass these
227            instructions.
228          - >
229            LLM as a Judge: A secondary, specialized LLM analyzes user
230            queries and the primary LLM's responses, categorizing
231            inputs/outputs for various risks such as prompt injection,
232            abuse, hate speech, and data leakage.
233          - >
234            Human Feedback Loop: Implementing a system where users can
235            easily report problematic AI responses provides a valuable
236            complementary control.
237      - id: AIR-PREV-003.3
238        title: Additional Considerations
239        text: ""
240        recommendations:
241          - >
242            API Security and Observability: Implementing a comprehensive
243            API monitoring and security solution offers benefits beyond
244            AI-specific threats. A security proxy can enforce encrypted
245            communication between all AI system components.
246          - >
247            Logging and Analysis: Detailed logging of interactions is
248            essential for understanding user behavior, system performance,
249            and detection of sophisticated attacks or anomalies.
250      - id: AIR-PREV-003.4
251        title: Challenges and Considerations
252        text: >
253          RAG Database Security: Vector databases make traditional security
254          filtering difficult once data is embedded. Filtering Efficacy:
255          Static filters may miss nuanced attacks or sophisticated content.
256          Streaming Outputs: Real-time filtering creates trade-offs between
257          security and user experience.
258    principles:
259      - reference-id: AIR-PRIN
260        entries:
261          - reference-id: AIR-PRIN-001
262            remarks: RAG data filtered before transmission to external services
263          - reference-id: AIR-PRIN-004
264            remarks: Layered filtering at user input, model output, and RAG ingestion boundaries
265          - reference-id: AIR-PRIN-005
266            remarks: Logging and analysis of all interactions for audit and anomaly detection
267    vectors:
268      - reference-id: AIR-VEC
269        entries:
270          - reference-id: AIR-OP-007-01
271          - reference-id: AIR-OP-007-02
272          - reference-id: AIR-OP-007-03
273          - reference-id: AIR-SEC-010-01
274          - reference-id: AIR-SEC-010-02
275          - reference-id: AIR-SEC-010-03
276          - reference-id: AIR-OP-018
277          - reference-id: AIR-OP-020
278    state: Active