1metadata:
2 id: FINOS-AIR
3 type: GuidanceCatalog
4 gemara-version: "1.1.0"
5 description: >
6 A comprehensive collection of risks and mitigations that support
7 on-boarding, development of, and running Generative AI solutions.
8 author:
9 id: finos
10 name: FINOS
11 type: Human
12 version: 0.1.0
13 mapping-references:
14 - id: AIR-PRIN
15 title: AI Governance Framework Principles
16 version: 0.1.0
17 url: "https://aigf.finos.org/principles"
18 description: Core principles underpinning the FINOS AI Governance Framework
19 - id: AIR-VEC
20 title: AI Governance Framework Risk Vectors
21 version: 0.1.0
22 url: "https://aigf.finos.org/risks"
23 description: AIGF risks expressed as Gemara vectors
24title: AI Governance Framework
25type: Framework
26front-matter: |
27 AI, especially Generative AI, is reshaping financial services, enhancing products, client interactions, and productivity. However, challenges like hallucinations and model unpredictability make safe deployment complex. Rapid advancements require flexible governance.
28 Financial institutions are eager to adopt AI but face regulatory hurdles. Existing frameworks may not address AI's unique risks, necessitating an adaptive governance model for safe and compliant integration.
29 The following framework has been developed by FINOS (Fintech Open Source Foundation) members, providing a comprehensive catalogue of risks and associated mitigations. We suggest using our heuristic risk identification framework to determine which risks are most relevant for a given use case.
30groups:
31 - id: DET
32 title: Detective
33 description: Detection and Continuous Improvement
34 - id: PREV
35 title: Preventive
36 description: Prevention and Risk Mitigation
37guidelines:
38 - id: AIR-PREV-002
39 group: PREV
40 title: Data Filtering From External Knowledge Bases
41 objective: >
42 This control addresses the critical need to sanitize, filter, and appropriately
43 manage sensitive information when AI systems ingest data from internal knowledge
44 sources such as wikis, document management systems, databases, or collaboration
45 platforms (e.g., Confluence, SharePoint, internal websites). The primary objective
46 is to prevent the inadvertent exposure, leakage, or manipulation of confidential
47 organizational knowledge when this data is processed by AI models, converted into
48 embeddings for vector databases, or used in Retrieval Augmented Generation (RAG) systems.
49 Given that many AI applications, particularly RAG systems, rely on internal knowledge bases to
50 provide contextually relevant and organization-specific responses, ensuring that sensitive
51 information within these sources is appropriately handled is paramount for maintaining
52 data confidentiality and preventing unauthorized access.
53 rationale:
54 importance: >
55 This control is particularly important given the evolving nature of AI technologies
56 and the sophisticated ways they interact with and process large volumes of organizational
57 information. A proactive approach to data sanitization helps maintain confidentiality,
58 integrity, and compliance while enabling the organization to benefit from AI capabilities.
59 goals:
60 - "Prevention of Data Leakage: Significantly reduces the risk of sensitive organizational information being inadvertently exposed through AI system outputs or stored in less secure external services."
61 - "Regulatory Compliance: Helps meet requirements under data protection regulations (e.g., GDPR, CCPA, GLBA) that mandate the protection of personal and sensitive business information."
62 - "Intellectual Property Protection: Safeguards valuable trade secrets, strategic information, and proprietary data from unauthorized disclosure or competitive exposure."
63 - "Reduced Attack Surface: By controlling the information that enters AI operational environments, organizations minimize the potential impact of AI-specific attacks like prompt injection or data extraction attempts."
64 - "Enhanced Trust and Confidence: Builds stakeholder confidence in AI systems by demonstrating rigorous data protection practices."
65 - "Compliance with Internal Data Governance: Supports adherence to internal data classification and handling policies within AI contexts."
66 - "Mitigation of Insider Risk: Reduces the risk of sensitive information being accessed by unauthorized internal users through AI interfaces."
67 see-also:
68 - AIR-DET-001
69 - AIR-PREV-006
70 - AIR-DET-016
71 statements:
72 - id: AIR-PREV-002.1
73 title: Rigorous Data Cleansing and Anonymization at Ingestion
74 text: "Identify and remove or appropriately anonymize sensitive details to ensure that data fed into the AI system is free from information that could pose a security or privacy risk if inadvertently exposed."
75 recommendations:
76 - >
77 Pre-Processing Review and Cleansing: Before any information from
78 internal knowledge sources is ingested by an AI system, it must
79 undergo a thorough review and cleansing process to identify and
80 remove or appropriately anonymize sensitive details.
81 - >
82 Categories of Data to Target for Filtering: Personally
83 Identifiable Information (PII), Proprietary Business Information,
84 Sensitive Internal Operational Data, Confidential Customer Data,
85 and Regulatory or Compliance-Sensitive Information.
86 - >
87 Filtering and Anonymization Methods: Data Masking, Redaction,
88 Generalization, Tokenization, and Synthetic Data Generation.
89 - id: AIR-PREV-002.2
90 title: Segregation for Highly Sensitive Data
91 text: ""
92 recommendations:
93 - >
94 Isolated AI Systems for Critical Data: For datasets containing
95 exceptionally sensitive information that cannot be adequately
96 protected through standard cleansing, implement separate, isolated
97 AI systems with stricter access controls, enhanced encryption,
98 and limited network connectivity.
99 - >
100 Access Domain-Based Segregation: Segment data and AI system access
101 based on clearly defined access domains that mirror the
102 organization's existing data classification and access control
103 structures.
104 - id: AIR-PREV-002.3
105 title: Filtering AI System Outputs (Secondary Defense)
106 text: ""
107 recommendations:
108 - >
109 Response Filtering and Validation: Responses generated by the AI
110 system should be monitored and filtered before being presented to
111 users, acting as a safety net to detect sensitive data that might
112 have bypassed initial input cleansing.
113 - >
114 Contextual Output Analysis: Implement intelligent filtering that
115 considers the context of the user's query and their authorization
116 level to determine what information should be included in the
117 response.
118 - id: AIR-PREV-002.4
119 title: Integration with Source System Access Controls
120 text: ""
121 recommendations:
122 - "Respect Original Permissions: Design the AI system to respect and replicate the original access control permissions from source systems."
123 - "Dynamic Source Querying: For real-time RAG systems, consider querying source systems dynamically while respecting user permissions, rather than pre-processing all data indiscriminately."
124 - id: AIR-PREV-002.5
125 title: Monitoring and Continuous Improvement
126 text: >
127 Periodically audit the effectiveness of data filtering processes
128 by sampling processed data and checking for any sensitive
129 information that may have been missed.
130 recommendations:
131 - "Regular Review of Filtering Effectiveness: Periodically audit the effectiveness of data filtering processes by sampling processed data."
132 - "Feedback Loop Integration: Establish mechanisms for users and reviewers to report instances where sensitive information may have been inappropriately exposed."
133 - "Threat Intelligence Integration: Stay informed about new types of data leakage vectors and attack techniques that might affect AI systems."
134 principles:
135 - reference-id: AIR-PRIN
136 entries:
137 - reference-id: AIR-PRIN-001
138 remarks: Filtering and anonymization applied before data enters AI pipelines
139 - reference-id: AIR-PRIN-002
140 remarks: Filtering strategies respect source data sensitivity and access controls
141 - reference-id: AIR-PRIN-003
142 remarks: Only necessary data included in AI systems, de-identified where possible
143 - reference-id: AIR-PRIN-004
144 remarks: Multiple filtering layers at ingestion, processing, and output
145 - reference-id: AIR-PRIN-005
146 remarks: Audit trails document what filtering has been applied and why
147 vectors:
148 - reference-id: AIR-VEC
149 entries:
150 - reference-id: AIR-RC-001-01
151 - reference-id: AIR-RC-001-02
152 - reference-id: AIR-RC-001-03
153 - reference-id: AIR-RC-001-04
154 - reference-id: AIR-RC-001-05
155 - reference-id: AIR-SEC-009-01
156 - reference-id: AIR-SEC-009-02
157 - reference-id: AIR-SEC-009-03
158 - reference-id: AIR-SEC-009-04
159 state: Active
160
161 - id: AIR-PREV-003
162 group: PREV
163 title: User/App/Model Firewalling
164 objective: >
165 User/App/Model Firewalling encompasses the set of security controls
166 applied at the boundaries between users, applications, AI models,
167 and supporting data stores such as RAG databases. When internal
168 company information is used to enrich a RAG database, especially if
169 this involves processing by external services, this data and the
170 external communication pathways must be carefully managed and
171 secured. Any proprietary or sensitive information sent to an external
172 service for such processing requires rigorous filtering before
173 transmission to prevent data leakage.
174 rationale:
175 importance: >
176 Implementing comprehensive user/app/model firewalling provides
177 critical security benefits including attack prevention, data
178 protection, service availability, reputation protection, and
179 compliance support. Firewalling blocks prompt injection attacks
180 and malicious inputs before they reach AI models, prevents
181 sensitive information leakage through AI outputs or RAG processing,
182 and helps meet regulatory requirements for data handling and
183 system security.
184 goals:
185 - "RAG Data Ingestion: Filter sensitive or private data before transmitting internal information to external services for embedding creation"
186 - "User Input to AI Model: Detect and block malicious or abusive user inputs such as Prompt Injection attacks"
187 - "AI Model Output: Detect excessively long responses, format deviations, evasion patterns, data leakage, and inappropriate content"
188 see-also:
189 - AIR-PREV-017
190 - AIR-PREV-008
191 - AIR-DET-015
192 statements:
193 - id: AIR-PREV-003.1
194 title: RAG Data Ingestion Filtering
195 text: ""
196 recommendations:
197 - >
198 RAG Database Security: While it's often more practical to
199 pre-process and filter data for RAG systems before sending it
200 for external embedding creation, organizations might also
201 consider in-line filters for real-time checks. Once internal
202 information is converted into embeddings and stored in vector
203 databases, the data becomes largely opaque to traditional
204 security tools.
205 - >
206 Filtering Efficacy: Static filters based on regular expressions
207 or keyword blocklists are effective for well-defined patterns
208 but less effective at identifying nuanced issues such as
209 generic private information or subtle Prompt Injection attacks.
210 - >
211 Streaming Outputs: Streaming responses improve user experience
212 but implementing output filtering can be challenging. An
213 approach is to stream while performing on-the-fly detection,
214 cancelling output if an issue is found.
215 - id: AIR-PREV-003.2
216 title: Remediation Techniques
217 text: ""
218 recommendations:
219 - >
220 Basic Filters: Simple static checks using blocklists and
221 regular expressions can detect rudimentary attacks or policy
222 violations.
223 - >
224 System Prompts (Caution Advised): While system prompts can
225 instruct an LLM on what to avoid, they are generally not a
226 robust security control. Attackers can often bypass these
227 instructions.
228 - >
229 LLM as a Judge: A secondary, specialized LLM analyzes user
230 queries and the primary LLM's responses, categorizing
231 inputs/outputs for various risks such as prompt injection,
232 abuse, hate speech, and data leakage.
233 - >
234 Human Feedback Loop: Implementing a system where users can
235 easily report problematic AI responses provides a valuable
236 complementary control.
237 - id: AIR-PREV-003.3
238 title: Additional Considerations
239 text: ""
240 recommendations:
241 - >
242 API Security and Observability: Implementing a comprehensive
243 API monitoring and security solution offers benefits beyond
244 AI-specific threats. A security proxy can enforce encrypted
245 communication between all AI system components.
246 - >
247 Logging and Analysis: Detailed logging of interactions is
248 essential for understanding user behavior, system performance,
249 and detection of sophisticated attacks or anomalies.
250 - id: AIR-PREV-003.4
251 title: Challenges and Considerations
252 text: >
253 RAG Database Security: Vector databases make traditional security
254 filtering difficult once data is embedded. Filtering Efficacy:
255 Static filters may miss nuanced attacks or sophisticated content.
256 Streaming Outputs: Real-time filtering creates trade-offs between
257 security and user experience.
258 principles:
259 - reference-id: AIR-PRIN
260 entries:
261 - reference-id: AIR-PRIN-001
262 remarks: RAG data filtered before transmission to external services
263 - reference-id: AIR-PRIN-004
264 remarks: Layered filtering at user input, model output, and RAG ingestion boundaries
265 - reference-id: AIR-PRIN-005
266 remarks: Logging and analysis of all interactions for audit and anomaly detection
267 vectors:
268 - reference-id: AIR-VEC
269 entries:
270 - reference-id: AIR-OP-007-01
271 - reference-id: AIR-OP-007-02
272 - reference-id: AIR-OP-007-03
273 - reference-id: AIR-SEC-010-01
274 - reference-id: AIR-SEC-010-02
275 - reference-id: AIR-SEC-010-03
276 - reference-id: AIR-OP-018
277 - reference-id: AIR-OP-020
278 state: Active