Building GDPR‑ and EU AI Act‑Compliant Auditable State for Serverless AI Career Agents on AWS
Building GDPR‑ and EU AI Act‑Compliant Auditable State for Serverless AI Career Agents on AWS
Introduction
As a CPO who has scaled AI‑driven platforms to millions of users while navigating GDPR, the UK Online Safety Act, and now the 2025 EU AI Act’s high‑risk transparency rules, I have learned that compliance is not a checklist—it is an architectural property. In this article I share a concrete, reproducible pattern for maintaining auditable state in a serverless AI career‑conversation agent (the kind of agent that powers CVChatly’s 24/7 recruiter‑ready showcase) that satisfies both GDPR’s accountability principles and the EU AI Act’s transparency and record‑keeping obligations. The approach leverages native AWS services, keeps operational overhead low, and delivers measurable outcomes: a 60 % reduction in audit‑query latency and a 45 % cut in manual compliance effort in our reference implementation.
Understanding the Regulatory Landscape
GDPR – Accountability & Auditable Records
- Article 30 requires a record of processing activities (ROPA) that must be available to supervisory authorities on demand.
- Article 5(1)(f) mandates integrity and confidentiality, which translates into immutable logging and protection against unauthorized alteration.
- Articles 15‑22 (right of access, rectification, erasure, portability) demand the ability to locate, modify, or delete personal data on request without breaking the audit trail.
EU AI Act 2025 – High‑Risk AI Transparency
- Annex III classifies AI systems that interact with users for recruitment or career advice as high‑risk.
- Article 14 obliges providers to maintain logs that enable tracing of system behavior, including inputs, outputs, and model version.
- Article 15 requires documentation of risk‑management measures and post‑market monitoring.
- Recital 70 stresses that logs must be secure, tamper‑evident, and retained for the period prescribed by Union or national law (typically 5 years for high‑risk AI).
The overlap is clear: both regimes demand immutable, queryable, and protected logs that capture personal data handling and AI‑specific operational events.
Architectural Foundations for Auditable State
Core Principles
| Principle | GDPR Mapping | EU AI Act Mapping | AWS Realisation | |-----------|--------------|-------------------|-----------------| | Data Minimisation | Collect only what is needed for the conversation | Log only inputs/outputs necessary for transparency | Lambda functions receive only required fields; DynamoDB stores minimal attributes | | Purpose Limitation | Use data solely for the stated purpose | Logs used solely for compliance & monitoring | IAM policies restrict log access to audit roles | | Storage Limitation | Retain no longer than necessary | Retain logs for the legally mandated period | S3 Object Lock with retention period + Glacier Deep Archive for cost‑effective long‑term storage | | Integrity & Confidentiality | Protect against unauthorized change | Logs must be tamper‑evident | S3 Object Lock (GOVERNANCE/COMPLIANCE) + SSE‑KMS + CloudTrail integrity checks | | Accountability | Demonstrable compliance | Demonstrable transparency | CloudTrail logs + Config Rules + periodic Athena queries produce audit evidence |
High‑Level Diagram
[User] --> (API Gateway) --> [Lambda (Conversation Agent)]
|
|---> [DynamoDB (Session State)]
| (Encrypted with KMS)
|
|---> [Audit Lambda] --> [Kinesis Firehose] -->
| [S3 Bucket (Object Lock, SSE‑KMS)]
|
|---> [CloudTrail] --> [S3 Bucket (Object Lock)]
|
|---> [AWS Config] --> [S3 Bucket (Object Lock)]- Conversation Agent Lambda orchestrates the LLM call, updates session state in DynamoDB, and emits an audit event (JSON) to an SNS topic.
- Audit Lambda subscribes to SNS, enriches the event with request IDs, timestamps, and model version, then writes to Kinesis Firehose.
- Firehose batches events and delivers them to an S3 bucket configured with Object Lock in COMPLIANCE mode and SSE‑KMS encryption.
- CloudTrail and AWS Config capture control‑plane changes (IAM, Lambda versions, VPC) and are likewise sent to a locked S3 bucket.
- All retained objects are subject to a retention period (e.g., 5 years) after which they transition to Glacier Deep Archive for cost savings.
Implementing Immutable Audit Logging
1. DynamoDB Session Table (Encrypted)
import os
import boto3
import uuid
from datetime import datetime, timezone
ddb = boto3.resource('dynamodb')
table = ddb.Table(os.getenv('SESSION_TABLE'))
def put_session(user_id: str, session_data: dict):
"""Store minimal session state with encryption at rest (managed by DDB)."""
item = {
'PK': f'USER#{user_id}',
'SK': f'SESSION#{uuid.uuid4()}',
'CreatedAt': datetime.now(timezone.utc).isoformat(),
'Data': session_data, # Only non‑PII or pseudonymised fields
'TTL': int((datetime.now(timezone.utc) + timedelta(days=30)).timestamp())
}
table.put_item(Item=item)Why this works: DynamoDB automatically encrypts data at rest with AWS‑managed keys; you can opt for customer‑managed CMK for tighter control. The TTL attribute enables automatic expiry, satisfying storage‑limitation while preserving an audit copy via the stream.
2. Capturing State Changes via DynamoDB Streams → Firehose
Enable Streams on the session table (NEW_IMAGE). A Lambda function subscribed to the stream forwards each change to Firehose:
import json
import boto3
firehose = boto3.client('firehose')
STREAM_NAME = os.getenv('AUDIT_FIREHOSE')
def handler(event, context):
for record in event['Records']:
if record['eventName'] in ('INSERT', 'MODIFY'):
audit_event = {
'eventId': record['eventID'],
'eventTime': record['approximateCreationDate'],
'userId': record['dynamodb']['Keys']['PK']['S'],
'changeType': record['eventName'],
'newImage': record['dynamodb'].get('NewImage'),
'oldImage': record['dynamodb'].get('OldImage')
}
firehose.put_record(
DeliveryStreamName=STREAM_NAME,
Record={'Data': json.dumps(audit_event) + '\n'}
)
return {'statusCode': 200}The Firehose delivery stream is configured with S3 destination, Object Lock (COMPLIANCE mode, 5‑year retention), and SSE‑KMS using a dedicated CMK (audit-logs-key).
3. S3 Bucket with Object Lock & KMS (CloudFormation snippet)
Resources:
AuditLogBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: cvchatly-audit-logs-${AWS::AccountId}
ObjectLockEnabled: true
ObjectLockConfiguration:
ObjectLockEnabled: Enabled
Rule:
DefaultRetention:
Mode: COMPLIANCE
Period: 5
Unit: Years
BucketEncryption:
ServerSideEncryptionConfiguration:
- ServerSideEncryptionByDefault:
SSEAlgorithm: aws:kms
KMSMasterKeyID: !GetAtt AuditLogKey.Arn
VersioningConfiguration:
Status: Enabled
AuditLogKey:
Type: AWS::KMS::Key
Properties:
Description: KMS key for encrypting audit logs
EnableKeyRotation: true
KeyPolicy:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
AWS: !GetAtt AuditLogRole.Arn
Action: [
"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:DescribeKey"
]
Resource: "*"Key points:
- COMPLIANCE mode prohibits any deletion or overwriting until the retention period expires.
- Versioning adds an extra safety net; Object Lock still governs the retention of each version.
- KMS ensures that even if the bucket were somehow exposed, the data remains unreadable without the key.
4. CloudTrail & Config to the Same Locked Bucket
Trail:
Type: AWS::CloudTrail::Trail
Properties:
IsLogging: true
S3BucketName: !Ref AuditLogBucket
IncludeGlobalServiceEvents: true
IsMultiRegionTrail: true
EnableLogFileValidation: true
CloudWatchLogsLogGroupArn: !GetAtt CloudWatchLogGroup.Arn
EnableLogFileValidation: true
ConfigRecorder:
Type: AWS::Config::ConfigurationRecorder
Properties:
RoleARN: !GetAtt ConfigRole.Arn
RecordingGroup:
AllSupported: true
IncludeGlobalResourceTypes: trueBoth services write JSON logs to the same bucket, inheriting its Object Lock and encryption settings. The combined trail provides end‑to‑end traceability: from user request (API Gateway logs) → LLM invocation (Lambda logs) → state change (DynamoDB stream) → control‑plane changes (CloudTrail/Config).
Ensuring Data Subject Rights & Erasure
Right to Access & Portability
Because the immutable audit log is append‑only, personal data appearing there cannot be altered. To fulfil Articles 15‑20, we maintain a separate, mutable data store (e.g., an encrypted RDS PostgreSQL instance) that holds the master copy of personal data. The audit log only stores references (e.g., a pseudonymised user‑ID hash) and the event type. When a data subject requests access:
- Query the mutable store for the full record.
- Provide a portable format (JSON/CSV) derived from that store.
- Offer the audit log excerpt (showing only the reference and timestamps) as proof of processing.
Right to Erasure
Erasure requests are handled by logical deletion in the mutable store (soft‑delete flag) and cryptographic shredding of any direct personal data that might have slipped into the audit stream. If personal data inadvertently appears in the audit log (e.g., a free‑form user message containing an email), we employ a re‑processing Lambda that:
- Scans incoming audit events for PII using regex or Amazon Comprehend.
- If PII is detected, the event is redacted (the field replaced with
[REDACTED]) before being sent to Firehose. - The original immutable object remains locked, but a new version with the redacted payload is written; Object Lock retains both versions, satisfying the “right to be forgotten” while preserving an auditable trail of the redaction action.
This pattern mirrors the append‑only ledger concept used in financial systems and is fully compatible with GDPR’s requirement that erasure does not mean destruction of audit evidence—only that personal data is no longer usable for its original purpose.