Automated Intelligent Document Processing System Achieves 90% Efficiency for Irish FOI Requests

A pioneering fully automated Intelligent Document Processing (IDP) system, built on Amazon Web Services (AWS), has dramatically streamlined the handling of Freedom of Information (FOI) requests from Irish citizens concerning a large insurance claims database. The system, developed by an internal engineering team, boasts an approximate 90% efficiency improvement over the manual process it replaced, marking a significant advancement in data access and compliance for the financial sector.

Background: The Imperative for Efficient Data Access and Privacy

In an era defined by stringent data privacy regulations like GDPR and increasing public demand for transparency, organizations managing vast datasets face considerable challenges in fulfilling information requests promptly and accurately. For an employer maintaining a substantial insurance claims database in Ireland, this challenge manifested in the laborious, manual processing of FOI-type requests. Irish citizens seeking to ascertain whether their personal details, such as name and address, were recorded within this database previously faced a slow and resource-intensive process. Each request necessitated manual intervention to review submitted documentation, extract personal identifiable information (PII), and cross-reference it against the vast claims repository. This manual workflow was not only time-consuming and costly but also prone to human error, creating bottlenecks and potential compliance risks. The need for a robust, scalable, and secure automated solution became paramount to enhance operational efficiency, reduce costs, and ensure adherence to data protection standards while improving citizen experience.

The Genesis of Automation: A Cloud-Native IDP Solution

Recognizing these inefficiencies, a lead engineer embarked on developing a sophisticated IDP system leveraging the power of AWS cloud services. The primary goal was to create an end-to-end automated pipeline capable of ingesting diverse document types, extracting crucial PII, and facilitating rapid database lookups, all while maintaining a high degree of accuracy and security. The system was meticulously designed to handle the nuances of FOI requests, where precise identification of individuals is critical.

The high-level workflow was conceived as a series of interconnected, automated steps, culminating in a streamlined process that drastically reduced manual effort. This involved:

• Ingestion of incoming email requests containing identity documents.
• Automated extraction of attachments and secure storage.
• Optical Character Recognition (OCR) to convert document images into machine-readable text.
• Intelligent classification of document types (e.g., passport, driving licence, bank statement).
• Extraction of specific PII fields, such as first name, last name, date of birth, and address.
• A crucial Human-In-The-Loop (HIL) process for final verification of extracted PII, ensuring accuracy and mitigating risks associated with fully automated decisions.
• Recording of success or failure outcomes, ready for integration with backend systems for database lookups.

This architecture fundamentally shifted the paradigm from a manual, reactive approach to a proactive, automated one, promising substantial improvements in processing speed and resource allocation.

AWS Architecture: Powering the Automated Pipeline

The backbone of this advanced IDP system is a carefully selected suite of AWS services, each playing a vital role in the automated workflow:

• Amazon S3 (Simple Storage Service): Utilized as the primary data lake, S3 provides highly scalable, durable, and secure object storage for all stages of document processing. It hosts raw incoming image attachments, intermediate Textract outputs, and final PII extraction results, organized into dedicated folders such as raw/, textract/, bedrock/, results/, and failed/. This compartmentalization ensures data integrity and facilitates auditing.
• AWS Secrets Manager: Critical for security, Secrets Manager securely stores sensitive credentials, including the OAuth tokens required for accessing email accounts (e.g., Gmail or corporate Outlook). This prevents hardcoding credentials in application code, enhancing the system’s security posture and compliance with best practices.
• AWS Lambda: The serverless compute service forms the operational core, executing specific tasks without provisioning or managing servers. Three key Lambda functions drive the IDP pipeline:
  • ingest_email_image: Responsible for polling designated email accounts, validating attachments (JPEG/PNG image files), and storing them in the raw/ S3 bucket.
  • extract_text: Invokes Amazon Textract to perform OCR on the images in raw/, converting them into structured text data, which is then stored in textract/ as JSON.
  • classify_and_extract_pii: Processes the Textract output using Amazon Bedrock with Claude Sonnet 4.6 to classify document types and extract targeted PII, storing the final results in results/.
• Amazon Textract: This machine learning service for intelligent document processing automatically extracts text, handwriting, and data from scanned documents. Its integration is crucial for accurately converting various image-based identity documents into usable text, even those with complex layouts or semi-structured data.
• Amazon Bedrock (Claude Sonnet 4.6): A foundational model service, Bedrock provides access to high-performing Large Language Models (LLMs) like Anthropic’s Claude Sonnet 4.6. This powerful generative AI model is employed for sophisticated document classification and precise PII extraction, leveraging its advanced natural language understanding capabilities. The choice of Claude Sonnet 4.6 was driven by its balance of performance, cost-effectiveness, and suitability for complex information extraction tasks, particularly when guided by well-crafted prompts.
• AWS Step Functions: As the orchestration layer, Step Functions define and manage the serverless workflow. Using Amazon States Language (ASL), it visually represents the sequence of Lambda invocations, handles state transitions, implements retries for transient failures, and manages error handling. This ensures the robustness and reliability of the entire pipeline, making it resilient to individual component failures.
• Amazon EventBridge Scheduler: This service acts as the trigger mechanism for the entire IDP workflow. Configured with a cron-like schedule, it automatically initiates the Step Functions workflow at predefined intervals (e.g., daily at 11:55 PM), ensuring continuous and timely processing of incoming FOI requests.

A Step-by-Step Walkthrough of the Automated Workflow

The automated IDP system follows a precise, event-driven chronology:

1. Scheduled Activation: The process begins with EventBridge Scheduler, configured to run daily at a specific time (e.g., 11:55 PM on weekdays). This schedule triggers the AWS Step Functions workflow.
2. Email Ingestion and Attachment Storage (Lambda 1: ingest_email_image):
   • The ingest_email_image Lambda function is invoked by the Step Function.
   • It securely retrieves OAuth credentials from AWS Secrets Manager to access the designated email account (e.g., a corporate Outlook account or a labeled Gmail inbox).
   • The Lambda polls for new emails matching a specific label (e.g., "DocumentProcessing").
   • For each eligible email, it validates that it contains exactly one image attachment (JPEG or PNG) within specified size limits.
   • Valid image attachments are then uploaded to the raw/ folder within the designated Amazon S3 bucket. Metadata, such as the original email message ID and attachment ID, is preserved for traceability.
   • Invalid emails or those with multiple/unsupported attachments are logged and reported separately, preventing unprocessed requests from being lost.
3. Optical Character Recognition (Lambda 2: extract_text):
   • Upon successful storage of an image in raw/, the Step Function triggers the extract_text Lambda.
   • This function retrieves the newly uploaded image from S3.
   • It then invokes Amazon Textract’s synchronous document text detection API, which performs high-accuracy OCR on the image.
   • The extracted text, including individual lines, associated confidence scores, and document metadata, is formatted into a JSON document.
   • This JSON output is securely stored in the textract/ folder of the S3 bucket, creating a machine-readable representation of the original image content.
4. Document Classification and PII Extraction (Lambda 3: classify_and_extract_pii):
   • Following Textract processing, the Step Function invokes the classify_and_extract_pii Lambda.
   • This Lambda retrieves the Textract-generated JSON text from the textract/ S3 folder.
   • It then constructs a prompt, incorporating the extracted text, and sends it to Amazon Bedrock, specifying Anthropic’s Claude Sonnet 4.6 model.
   • The Bedrock model, guided by the prompt, performs two critical tasks:
     • Document Classification: It identifies the type of document (e.g., "bank_statement," "passport," "driving_licence").
     • PII Extraction: It accurately extracts specific PII fields: "first_name," "last_name," "date_of_birth," and "address."
   • The classified document type and extracted PII are then compiled into a structured JSON result.
   • This final result is stored in the results/ folder within the S3 bucket, indicating the successful processing and data extraction.
5. Human-In-The-Loop (HIL) Verification:
   • Although the core extraction is automated, a critical HIL process is integrated downstream. This allows human operators to perform a final cross-check of the PII extracted from different documents, ensuring maximum accuracy before the data is used for database lookups or other actions. This step adds a layer of assurance, particularly vital for sensitive FOI requests.
6. Outcome Recording: The Step Function records the overall success or failure of the entire workflow for each processed document, providing a comprehensive audit trail.

Demonstrative Results and Efficiency Gains

The system’s efficacy was demonstrated using synthetic but realistic identity documents—a passport, a driving licence, and a bank statement—created via an AI image generator (Codex). The processing of these documents yielded highly accurate classification and PII extraction:

• Bank Statement: Classified as "bank_statement," with PII for "John Smith" and address "11 The High Street, NewTown, NT1 3WE." Date of birth was correctly identified as null, as it’s not typically found on bank statements.
• Passport: Classified as "passport," with PII for "AVERY EXAMPLE," date of birth "1991-08-30," and address "14 Example Lane, Testford, TE1 2ST."
• Driver’s Licence: Classified as "driving_licence," extracting identical PII as the passport, demonstrating consistency across document types.

The most compelling outcome of this implementation is the staggering 90% efficiency gain compared to the previous manual system. This translates directly into substantial operational cost savings, significantly faster response times for FOI requests, and a reduction in the potential for human error. For the insurance company, this means reallocating human resources to more complex tasks, improving compliance with regulatory mandates, and enhancing its reputation for transparency and responsiveness. For Irish citizens, it promises a much quicker and more reliable process for exercising their right to information.

Enhancing for Production Readiness

While the core system demonstrates robust automation, transitioning to a full production environment necessitates additional considerations to ensure resilience, security, and scalability:

• Integration with Corporate Email Systems: Replacing personal Gmail with a corporate Outlook email account via Microsoft Graph API for robust enterprise integration.
• Comprehensive Error Handling and Alerts: Implementing sophisticated error handling, including dead-letter queues (DLQs), automated retries, and proactive alerts (e.g., via Amazon SNS or CloudWatch Alarms) for failures at any stage.
• Advanced Attachment Handling: Expanding the system to handle multiple attachments per email, various document formats (e.g., PDFs), and potentially encrypted documents.
• Duplicate Request Management: Developing logic to identify and manage duplicate FOI requests to prevent redundant processing.
• Scalable Database Integration: Implementing a robust mechanism to connect the extracted PII with the large insurance claims database for automated lookups, potentially using services like Amazon RDS or Aurora.
• Enhanced Security Measures: Strengthening IAM policies, implementing VPC endpoints, and leveraging AWS Security Hub for continuous security monitoring.
• Observability and Monitoring: Implementing comprehensive logging, metrics (via CloudWatch), and tracing (via AWS X-Ray) to monitor system performance, troubleshoot issues, and gain insights into operational health.
• Cost Optimization: Continuously monitoring and optimizing AWS resource usage to manage operational costs effectively, especially for services like Textract and Bedrock which have usage-based pricing.
• Version Control and CI/CD: Implementing a robust Continuous Integration/Continuous Deployment (CI/CD) pipeline for managing code deployments, ensuring consistency and reliability of updates.
• Data Retention Policies: Defining and enforcing clear data retention policies for processed documents and extracted PII to comply with legal and regulatory requirements.

Broader Implications and the Future of IDP

This automated IDP system represents more than just an operational improvement; it signifies a strategic embrace of cloud-native AI/ML capabilities to solve complex business challenges. The combination of OCR (Textract) with generative AI (Bedrock’s Claude Sonnet) demonstrates a powerful pattern for intelligent document processing that can be replicated across various industries. From legal firms processing contracts to healthcare providers managing patient records, the ability to automatically ingest, classify, and extract critical information from unstructured and semi-structured documents offers unprecedented opportunities for efficiency, accuracy, and compliance.

The Human-In-The-Loop component remains a critical safeguard, highlighting the ongoing synergy between artificial intelligence and human oversight, particularly in sensitive domains like data privacy. As AI models continue to evolve, systems like this will become increasingly sophisticated, capable of handling even greater complexities and providing deeper insights from enterprise data. This project underscores the transformative potential of AWS cloud services in building resilient, scalable, and intelligent solutions that redefine operational benchmarks and enhance service delivery for citizens.

For all the code and ancillary files, a GitHub repository is available at: https://github.com/taupirho/document-intake-aws-pipeline