Intelligent Document Processing for Education Quality Assessment Reports

bqa · Education · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Education quality-assurance report review

BQA needed to review self-evaluation reports and supporting evidence from education and training institutions more efficiently. Manual review was slow, resource-intensive, and prone to incomplete submissions, missing evidence, and follow-up loops. The proof of concept tested whether AWS document processing and LLMs could extract, summarize, and assess evidence against BQA standards while preserving structured reviewer-facing output.

Constraints:

• BQA reviews schools, universities, and vocational institutes in Bahrain
• institutions submit SERs, supporting evidence, and media formats
• submissions can be inaccurate, incomplete, or unsupported by evidence
• manual review and follow-up consume significant time and resources
• the solution was a proof of concept, not a confirmed production deployment
• LLM output had to follow BQA rubric criteria, scoring rules, and concise formatting
• the prompt prioritized deterministic behavior with temperature 0 and topP 0.1

Architecture

BQA and AWS built a serverless IDP proof of concept using S3, SQS, Lambda, Textract, SageMaker JumpStart, Meta Llama, DynamoDB, Amazon Bedrock, and Amazon Titan Text Express. Documents are uploaded to S3, SQS decouples processing stages, Lambda functions invoke Textract for text extraction, extracted text is queued for summarization, a SageMaker JumpStart Meta Llama model summarizes content and compares extracted text to BQA standards, and DynamoDB stores summaries and assessment results. On request, a Bedrock Lambda invokes Titan Text Express with a carefully engineered rubric prompt to generate evaluation summaries and compliance comments.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  S3_document_bucket["S3 document bucket"] --> S3_event_notification["S3 event notification"]
  S3_event_notification --> Initial_SQS_queue["Initial SQS queue"]
  Initial_SQS_queue --> Textract_extraction_Lambda["Textract extraction Lambda"]
  Textract_extraction_Lambda --> Amazon_Textract["Amazon Textract"]
  Textract_extraction_Lambda --> Extracted_text_SQS_queue["Extracted text SQS queue"]
  Extracted_text_SQS_queue --> Summarization_Lambda["Summarization Lambda"]
  Extracted_text_SQS_queue --> Assessment_Lambda["Assessment Lambda"]
  Summarization_Lambda --> SageMaker_JumpStart_Llama["SageMaker JumpStart Llama"]
  Assessment_Lambda --> SageMaker_JumpStart_Llama
  SageMaker_JumpStart_Llama --> DynamoDB_results_table["DynamoDB results table"]
  Assessment_Lambda --> DynamoDB_results_table
  DynamoDB_results_table --> InvokeBedrock_Lambda["InvokeBedrock Lambda"]
  InvokeBedrock_Lambda --> Prompt_template["Prompt template"]
  Prompt_template --> Amazon_Titan_Text_Express["Amazon Titan Text Express"]
  Amazon_Titan_Text_Express --> Reviewer_facing_output["Reviewer-facing output"]

Components

Data flow

Relevant documents are stored in S3. An S3 event triggers an SQS queue, and a Lambda function consumes the queue to run Textract over each file. Extracted text is placed onto another SQS queue. A summarization Lambda sends the text to SageMaker JumpStart, where Meta Llama summarizes it. In parallel, an assessment Lambda compares extracted text against BQA standards for compliance and quality. DynamoDB stores summarized data and assessment results. When reviewers request generative feedback, another Lambda calls Amazon Bedrock with Titan Text Express and a structured prompt containing indicator names, submitted evidence, rubric comments, scoring instructions, and output constraints.

Why this architecture

BQA's process needs a decoupled document pipeline because submissions can be long, multimodal, and variable in processing cost. S3 and SQS buffer ingestion; Lambda and Textract extract text; SageMaker JumpStart handles summarization and standards comparisons; DynamoDB stores intermediate results; and Bedrock/Titan generates reviewer-facing comments from a strict rubric prompt. The prompt replaces model fine-tuning by embedding evidence, rubric criteria, edge-case handling, scoring, and formatting into each evaluation request.

Models used: Amazon Titan Text Express · Meta Llama text generation model · Amazon Textract OCR · foundation models through Amazon Bedrock

Scale

• BQA reviews all education and training institutions, including schools, universities, and vocational institutes.
  “BQA reviews the performance of all education and training institutions, including schools, universities, and vocational institutes”article, stated
• The output prompt limits responses to 100 words.
  “Specify the response as bullet points, focusing on relevant analysis and evidence, with a word limit of 100 words”article, stated
• The model configuration uses maxTokenCount 4096, temperature 0, and topP 0.1.
  “maxTokenCount: 4096, stopSequences: [], temperature: 0, topP: 0.1,”article, stated
• Anticipated outcomes include 70% accurate standards-compliant reports.
  “Faster turnaround times for generating 70% accurate and standards-compliant self-evaluation reports”article, stated
• Anticipated evidence-analysis time reduction is 30%.
  “reducing evidence analysis time by 30%.”article, stated
• Anticipated operational cost reduction is 30%.
  “achieving 30% reduced operational costs through process optimizations.”article, stated

Alternatives considered (roads not taken)

Manual review and follow-up

Rejected because: Manual review was time-consuming, resource-intensive, and slowed the review process when submissions were incomplete or unsupported.

Directly couple upload processing stages

Rejected because: SQS buffering improves scalability and fault tolerance by decoupling components.

Fine-tune a BQA-specific standards model

Rejected because: The PoC embeds rubric standards directly in the prompt so it can evaluate against criteria without fine-tuning.

Use one model for all generation tasks

Rejected because: The architecture uses Meta Llama on SageMaker JumpStart for summarization and Titan Text Express on Bedrock for evaluation comments.

Patterns

async-message-orchestration (workflow)

S3 events and SQS queues decouple ingestion, extraction, summarization, and assessment stages.

tool-use-orchestration (workflow)

Lambda functions orchestrate Textract, SageMaker JumpStart, DynamoDB, and Bedrock calls across the document-processing pipeline.

structured-output-contract (interface)

The prompt constrains output to bullet points, three compliance score classes, direct evidence, and a 100-word limit.

model-routing (modeling)

The PoC uses Meta Llama through SageMaker JumpStart for summarization and Titan Text Express through Bedrock for evaluation responses.

eval-driven-development (quality)

The prompt encodes rubric criteria, scoring instructions, direct evidence citation, and deterministic model parameters for consistent reviewer-facing evaluation.

human-in-the-loop-gate (workflow)

Generated summaries and compliance comments support BQA reviewers rather than replacing the authority's review function.

Tradeoffs

Use prompt-embedded standards instead of fine-tuning.

Chose rubric criteria and evidence supplied dynamically in the prompt over fine-tuning a model on BQA-specific standards.

Rationale: The PoC can evaluate against standards without maintaining a fine-tuned domain model.

Use event-driven queues between stages.

Chose S3 notifications, SQS queues, and Lambda consumers over synchronous end-to-end document processing.

Rationale: Decoupling improves scalability and fault tolerance for variable-duration document workloads.

Split summarization and evaluation across model paths.

Chose Meta Llama via SageMaker JumpStart for summarization and Titan Text Express via Bedrock for compliance comments over one model endpoint for every step.

Rationale: The architecture assigns separate model components to summarization, comparison, and reviewer-facing generation.

Constrain generation parameters for consistency.

Chose temperature 0 and topP 0.1 over creative or diverse sampling.

Rationale: Compliance assessment benefits from deterministic, focused outputs rather than varied prose.

Gotchas (what bit them)

Submitted evidence often did not support institutional claims.

Root cause: Institutions provided incomplete, inaccurate, missing, or insufficient supporting material.

Resolution: The pipeline extracts evidence and compares it against standards so reviewers can detect gaps and request clarification.

Manual review created workflow bottlenecks.

Root cause: BQA staff had to manually inspect submissions and follow up with institutions for missing or inadequate material.

Resolution: Automated extraction, summarization, and compliance feedback are intended to reduce review time and follow-up friction.

Direct component coupling would reduce reliability under variable document workloads.

Root cause: Document processing can be slow and uneven, so upload, extraction, and generation stages need buffering.

Resolution: The architecture uses SQS between stages for reliable decoupling, scalability, and fault tolerance.

LLM output needed strict scoring and formatting controls.

Root cause: Free-form feedback would be hard for reviewers to consume consistently across submissions.

Resolution: The prompt enforces score categories, direct evidence citation, bullet points, and a 100-word limit.

The case should not be overread as deployed production impact.

Root cause: The article explicitly describes the work as a proof of concept with anticipated success metrics.

Resolution: Treat the reported 70%/30% metrics as expected benefits from the PoC, not independently verified production outcomes.

Evaluation

The project is evaluated as a proof of concept with anticipated success metrics. The prompt-level evaluation contract includes rubric criteria, score classes, direct evidence citation, 1-5 scoring per comment, N/A handling for irrelevant evidence, 100-word output limits, and deterministic generation settings. Reported benefits are expected improvements, not confirmed production measurements.

Metrics: 70% accurate and standards-compliant self-evaluation reports · 30% evidence analysis time reduction · 30% operational cost reduction · score from 1-5 for each comment · Non-compliant / Compliant with recommendation / Compliant labels · 100-word response limit · temperature 0 · topP 0.1

Outcomes

• BQA built a proof of concept for generative-AI-assisted education quality reporting.
• The system extracts text from submitted documents, summarizes lengthy submissions, and compares evidence against standards.
• Anticipated benefits include 70% accurate standards-compliant reports, 30% reduced evidence-analysis time, and 30% reduced operational costs.
• Reviewer-facing outputs include concise bullet points, compliance scores, analysis, recommendations, and evidence.
• The full code for the implementation is available in a public GitHub repository linked by the article.

The article describes BQA's national education quality-assurance remit and a PoC architecture. Its numeric benefits are explicitly anticipated metrics rather than measured production outcomes.