Agentic AI Search with Custom Evaluation Framework for Church Management

pushpay · Tech · 2026 · source Agentic: yes · promotional bias: high · source-backed · extracted with prompt 0.5.2 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Church management search

Pushpay needed reliable natural-language search over community data after an initial agentic AI search feature reached only moderate accuracy.

Constraints:

Domain-specific query behavior
Need confidence before production
Natural language to structured application output

Architecture

Pushpay's AI search system combines a customer-facing church-management search agent with a closed-loop evaluation architecture. A single Bedrock-backed agent receives ministry-leader natural-language questions; a dynamic prompt constructor adds church context, sample queries, and relevant filter inventory selected through semantic search; Claude Sonnet 4.5 generates the JSON output the application uses to apply filters and show insights. User interactions feed an offline evaluation system with a curated 300-plus-query golden dataset, LLM-as-judge comparison, domain categories, Wilson intervals, latency dashboards, and domain-level rollout controls.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  ministry-user[Ministry user] --> ai-search-agent[AI search agent]
  ai-search-agent --> dynamic-prompt-constructor[Dynamic prompt constructor]
  dynamic-prompt-constructor --> semantic-filter-search[Semantic filter search]
  ai-search-agent --> bedrock-claude[Bedrock Claude]
  bedrock-claude --> application-filter-engine[Application filter engine]
  application-filter-engine --> evaluation-capture[Evaluation capture]
  evaluation-capture --> golden-dataset[Golden dataset]
  golden-dataset --> llm-judge-evaluator[LLM judge evaluator]
  llm-judge-evaluator --> evaluation-dashboard[Evaluation dashboard]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Ministry user`	Church administrator or ministry leader asking a plain-English data question.	`AI search agent`
`AI search agent`	Single agent that converts natural-language questions into structured application output.	`Ministry user`, `Dynamic prompt constructor`, `Bedrock Claude`
`Dynamic prompt constructor`	Builds user-, tenant-, and query-specific prompt context and relevant filter inventory.	`AI search agent`, `Semantic filter search`
`Semantic filter search`	Selects relevant filters from hundreds of configurable application filters.	`Dynamic prompt constructor`
`Bedrock Claude`	Claude Sonnet 4.5 processing path that generates JSON output.	`AI search agent`, `Application filter engine`
`Application filter engine`	Applies generated JSON filters to community data and displays insights.	`Bedrock Claude`, `Evaluation capture`
`Evaluation capture`	Instrumentation that records interactions for offline evaluation.	`Application filter engine`, `Golden dataset`
`Golden dataset`	Curated 300-plus query set with expected outputs.	`Evaluation capture`, `LLM judge evaluator`
`LLM judge evaluator`	Scores agent outputs and records metrics/logs/latency.	`Golden dataset`, `Evaluation dashboard`
`Evaluation dashboard`	Domain and latency dashboards for rollout and iteration decisions.	`LLM judge evaluator`

Data flow

A ministry user asks a natural-language question. The dynamic prompt constructor builds prompt context from church/user information and semantically relevant filters, Bedrock/Claude generates application JSON, the app applies filters and displays insights, interactions are captured offline, the evaluator compares outputs to the golden dataset with LLM-as-judge scoring, and dashboards guide prompt/domain rollout changes.

3 supporting quotes

“automatically constructs additional customized system prompts based on the user specific information, such as church context, sample queries, and application filter inventory.”article, stated

“Uses Claude Sonnet 4.5 to process prompts and generate JSON output required by the application to display the desired query results as insights to users.”article, stated

“* Evaluation system: The evaluation system implements a closed-loop improvement solution where user interactions are instrumented, captured and evaluated offline.”article, stated

Why this architecture

Pushpay added the evaluation system because the first prompt-only agent hit a 60-70% accuracy plateau and manual evaluation was too slow. Domain-level metrics and dynamic prompt construction let the team suppress weak categories, improve filter relevance, and move from aggregate intuition to data-driven production readiness.

3 supporting quotes

“While it demonstrated a 60-70% success rate with basic business queries, the team reached an accuracy plateau.”article, stated

“The evaluation of the agent was a manual and tedious process Tuning the system prompt beyond this accuracy threshold proved challenging given the diverse spectrum of user queries and the application’s coverage of over 100 distinct configurable filters.”article, stated

“By using categories, the team can pinpoint the AI agent’s weaknesses by domain. In the following example , the “activity” domain shows significantly lower accuracy than other categories.”article, stated

Models used: Claude Sonnet 4.5 · LLM-as-judge

Scale

The golden dataset contains over 300 representative queries.
“The golden dataset: A curated golden dataset containing over 300 representative queries, each paired with its corresponding expected output, forms the foundation of automated evaluation.”article, stated
Strategic domain rollout achieved 95% overall accuracy.
“By temporarily suppressing underperforming categories—such as activity queries—while undergoing optimization, the system achieved 95% overall accuracy. By using this approach, users experienced only the highest-performing features while the team refined others to production standards.”article, stated
Time to insight dropped from about 120 seconds to under 4 seconds.
“reduced time-to-insight from approximately 120 seconds (experienced users manually navigating complex UX) to under 4 seconds”article, stated

Alternatives considered (roads not taken)

Continue tuning a static system prompt

Rejected because: The first iteration plateaued and covered over 100 configurable filters.

“over 100 distinct configurable filters. These presented critical blockers for the team’s path to production.”article, implied

Use one aggregate accuracy score

Rejected because: Domain-level categorization exposed weak categories hidden by aggregate metrics.

“3/ Think beyond aggregate metrics Aggregate accuracy scores can sometimes mask critical performance variations. By evaluating agent performance at the domain category level, Pushpay uncovered weaknesses beyond what a single accuracy metric can capture.”article, stated

Patterns

`eval-driven-development` (quality)

Pushpay uses offline evaluation results and dashboards to iterate the search agent.

“* Evaluation system: The evaluation system implements a closed-loop improvement solution where user interactions are instrumented, captured and evaluated offline.”article, stated

`llm-as-judge` (evaluation)

The evaluation framework includes an LLM-as-judge over a golden dataset.

“To address these limitations, Pushpay built a comprehensive generative AI evaluation framework on Amazon Bedrock, incorporating a curated golden dataset of over 300 queries, an LLM-as-judge evaluator, domain-based categorization, and performance dashboards.”summary, stated

`prompt-chaining` (prompting)

Dynamic prompt construction customizes system prompts from user and church context.

“* Dynamic prompt constructor (DPC): automatically constructs additional customized system prompts based on the user specific information, such as church context, sample queries, and application filter inventory.”article, stated

Tradeoffs

How to ship production search safely.

Chose domain-level rollout informed by eval dashboards over enabling all query categories at once.

Rationale: Pushpay could expose high-performing domains while lower-performing ones stayed under optimization.

“temporarily suppressing underperforming categories”article, stated

Gotchas (what bit them)

The first solution lacked systematic evaluation.

Root cause: Accuracy was not high enough to confidently scale all domains.

Resolution: Pushpay built a golden dataset and evaluation dashboard before broader rollout.

“## The challenges of initial solution without evaluation”article, stated

Evaluation

The framework uses a golden dataset, domain categories, LLM judge, dashboards, and domain-level rollout.

Metrics: accuracy · domain performance · time-to-insight

Outcomes

High-priority domains reached 95% accuracy and time-to-insight under 4 seconds, according to the summary.

The outcomes are vendor-reported through an AWS customer story.

“The solution ultimately achieved 95% accuracy in high-priority domains, reduced time-to-insight from 120 seconds to under 4 seconds, and provided the confidence needed for production deployment.”summary, stated