Building and Evaluating a Financial Earnings Call Summarization System

aiera · Finance · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Financial earnings-call summarization

Aiera needed a reliable way to build and evaluate LLM-generated insights from earnings-call transcripts across a large financial-events corpus, where traditional summarization metrics poorly capture meaning.

Constraints:

Financial summarization quality is subjective and domain-specific.
Public datasets did not match Aiera's event-transcription use case.
Abstractive summaries may use different wording from references while preserving meaning.
The platform covers tens of thousands of events per year.

Architecture

Aiera built a domain-specific earnings-call summarization dataset from its own transcriptions, generated reference insights with Claude 3 Opus, compared generated summaries across providers, and evaluated them with ROUGE and BERTScore, ultimately favoring semantic similarity metrics for abstractive summarization.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  earnings-transcript-corpus[Earnings Transcript Corpus] --> dataset-builder[Dataset Builder]
  dataset-builder --> reference-insight-generator[Reference Insight Generator]
  dataset-builder --> candidate-summary-models[Candidate Summary Models]
  reference-insight-generator --> scoring-pipeline[Scoring Pipeline]
  candidate-summary-models --> scoring-pipeline[Scoring Pipeline]
  rouge-scorer[ROUGE Scorer] --> scoring-pipeline[Scoring Pipeline]
  bertscore-scorer[BERTScore Scorer] --> scoring-pipeline[Scoring Pipeline]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Earnings Transcript Corpus`	Domain text source for financial-event summarization.	`Dataset Builder`
`Dataset Builder`	Creates the summarization dataset from Aiera transcripts.	`Reference Insight Generator`, `Candidate Summary Models`
`Reference Insight Generator`	Uses Claude 3 Opus to create reference insights.	`Dataset Builder`, `Scoring Pipeline`
`Candidate Summary Models`	Generate summaries across model providers for comparison.	`Dataset Builder`, `Scoring Pipeline`
`ROUGE Scorer`	Measures lexical overlap between candidates and references.	`Scoring Pipeline`
`BERTScore Scorer`	Measures semantic similarity with contextual embeddings.	`Scoring Pipeline`
`Scoring Pipeline`	Compares model outputs and summarizes evaluation results.	`ROUGE Scorer`, `BERTScore Scorer`, `Reference Insight Generator`, `Candidate Summary Models`

Data flow

Aiera collects earnings-call transcripts, turns them into a summarization dataset, generates reference insights with Claude 3 Opus, produces candidate summaries from multiple models, and scores candidates with both n-gram and semantic metrics to compare model quality.

“began the process of building a comprehensive data set from eras earnings call transcriptions for use with large language models”transcript, stated

“step we used the February release of anthropics clae 3 Opus which was the highest ranking model on hugging faces open llm Leaderboard on text comprehension tasks at the time of analysis we compiled a list of guided insights for use with the model”transcript, stated

Why this architecture

Aiera needed its own dataset because the target task is domain-specific financial summarization. It kept both extractive/abstractive framing and ROUGE/BERTScore comparison because lexical overlap is interpretable but semantic similarity better matches abstractive insight quality.

“scoring generative text tasks like summarization is difficult for a number of reasons First Quality is subjective”transcript, stated

“the strengths first Bert score captures the deeper semantic meaning moving beyond surface level word overlap this allows it to recognize when two summaries convey the same idea even though they are using different wording second it handles paraphrasing well”transcript, stated

Models used: Claude 3 Opus · Claude 3.5 Sonnet · OpenAI models · Anthropic models · Google models · BERTScore embedding model

Scale

Aiera covers over 45,000 financial events per year.
“feeds era covers over 45,000 events per year with a team of Agents responsible for ensuring consistency of connection the quality of audio and transcription and data intake event coverage includes earnings”transcript, stated
The evaluation compared model providers including OpenAI, Anthropic, and Google.
“scoring for major model providers such as open AI anthropic or Google as well as large context open source models available through hugging faces serverless inference API thank you for joining me once again my name is”transcript, stated

Alternatives considered (roads not taken)

Extractive summarization only.

Rejected because: Abstractive summaries provide more natural and flexible output even though they are harder to evaluate.

“while extractive summarization is simpler and faster for models abstractive summarization offers a more natural and flexible output”transcript, stated

ROUGE-only scoring.

Rejected because: ROUGE ignores semantics and favors word-level similarity.

“include that it ignores semantics and context it favors Word level similarity so it doesn't handle paraphrasing well and it often rewards longer summaries due to recall bias BT scoring is a modern alternative”transcript, stated

Assume a single reference model is neutral.

Rejected because: The team explicitly questioned whether Claude 3 Opus references could bias scoring.

“because we used Claud 3 Opus to generate our initial set of insights does this structurally impact the way that we're scoring”transcript, stated

Patterns

`eval-driven-development` (evaluation)

The work centers on creating a domain dataset and scoring competing summarization models before productizing insights.

“scoring generative text tasks like summarization is difficult for a number of reasons”transcript, stated

`llm-as-judge` (evaluation)

Claude 3 Opus generated reference insights used to score model summaries.

“we used the February release of anthropics clae 3 Opus which was the highest ranking model”transcript, stated

`model-routing` (model selection)

Aiera evaluated different model providers and selected the best-performing model for its summarization use case.

“we found that anthropics latest model Claude 35 Sonet was the winner”transcript, stated

Tradeoffs

Summary style

Chose Abstractive summarization over Extractive-only summarization.

Rationale: Abstractive summaries are more natural and flexible, though harder to score.

“abstractive summarization offers a more natural and flexible output”transcript, stated

Evaluation metric

Chose BERTScore for semantic similarity over ROUGE alone.

Rationale: BERTScore better captures meaning when wording differs.

“semantic similarity rather than simple engram overlap”transcript, stated

Gotchas (what bit them)

ROUGE can underrate semantically equivalent abstractive summaries.

Root cause: It measures word overlap and ignores semantic/contextual similarity.

Resolution: Use BERTScore to compare contextual embeddings alongside ROUGE.

“BT scoring is a modern alternative to Rouge that leverages deep learning models to evaluate summaries based on semantic similarity”transcript, stated

Summarization scoring remains subjective.

Root cause: Quality judgments depend on what makes a good financial insight.

Resolution: Compare multiple metrics and continue exploring bias, provider specificity, cost, and performance questions.

“scoring generative text tasks like summarization is difficult for a number of reasons First Quality is subjective”transcript, stated

Evaluation

Aiera compared generated financial summaries against reference insights using ROUGE and BERTScore, then compared F1 scores across multiple model providers.

Metrics: ROUGE · BERTScore precision · BERTScore recall · BERTScore F1 · Cost and performance questions

“here we report The Bert F1 scores across a number of models and providers recall that the F1 score provides a balanced metric of both precision and recall”transcript, stated

Outcomes

Aiera built a summarization dataset from earnings-call transcriptions.
BERTScore proved better aligned than ROUGE for semantic equivalence in abstractive summaries.
Claude 3.5 Sonnet was reported as the winner for the evaluated use case.

Aiera's platform covers over 45,000 events per year, and the evaluation compared major model providers using semantic scoring metrics.

“we found that anthropics latest model Claude 35 Sonet was the winner”transcript, stated

“Bert does a much better job of suggesting the equivalence of the two statements”transcript, stated

Building and Evaluating a Financial Earnings Call Summarization System

Problem

Architecture

Components

Data flow

Why this architecture

Scale

Alternatives considered (roads not taken)

Extractive summarization only.

ROUGE-only scoring.

Assume a single reference model is neutral.

Patterns

eval-driven-development (evaluation)

llm-as-judge (evaluation)

model-routing (model selection)

Tradeoffs

Summary style

Evaluation metric

Gotchas (what bit them)

ROUGE can underrate semantically equivalent abstractive summaries.

Summarization scoring remains subjective.

Evaluation

Outcomes

`eval-driven-development` (evaluation)

`llm-as-judge` (evaluation)

`model-routing` (model selection)