Automated Prompt Optimization for Intelligent Text Processing using Amazon Bedrock

yuewen_group · Media & Entertainment · 2025 · source Agentic: no · promotional bias: high · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Online literature text analysis

Yuewen Group wanted to move intelligent web-novel text analysis from slower proprietary NLP development to LLM-based processing, but unoptimized prompts made Claude 3.5 Sonnet underperform the traditional NLP baseline on key tasks.

Constraints:

the WebNovel platform has about 260 million users in over 200 countries and regions
prompt engineering expertise was limited during the LLM transition
manual prompt optimization is difficult to evaluate and does not scale across many prompts
prompt effectiveness depends on the target model and use case
the case study is AWS promotional and reports customer experimentation without detailed methodology

Architecture

Yuewen Group moved intelligent text analysis to Anthropic Claude 3.5 Sonnet on Amazon Bedrock, then used Amazon Bedrock Prompt Optimization to rewrite under-developed prompts. Users provide an original prompt template or filled prompt in Bedrock Prompt Management/Playground, choose a target LLM, and start optimization by console click or API. Under the hood, a fine-tuned Prompt Analyzer decomposes the prompt into task instructions, input context, and few-shot demonstrations. A Prompt Rewriter then uses LLM-based meta-prompting to restructure and improve the prompt for the target model. The optimized prompt is compared side-by-side with the original and used for tasks such as character dialogue attribution, name extraction, and multi-option reasoning.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  original-prompt-input[Original prompt input] --> bedrock-prompt-optimization[Bedrock Prompt Optimization]
  bedrock-prompt-optimization --> prompt-analyzer[Prompt Analyzer]
  prompt-analyzer --> prompt-rewriter[Prompt Rewriter]
  prompt-rewriter --> compare-variants-console[Compare Variants console]
  bedrock-prompt-optimization --> compare-variants-console
  compare-variants-console --> claude-3-5-sonnet-text-analyzer[Claude 3.5 Sonnet text analyzer]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Original prompt input`	Provides a full prompt or template with variables such as document placeholders.	`Bedrock Prompt Optimization`
`Bedrock Prompt Optimization`	Coordinates prompt analysis, rewriting, variant comparison, and optimized prompt generation.	`Original prompt input`, `Prompt Analyzer`, `Prompt Rewriter`, `Compare Variants console`
`Prompt Analyzer`	Fine-tuned LLM that decomposes prompt structure into task instruction, input context, and examples.	`Bedrock Prompt Optimization`, `Prompt Rewriter`
`Prompt Rewriter`	Uses LLM meta-prompting to improve prompt signatures, layout, instructions, and output format.	`Prompt Analyzer`, `Compare Variants console`
`Compare Variants console`	Shows the original and optimized prompts side by side for developer review.	`Bedrock Prompt Optimization`, `Claude 3.5 Sonnet text analyzer`
`Claude 3.5 Sonnet text analyzer`	Runs Yuewen's intelligent text-processing tasks using optimized prompts on Amazon Bedrock.	`Compare Variants console`

Data flow

A developer submits an original prompt or prompt template to Bedrock Prompt Optimization and selects a target LLM. Prompt Analyzer extracts the prompt's structural components, Prompt Rewriter rewrites instructions/layout/output formatting for the target model, and the console returns an optimized prompt variant. Yuewen evaluates the optimized prompt on intelligent text-analysis tasks and uses it with Claude 3.5 Sonnet through Amazon Bedrock.

4 supporting quotes

“users input their original prompt. The prompt can be a template with the required variables represented by placeholders”article, stated

“After selecting a target LLM from the supported list, users can kick off the optimization process with a single click”article, stated

“Prompt Analyzer is a fine-tuned LLM which decomposes the prompt structure by extracting its key constituent elements”article, stated

“As the result, Prompt Rewriter produces a refined and enhanced version of the initial prompt tailored to the target LLM.”article, stated

Why this architecture

Yuewen did not get better results simply by moving from traditional NLP to LLMs; prompt quality became the bottleneck. Bedrock Prompt Optimization is structured as an analysis-and-rewrite system because prompt failures come from task instruction, context, examples, layout, output formatting, and target-model fit. The console/API workflow reduces manual trial-and-error and lets developers inspect the optimized prompt before use.

4 supporting quotes

“In certain scenarios, the LLM’s performance fell short of traditional NLP models. For example, in the task of “character dialogue attribution”, traditional NLP models achieved around 80% accuracy, while LLMs with unoptimized prompts only reached around 70%.”article, stated

“This discrepancy highlighted the need for strategic prompt optimization to enhance capabilities of LLMs in these specific use cases.”article, stated

“Context Dependency: Prompt effectiveness is highly contigent on the specific contexts and use cases. A prompt that works well in one scenario may underperform in another, necessitating extensive customization and fine-tuning for different applications.”article, stated

“The console then displays the Compare Variants tab, presenting the original and optimized prompts side-by-side”article, stated

Models used: Anthropic Claude 3.5 Sonnet · Prompt Analyzer fine-tuned LLM · Prompt Rewriter LLM · foundation models

Scale

WebNovel has about 260 million users in over 200 countries and regions.
“Through its overseas platform WebNovel, it has attracted about 260 million users in over 200 countries and regions, promoting Chinese web literature globally.”article, stated
Traditional NLP achieved around 80% accuracy on character dialogue attribution.
“In certain scenarios, the LLM’s performance fell short of traditional NLP models. For example, in the task of “character dialogue attribution”, traditional NLP models achieved around 80% accuracy, while LLMs with unoptimized prompts only reached around 70%.”article, stated
Unoptimized LLM prompts reached around 70% accuracy on the same task.
“In certain scenarios, the LLM’s performance fell short of traditional NLP models. For example, in the task of “character dialogue attribution”, traditional NLP models achieved around 80% accuracy, while LLMs with unoptimized prompts only reached around 70%.”article, stated
Optimized prompts reportedly reached 90% accuracy, 10 points above the traditional NLP baseline.
“optimized prompts reached 90% accuracy, surpassing traditional NLP models by 10% per customer’s experimentation.”article, stated

Alternatives considered (roads not taken)

Continue proprietary traditional NLP models

Rejected because: Yuewen wanted better performance and efficiency than the prolonged development cycles and slow updates of proprietary NLP models.

“Initially relying on proprietary natural language processing (NLP) models, Yuewen Group faced challenges with prolonged development cycles and slow updates.”article, stated

Use Claude 3.5 Sonnet with unoptimized prompts

Rejected because: Unoptimized prompts underperformed the NLP baseline on character dialogue attribution.

“traditional NLP models achieved around 80% accuracy, while LLMs with unoptimized prompts only reached around 70%.”article, stated

Manual prompt optimization

Rejected because: Manual prompt optimization is difficult to evaluate, context-dependent, and increasingly time-consuming as use cases grow.

“Manual prompt optimization can be challenging due to the following reasons: Difficulty in Evaluation: Assessing the quality of a prompt and its consistency in eliciting desired responses from a language model is inherently complex.”article, stated

“This makes manual optimization increasingly time-consuming and labor-intensive. Crafting and iterating prompts for large-scale applications can quickly become impractical and inefficient.”article, stated

Patterns

`prompt-versioning` (prompt-operations)

Bedrock produces an optimized prompt variant and shows it side-by-side with the original for comparison and use.

“presenting the original and optimized prompts side-by-side for quick comparison.”article, stated

`prompt-chaining` (orchestration)

Prompt optimization chains a Prompt Analyzer into a Prompt Rewriter to decompose and then rewrite the original prompt.

“a Prompt Analyzer and a Prompt Rewriter are combined to optimize the original prompt.”article, stated

`fine-tuning-adaptation` (model-operations)

The Prompt Analyzer itself is described as a fine-tuned LLM specialized for decomposing prompts.

“Prompt Analyzer is a fine-tuned LLM which decomposes the prompt structure”article, stated

`structured-output-contract` (output-control)

The rewriter often makes instructions and desired output format more explicit for text-processing tasks.

“The optimized prompt often includes more explicit instructions on processing the input variables and generating the desired output format.”article, stated

Tradeoffs

Use managed Amazon Bedrock infrastructure

Chose Claude 3.5 Sonnet on Amazon Bedrock over continuing only with proprietary NLP infrastructure.

Rationale: Bedrock reduced technical overhead and enabled a managed path to LLM-based text analysis.

“Using Amazon Bedrock significantly reduced technical overhead and streamlined development process.”article, stated

Automate prompt rewriting

Chose Prompt Analyzer plus Prompt Rewriter over manual prompt trial and error.

Rationale: Manual prompt search grows impractical as use cases, models, and prompt variations increase.

“alleviating the need for tedious manual trial and error in model-specific prompt engineering.”article, stated

“the search space for optimal prompts grows exponentially”article, stated

Optimize early-stage prompts rather than fully expert-curated prompts

Chose apply optimizer to under-developed/lazy prompts over expecting equal gains on already expert prompts.

Rationale: AWS notes the optimizer gives more significant gains on less structured early prompts.

“Prompt Optimization excels at quickly optimizing less-structured prompts (a.k.a. “lazy prompts”)”article, stated

“The improvement is likely to be more significant for such prompts compared to those already carefully curated”article, stated

Gotchas (what bit them)

Adopting an LLM did not automatically beat the traditional NLP baseline.

Root cause: Yuewen had limited prompt-engineering experience, and unoptimized prompts failed to unlock the model's capability.

Resolution: Use Bedrock Prompt Optimization to rewrite task prompts for the target LLM.

“Yuewen Group initially struggled to fully harness LLM’s potential due to limited experience in prompt engineering.”article, stated

“LLMs with unoptimized prompts only reached around 70%.”article, stated

Prompt quality is difficult to evaluate objectively.

Root cause: Effectiveness depends on prompt quality, model architecture, training data, and subjective response judgments.

Resolution: Automate prompt analysis/rewriting and evaluate optimized prompts on target tasks.

“Assessing the quality of a prompt and its consistency in eliciting desired responses from a language model is inherently complex.”article, stated

“evaluating LLM response quality for open-ended tasks often involves subjective and qualitative judgements”article, stated

Very long prompts and many examples can hurt optimization.

Root cause: Long prompts increase semantic-understanding difficulty and challenge the rewriter's output length limit.

Resolution: Avoid overly long prompts/examples and structure placeholders with clear contextual sections.

“Excessively long prompts and few-shot examples significantly increase the difficulty of semantic understanding”article, stated

“Another tip is to avoid excessive placeholders among the same sentence and removing actual context about the placeholders from the prompt body”article, stated

Non-English input prompts may not optimize as well.

Root cause: The best-practices section says prompts with a large extent of other languages may not yield the best results.

Resolution: Use English as the input language for the prompt optimizer.

“We recommend using English as the input language for Prompt Optimization.”article, stated

“prompts containing a large extent of other languages might not yield the best results.”article, stated

Evaluation

The AWS post reports open-source dataset evaluation across several task types and Yuewen customer experimentation on text-analysis tasks, especially character dialogue attribution.

Metrics: 70% unoptimized LLM accuracy on character dialogue attribution · 80% traditional NLP accuracy on character dialogue attribution · 90% optimized prompt accuracy on character dialogue attribution · classification/summarization/open-book QA/RAG/agent/function-calling evaluations · prompt-engineering time reduction

“Comprehensive evaluation was done on open-source datasets across tasks including classification, summarization, open-book QA / RAG”article, stated

“optimized prompts reached 90% accuracy, surpassing traditional NLP models by 10% per customer’s experimentation.”article, stated

Outcomes

Prompt optimization reportedly raised character dialogue attribution accuracy from about 70% with unoptimized prompts to 90%.
The optimized prompt surpassed the traditional NLP baseline by 10 percentage points in the reported customer experiment.
Yuewen completed prompt engineering in a fraction of the time, according to the AWS case study.

The reported business context is WebNovel's about 260 million users across more than 200 countries and regions, but the post does not disclose production request volume.

3 supporting quotes