Context Engineering for AI-Assisted Employee Onboarding

etsy · E-commerce · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Employee onboarding and Etsy-specific question answering

Etsy wanted to test whether prompt engineering plus retrieval could answer internal Travel & Entertainment policy questions and seller-community forum questions truthfully, without the cost and data requirements of fine-tuning a domain-specific model.

Constraints:

The system must answer Etsy-specific questions using task/domain documents rather than only model pretraining.
The initial pilot was limited in scale and focused on truthfulness rather than broad product deployment.
Travel & Entertainment policies are relatively clear, while forum answers are heterogeneous and opinionated.
Hallucinated answers are especially risky because wrong answers can be confidently stated.
The team needed evaluation based on manually curated question-answer pairs.

Architecture

Etsy used prompt-based tuning rather than parameter fine-tuning: fold task-specific documents into an embedding/search index, retrieve relevant text sections for each query, augment the final prompt with that context, and evaluate generated answers against manually extracted reference answers.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  user-question["User question"] --> embedding-search["Embedding search"]
  task-document-corpus["Task document corpus"] --> embedding-search
  embedding-search --> prompt-builder["Prompt builder"]
  task-document-corpus --> prompt-builder
  prompt-builder --> llm-answer-generator["LLM answer generator"]
  llm-answer-generator --> human-evaluation-set["Human evaluation set"]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`User question`	Employee or seller-support question submitted to the question-answering system.	`Embedding search`
`Task document corpus`	T&E policy documents or Etsy forum data folded into the embedding space.	`Embedding search`, `Prompt builder`
`Embedding search`	Retrieval step that embeds the query and finds relevant Etsy-specific text sections.	`User question`, `Task document corpus`, `Prompt builder`
`Prompt builder`	Combines the user question, retrieved text, and explicit instructions or reasoning prompts.	`Embedding search`, `LLM answer generator`
`LLM answer generator`	General-purpose LLM prompted to answer using Etsy-specific retrieved context.	`Prompt builder`, `Human evaluation set`
`Human evaluation set`	Manually curated question-answer pairs and extracted reference answers used to judge quality.	`LLM answer generator`

Data flow

Task documents are embedded into an index. For each onboarding or forum query, the query is embedded, relevant Etsy-specific text sections are retrieved, and those sections are added to the final prompt before the LLM answers. Human evaluators compare the LLM answer with a reference answer.

“For every query, embedding the initial formulation led to relevant Etsy-specific text sections, which were then added to the question to form the final prompt.”article, stated

“For each question in the test set, we compared the answer generated by the LLM with the answer we had extracted from the relevant policy document to form a judgment of answer quality.”article, stated

Why this architecture

Prompt-based retrieval was chosen because fine-tuning was more expensive and required enough labeled data, while prompt engineering could use an adequate representation of documents and leave model weights unchanged.

“This is a robust but somewhat expensive approach, and only possible when there’s a sufficiently large dataset to support it.”article, stated

“Prompt-based tuning was an attractive proposition in our case because all that it required was simply an adequate representation of task-specific documents.”article, stated

Models used: OpenAI o-series · Google Gemini family · Embedding model/API · General-purpose LLM

Scale

The T&E pilot used 40 manually curated question-answer pairs.
“We tested the resulting system’s performance on a manually curated set of 40 question-and-answer pairs.”article, stated
The forum experiment used 50 manually curated question-answer pairs.
“we tested performance on a manually curated set of 50 question-and-answer pairs, comparing LLM-generated answers with answers extracted from the relevant Etsy forum”article, stated
The internal T&E use case reached approximately 86% satisfactory/correct answers before further prompt maneuvering.
“the LLM answered approximately 86% of the questions correctly, or at least satisfactorily enough that there was no need for further prompt maneuvering.”article, stated
The forum use case reached approximately 72% correct answers before further prompt maneuvering.
“the LLM answered approximately 72% of the questions correctly before any prompt maneuvering. In the remaining 28% of cases, the LLM generated an answer which was either wrong or misleading.”article, stated

Alternatives considered (roads not taken)

Fine-tuning on Etsy documents

Rejected because: Fine-tuning was robust but expensive and only feasible with a sufficiently large labeled dataset.

“This is a robust but somewhat expensive approach, and only possible when there’s a sufficiently large dataset to support it.”article, stated

Explicit uncertainty instruction alone

Rejected because: It prevented a hallucination but also missed the correct answer that was present in the documents.

“We at least avoid the hallucination, but not in a way that will be very helpful to the user. Also note that the LLM misses the correct answer, which is actually present in the collection of documents.”article, stated

Generic contextual preamble for complex forum questions

Rejected because: Adding context did not solve cases where the answer required more careful reasoning over the forum data.

“However, adding a contextual preamble did not help:”article, stated

“when complex reasoning is required, chain-of-thought prompting must be carefully set up.”article, stated

Patterns

`rag-retrieval` (retrieval)

Etsy retrieves relevant domain text sections with embedding-based search and adds them to the prompt before answer generation.

“For every query, embedding the initial formulation led to relevant Etsy-specific text sections, which were then added to the question to form the final prompt.”article, stated

`query-rewriting` (retrieval)

The forum experiment explored query expansion and chain-of-thought milestones, discovering that expansion chunks must align with the reasoning path.

“This observation has concrete implications when it comes to chain-of-thought prompting: specific logical milestones must be congruent with query expansion chunks to maximize alignment.”article, stated

`eval-driven-development` (evaluation)

The team used manually curated QA pairs and reference answers to judge whether prompt/retrieval variants produced truthful answers.

`confidence-thresholding` (reliability)

The prompts explored uncertainty and source-snippet disclosure as ways to expose or reduce overconfident hallucinations.

“asking the LLM to identify specific source snippets is a good way to flag potential hallucinations.”article, stated

Tradeoffs

Adaptation method

Chose prompt-based tuning with retrieval over parameter fine-tuning.

Rationale: Prompting avoided the cost and data requirements of fine-tuning while still incorporating Etsy-specific documents through retrieval.

“Prompt engineering, on the other hand, freezes the model weights, treating the LLM as a black box.”article, stated

“Prompt-based tuning was an attractive proposition in our case because all that it required was simply an adequate representation of task-specific documents.”article, stated

Reliability intervention

Chose chain-of-thought explanation prompt for a failed policy answer over uncertainty-only instruction.

Rationale: Uncertainty avoided a false answer but did not recover the correct one; the explanation prompt raised the fact-checking bar and found the answer.

“prompting for chain-of-thought reasoning is often beneficial, as it provides a way to structure the prompt with logical milestones to ensure that the LLM follows a well-circumscribed reasoning path.”article, stated

Forum answer quality target

Chose measure surfacing of input-data responses over treating forum answers as official policy truth.

Rationale: Forum content may diverge from official policy, so the experiment measured faithfulness to the supplied forum data rather than policy correctness.

“For the purposes of our investigation, we measured only whether the LLM could accurately surface responses based on the input data it was given.”article, stated

Gotchas (what bit them)

The LLM could be confidently wrong even with retrieved policy context.

Root cause: The model's answer was not guaranteed to use the correct retrieved fact, and it could assert false policy details with high confidence.

Resolution: Use source-snippet disclosure and careful reasoning prompts to expose hallucinations and improve fact checking.

“In the remaining 14% of cases, the LLM generated an answer which was either wrong or misleading. For example, the LLM asserted with high confidence”article, stated

Simple uncertainty prompting avoided hallucination but lost useful answers.

Root cause: An instruction to say 'I have no idea' made the model abstain even when the correct answer existed in the documents.

Resolution: Prompt for explanations or source snippets rather than only abstention.

“We at least avoid the hallucination, but not in a way that will be very helpful to the user. Also note that the LLM misses the correct answer”article, stated

Forum data was harder than policy data.

Root cause: Forum answers were heterogeneous, opinionated, and sometimes diverged from official Etsy policy.

Resolution: Measure whether the model accurately surfaces responses from the input data and use source snippets to locate answer provenance.

“Etsy forum data is substantially more complex: much less homogeneous (both in terms of style and scope), with answers by nature much more opinionated.”article, stated

Evaluation

Etsy evaluated generated answers against manually extracted reference answers on two curated test sets: 40 T&E policy questions and 50 forum questions.

Metrics: approximately 86% correct or satisfactory T&E answers · 14% wrong or misleading T&E answers · approximately 72% correct forum answers · 28% wrong or misleading forum answers

“We tested the resulting system’s performance on a manually curated set of 40 question-and-answer pairs.”article, stated

“the LLM answered approximately 72% of the questions correctly before any prompt maneuvering. In the remaining 28% of cases, the LLM generated an answer which was either wrong or misleading.”article, stated

Outcomes

Prompt-based retrieval worked reasonably well for the internal T&E policy case.
Forum QA exposed a higher failure rate and showed that prompt engineering has limits on heterogeneous data.
Source-snippet disclosure emerged as the most useful hallucination flag.

The work was an investigation/pilot, not a full production deployment; scale is reported as 40 T&E QA pairs and 50 forum QA pairs.

“Our experiments with two flavors of AI-assisted onboarding (internal and external) indicate that, most of the time, prompt engineering works reasonably well as a practical alternative to parameter-based fine-tuning.”article, stated

“It has limitations, however, so care should still be taken when assessing answer truthfulness. Asking the LLM to disclose specific sources is a good way to flag potential hallucinations.”article, stated