Hybrid ML and LLM Approach for Automated Question Quality Feedback

stack_overflow · Tech · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: developer Q&A question-quality feedback and moderation assistance

Stack Overflow needed to help askers improve draft questions before public posting while reducing repetitive human reviewer feedback, but pure LLM quality scoring proved unreliable and generic.

Constraints:

• new askers need feedback before questions are posted publicly
• human reviewers repeatedly apply similar comments and templates
• question quality is subjective and hard to reduce to numeric scores
• LLM ratings and feedback did not reliably correlate
• feedback must be specific and actionable rather than generic
• the system needs production monitoring and experimental validation
• the final rollout must improve real question success, not just intermediate review metrics

Architecture

Stack Overflow built Question Assistant as a hybrid ML/LLM workflow. After a pure LLM approach failed to produce reliable quality ratings or category-specific feedback, the team trained individual logistic regression classifiers on historical reviewer comments and close comments. The models use TF-IDF features over question title and body to produce binary indicators for specific feedback needs, initially in the context/background category: problem definition, attempt details, error details, and missing minimal reproducible example. When an indicator fires, the service combines preloaded response text, the question content, and system prompts, then sends that structured input to Gemini to generate question-specific feedback. Models are trained and stored in Azure Databricks/Unity Catalog, served by an Azure Kubernetes service, and instrumented through Azure Event Hub and Datadog for experiments and iteration.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  Question_draft["Question draft"] --> Feedback_classifier_service["Feedback classifier service"]
  Historical_reviewer_data["Historical reviewer data"] --> TF_IDF_vectorizer["TF-IDF vectorizer"]
  TF_IDF_vectorizer --> Feedback_indicator_models["Feedback indicator models"]
  Azure_Databricks["Azure Databricks"] --> Databricks_Unity_Catalog["Databricks Unity Catalog"]
  Databricks_Unity_Catalog --> Feedback_classifier_service
  Feedback_classifier_service --> Feedback_indicator_models
  Feedback_classifier_service --> Preloaded_feedback_library["Preloaded feedback library"]
  Preloaded_feedback_library --> Gemini_feedback_generator["Gemini feedback generator"]
  Feedback_classifier_service --> Gemini_feedback_generator
  Gemini_feedback_generator --> Question_feedback_response["Question feedback response"]
  Question_feedback_response --> Question_draft
  Feedback_classifier_service --> Azure_Event_Hub["Azure Event Hub"]
  Azure_Event_Hub --> Datadog_observability["Datadog observability"]
  A_B_testing_framework["A/B testing framework"] --> Azure_Event_Hub

Components

Data flow

A user drafts a question with a title and body. Feedback classifiers evaluate the draft against predefined indicators derived from historical Staging Ground comments and close reasons. If an indicator is positive, the service retrieves preloaded feedback text, combines it with the question and system prompts, sends it to Gemini, and returns contextual feedback to the asker. Events, predictions, and results flow through Azure Event Hub and Datadog so the team can measure usefulness, run A/B tests, and iterate on indicator models.

Why this architecture

Stack Overflow split classification from language generation because LLM-only quality scoring failed, while historical reviewer comments provided enough signal for discrete binary indicators. Logistic regression gives consistent, interpretable indicator decisions; Gemini is then used where it is stronger, turning structured indicator text plus the draft question into contextual feedback. The architecture also fits production operations by separating Databricks training/model management from Kubernetes serving and instrumenting outcomes through event and observability pipelines.

Models used: Gemini · individual logistic regression models · TF-IDF feature extraction

Scale

• The ground-truth survey was sent to 1,000 reviewers and completed by 152 participants.
  “In a survey sent to 1,000 question-reviewers, we asked them to rate the quality of questions on a scale of 1 to 5 in the three categories. 152 participants fully completed the survey.”article, stated
• The first model category was decomposed into four actionable feedback indicators.
  “For our first experiment, we chose a single category to build models for: context and background. We broke the category into four individual and actionable feedback indicators:”article, stated
• The first A/B test split eligible Staging Ground askers 50/50 between control and variant groups.
  “all eligible Staging Ground askers were allocated for the experiment, split 50/50 between the control and variant groups.”article, stated
• The system produced a steady +12% success-rate improvement across two experiments.
  “We saw a steady success rate of +12% across both experiments.”article, stated
• Question Assistant was made available to all askers on March 6, 2025.
  “we made Question Assistant available to all askers on Stack Overflow on March 6, 2025.”article, stated

Alternatives considered (roads not taken)

Use LLMs to directly rate question quality

Rejected because: The LLM could not reliably align quality ratings with generated feedback and did not change ratings after drafts were edited.

Create numeric ground truth from reviewer ratings

Rejected because: A 1-to-5 survey produced low inter-rater reliability and numeric ratings did not explain what to fix.

Use generic LLM feedback without indicator classifiers

Rejected because: Pure LLM feedback repeated across categories and did not correspond to the category being assessed.

Judge value only by approval rate and review time

Rejected because: Those original metrics were inconclusive, while success rate captured the real quality impact.

Patterns

eval-driven-development (quality)

The team moved through failed LLM tests, a failed ground-truth survey, two A/B tests, and a metric pivot before full rollout.

tool-use-orchestration (system-design)

The system orchestrates classical classifiers, preloaded response templates, system prompts, and Gemini rather than using Gemini for the whole task.

structured-output-contract (data-modeling)

Reviewer feedback is broken into explicit binary indicators so the LLM receives structured intent rather than making a subjective holistic rating.

data-quality-loop (quality)

Historical reviewer comments, close comments, templates, and production event results feed the indicator-model training and iteration loop.

observability-tracing (operations)

Azure Event Hub and Datadog capture user events, predictions, and outcomes so the team can evaluate the production feedback system.

Tradeoffs

Hybrid classifiers plus Gemini

Chose logistic regression indicators for classification and Gemini for contextual feedback over LLM-only quality scoring and feedback.

Rationale: Traditional ML gives more consistent binary indicator decisions, while Gemini generates specific natural-language guidance once the indicator is known.

Feedback indicators instead of score prediction

Chose actionable binary indicators over 1-to-5 quality ratings.

Rationale: A score did not provide actionable information and the survey-based ground truth was unreliable.

Historical reviewer artifacts as training data

Chose clustered reviewer comments, close comments, templates, and close reasons over new numeric survey labels.

Rationale: Past human decisions aligned naturally with actionable feedback indicators, while survey ratings had poor agreement.

Success rate as deployment signal

Chose questions staying open and receiving an answer or +2 score over approval rate and review time alone.

Rationale: The original metrics were inconclusive, but success rate captured actual question quality and downstream usefulness.

Separate training and serving infrastructure

Chose Databricks/Unity Catalog for model training and storage, Azure Kubernetes for serving over a single monolithic training-and-serving service.

Rationale: The setup uses managed ML governance for models while keeping prediction serving in a dedicated production service.

Gotchas (what bit them)

LLM-generated quality ratings were not reliable enough for subjective moderation feedback.

Root cause: The LLM's ratings did not correlate with its feedback, repeated the same advice across categories, and did not respond to draft revisions.

Resolution: Replace holistic LLM scoring with discrete logistic-regression indicator models and use Gemini only for contextual feedback generation.

Reviewer survey labels did not create reliable ground truth.

Root cause: Question quality is subjective, and reviewers disagreed when asked to assign numeric 1-to-5 category scores.

Resolution: Use historical reviewer comments, comment templates, and close reasons as operational signals for specific feedback indicators.

Numeric ratings did not tell askers how to improve.

Root cause: A score alone lacks location, context, and concrete remediation steps.

Resolution: Shift from predicting scores to identifying actionable feedback indicators and generating targeted feedback.

Original experiment metrics missed the system's actual value.

Root cause: Approval rates and review time did not significantly improve, but those metrics did not capture whether posted questions succeeded later.

Resolution: Analyze success rates defined by questions staying open and receiving an answer or a score of at least +2.

Evaluation

Evaluation proceeded through several layers: initial LLM tests, a 1,000-reviewer survey analyzed with Krippendorff's alpha, a Staging Ground A/B test with 50/50 control and variant groups, and a second Ask Wizard A/B test for all eligible askers. The team monitored approval rate, review time, and ultimately question success rate.

Metrics: Krippendorff's alpha for reviewer agreement · 50/50 Staging Ground A/B split · approval rate · average review time · question success rate · +12% success rate across two experiments · full rollout date March 6, 2025

Outcomes

• The pure LLM approach was rejected after unreliable ratings, repetitive feedback, and poor response to draft edits.
• The hybrid system produced contextual feedback by combining indicator classifiers with Gemini.
• The system did not significantly improve approval rates or average review time.
• Question success rates improved by +12% across two experiments.
• Question Assistant was deployed to all askers on March 6, 2025.

The case gives strong rollout and experiment evidence but limited operational detail about Gemini latency, cost, prompt design, or failure handling. The key reusable outcome is the architecture pivot: use ML classifiers for stable indicator decisions and LLMs for language generation.