Large-Scale Aviation Content Classification on Hacker News Using Small Language Models

skysight · Tech · 2025 · source Agentic: no · promotional bias: high · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Large-scale batch classification over public web/forum data

Skysight wanted to classify the entire Hacker News corpus for aviation-related content using language models, but larger models were too expensive and existing tooling was too unwieldy for offline data-processing workloads. The project needed to fetch and preprocess tens of millions of items, run small-model inference at billion-token scale, emit structured outputs, and aggregate results while acknowledging limited formal evaluation.

Constraints:

• the corpus contains about 42 million Hacker News items
• raw posts need filtering and transformation before inference
• inputs must provide enough detail without overwhelming the model with irrelevant metadata
• the workload requires batch inference over 10.7 billion input tokens
• outputs need to be parseable across tens of millions of inferences
• the project lacks labeled ground truth and formal statistical evaluation
• the result is a first-pass demonstration rather than a rigorously validated classifier

Architecture

Skysight built a batch classification pipeline using Hacker News's public API, highly parallelized fetching, Cloudflare R2 object storage, Parquet files, Polars preprocessing, Llama 3.1 8B inference, prompt prototyping, and JSON-schema structured outputs. The pipeline filters posts with no title or text, concatenates title and text into a labeled prompt input, prototypes on the first 10,000 rows, manually inspects positive classifications and false positives, then scales to about 40 million non-null stories and 10.7 billion input tokens. The model returns a boolean aviation label plus a justification as JSON for downstream aggregation.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  api["Hacker News API"] --> fetch["Parallel fetcher"]
  fetch --> r2["Cloudflare R2 Parquet store"]
  r2 --> prep["Polars preprocessing"]
  prep --> proto["Prototype subset"]
  proto --> inspect["Manual inspection"]
  inspect --> prompt["System prompt and JSON schema"]
  prep --> full["Full inference dataset"]
  prompt --> model["Llama 3.1 8B inference"]
  full --> model
  model --> json["Structured JSON outputs"]
  json --> analysis["Aggregate aviation analysis"]

Components

Data flow

The pipeline fetches historical Hacker News items through the free API, stores them in Cloudflare R2 as more than 900 Parquet files, filters out rows without title or text, and creates an input string with Title and Text fields. A 10,000-row prototype subset is used for model, prompt, and schema iteration. The final run sends the full filtered dataset to Llama 3.1 8B with a system prompt and JSON schema, producing boolean aviation classifications and justifications. The structured JSON outputs are parsed to calculate aviation-related post counts, percentages, time trends, event spikes, and top contributors.

Why this architecture

The pipeline uses a small pre-trained model and batch data-engineering stack because the task is nuanced enough for language understanding but exploratory enough that labeled training data and custom ML engineering would be expensive. Parquet and R2 support scalable corpus storage, Polars prepares compact inputs, JSON schema makes millions of outputs parseable, and a 10,000-row prototype loop keeps prompt and schema iteration cheap before the full run.

Models used: Llama 3.1 8B · small language models · pre-trained language models · Polars · Cloudflare R2 · Parquet · JSON schema

Scale

• The pipeline classifies more than 40 million Hacker News stories.
  “A crude visual of the pipeline we used to classify all of Hacker News - >40M stories, 10.7B input tokens!”article, stated
• The fetched corpus contained about 42 million items.
  “This left us with about 42 million items in total.”article, stated
• The raw data was distributed across more than 900 Parquet files.
  “distributed across >900 Parquet files.”article, stated
• The prototype loop used the first 10,000 rows.
  “After pre-processing, we took the first 10,000 rows to do some prototyping work”article, stated
• The full run processed about 10.7 billion input tokens.
  “about 40 million total stories to analyze, weighing a whopping 10.7 billion input tokens.”article, stated
• Inference completed in a couple of hours.
  “completed inference in a couple of hours”article, stated
• The model generated 1.9 billion structured output tokens.
  “generated an additional 1.9 billion output tokens, neatly structured as JSON strings for easy parsing.”article, stated
• 249,847 items were classified as aviation-related.
  “249,847 of the ~40.5 million posts were classified as being related to aviation”article, stated
• The analysis found 1.13% of story posts were aviation-related.
  “of the nearly 5 million total posts (not comments or other story types), 1.13% are aviation related!”article, stated
• Aviation-related post share rose from around 0.5% in 2007-2012 to around 1.5% in the last four years.
  “hovering around 0.5% from 2007 to 2012, and stabilizing around 1.5% in the last four years.”article, stated

Alternatives considered (roads not taken)

Use larger LLMs for offline corpus classification

Rejected because: The author says larger models were too expensive and tooling was too unwieldy for this offline data-processing task.

Include comments, ancestors, user profiles, and other user activity in each input

Rejected because: The team chose title and text only to provide enough detail without overwhelming the model.

Build a formal evaluation framework before the first full run

Rejected because: For this demonstration, the team manually inspected prototype results and deferred rigorous evaluation to later posts.

Train a specialized model or label a ground-truth dataset first

Rejected because: The project demonstrates useful insights without specialized ML engineering or data-labeling work.

Patterns

batch-offline-processing (inference)

Skysight runs small-model classification as a full-corpus offline batch job over tens of millions of Hacker News items.

structured-output-contract (output)

The classifier is constrained to emit JSON with a boolean aviation flag and justification, enabling automated parsing after billions of output tokens.

prompt-versioning (prompting)

The team explicitly iterates on model selection, system prompt, and output schema during a prototype phase before scaling up.

eval-driven-development (evaluation)

The prototype loop uses manual inspection of true positives and false positives to steer the prompt toward a good-enough first-pass classifier, while explicitly noting the absence of rigorous evaluation.

Tradeoffs

How much input context to include

Chose Title and text only over Comments, ancestors, same-user posts, profile data, and other user activity.

Rationale: The chosen input balances enough information for classification against irrelevant context and token cost.

How much rigor to apply before a first full run

Chose Fast manual inspection and a good-enough prototype over Formal statistical evaluation before scaling.

Rationale: The project was exploratory and prioritized speed, cost, and demonstration value, while acknowledging evaluation blind spots.

Which model class to use

Chose A small pre-trained model, Llama 3.1 8B over Larger, more expensive LLMs or specialized ML model development.

Rationale: The article argues smaller pre-trained models are becoming performant enough for nuanced classification and can run quickly and inexpensively at scale.

Gotchas (what bit them)

The analysis lacks formal accuracy guarantees.

Root cause: The team did not build a ground-truth evaluation set or measure false positives and false negatives systematically.

Resolution: Treat the result as a first-pass demonstration and defer rigorous language-model evaluation to future work.

Adding all available context could overwhelm or distract the model.

Root cause: Hacker News items have comments, ancestors, user profiles, and user activity that may be irrelevant to direct aviation classification.

Resolution: Concatenate only title and text, dropping posts that have neither.

Manual prototype inspection can miss aggregate error patterns.

Root cause: The team inspected positive examples and some false positives but did not quantify false negatives before scaling.

Resolution: Future work should add rigorous evaluation, ensemble modeling, distillation, and adversarial approaches.

The technical result is also a vendor demonstration.

Root cause: Skysight sells infrastructure for large-scale inference workloads and explicitly invites similar use cases.

Resolution: Discount claims about ease and tooling accordingly while preserving the concrete pipeline and scale facts.

Evaluation

The project used prototype-stage manual inspection of outputs and inputs, including reviewing positive classifications and some false positives, but did not perform formal statistical evaluation, ground-truth comparison, or false-negative accounting.

Metrics: manual true-positive inspection · manual false-positive inspection · aviation-related item count · aviation-related story percentage · time-series trend and event-spike inspection · no formal ground-truth accuracy · no measured false-negative rate

Outcomes

• The pipeline processed about 40 million non-null stories and 10.7 billion input tokens.
• Inference completed in a couple of hours and generated 1.9 billion structured output tokens.
• 249,847 items were classified as aviation-related.
• Aviation-related content represented about 0.62% of all Hacker News items and 1.13% of story posts.
• The analysis estimated about a 29% chance that Hacker News top stories contain an aviation-related post at a given time.
• Aviation-related post share increased from about 0.5% in 2007-2012 to about 1.5% in the last four years.
• The author concludes that a small pre-trained model generated useful insights without specialized ML engineering or data-labeling work.

The project is a high-volume offline inference demonstration: 42 million fetched items, more than 900 Parquet files, around 40 million filtered stories, 10.7 billion input tokens, 1.9 billion output tokens, and a couple-of-hours inference run.