Building a RAG System for Cybersecurity Research and Reporting

trainingracademy · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Cybersecurity research and reporting RAG

Trainingracademy built Stinkbait.io to support cybersecurity research and reporting in a domain where knowledge is fragmented, mainstream LLMs may censor security topics, and source data arrives as messy PDFs, HTML pages, and other difficult formats.

Constraints:

• Cybersecurity knowledge is fragmented across subcommunities
• Mainstream LLMs may refuse or censor security-related requests
• PDFs can be invalid, poorly photocopied, or otherwise difficult for OCR
• HTML scraping requires site-specific inspection to get clean data
• Vector database and index choices trade recall and accuracy against latency
• Chunking along conceptual boundaries is ideal but hard to automate at scale

Architecture

Trainingracademy described Stinkbait.io as a cybersecurity RAG application whose most important engineering work happens before generation: ingest heterogeneous security sources, rescue invalid PDFs through Textract and image conversion, manually inspect HTML structure for useful content, choose vector database/index technology based on recall-latency tradeoffs, select embedding models based on context length and training data, chunk using formatting as a proxy for conceptual boundaries, and store vectors externally to survive future database or model changes.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  cybersecurity-researcher[Cybersecurity Researcher] --> stinkbait[Stinkbait]
  stinkbait[Stinkbait] --> data-processing-pipeline[Data Processing Pipeline]
  data-processing-pipeline[Data Processing Pipeline] --> amazon-textract[Amazon Textract]
  data-processing-pipeline[Data Processing Pipeline] --> html-scraper[HTML Scraper]
  data-processing-pipeline[Data Processing Pipeline] --> chunker[Chunker]
  chunker[Chunker] --> embedding-model[Embedding Model]
  embedding-model[Embedding Model] --> vector-store[Vector Store]
  vector-store[Vector Store] --> blob-storage[Blob Storage]
  stinkbait[Stinkbait] --> generation-layer[Generation Layer]

Components

Data flow

Security documents and websites enter a preprocessing pipeline. PDFs go through Textract, with invalid PDFs converted to JPEGs for a backup OCR path. Web pages require manual inspection of relevant HTML elements. Cleaned text is chunked, embedded with a chosen embedding model, stored in a vector database or blob-backed vector store, and retrieved later for cybersecurity research and report generation.

Why this architecture

The architecture centers on retrieval and preprocessing because cybersecurity knowledge is fragmented and LLM safety filters make direct generation unreliable. The talk repeatedly argues that RAG quality depends on input data quality, index latency/recall decisions, embedding model fit, chunking boundaries, and future-proof vector storage.

Models used: OpenAI Ada embeddings · Sentence Transformers · multi QA models · MS MARCO passage models · multilingual models · multimodal models

Scale

• PDF has accumulated decades of variants and extensions since 1993, creating ingestion complexity.
  “really a PDF uh PDF's been around since 1993 and over the decade since as there's lots of extensions to PDFs uh various file formats that are called PDF may not actually be a valid PDF and PDF generator plugins that are associated with various websites they can often”transcript, stated
• Ada embeddings can hold almost five pages of a document in one embedding, making context length part of model selection.
  “the context length for Ada embeddings is very long you can get almost five pages worth a document uh under a single embedding”transcript, stated

Alternatives considered (roads not taken)

Use PostgreSQL with pgvector and IVF-flat

Rejected because: The team saw IVF-flat latency results and concluded it might need a different vector database route.

Chunk only by ideal conceptual boundaries

Rejected because: Conceptual chunking is ideal, but at scale formatting is the realistic proxy.

Store vectors only inside the selected vector database

Rejected because: The speaker recommends disk or blob storage so future database or embedding-model changes do not require recomputing vectors.

Patterns

rag-retrieval (retrieval)

Stinkbait uses retrieval-augmented generation to support cybersecurity research despite fragmented sources and LLM censorship.

data-quality-loop (data)

The system treats PDF repair, OCR, web scraping inspection, and preprocessing as core to RAG quality.

retrieval-ranking (retrieval)

The vector database/index choice is framed as recall or accuracy versus latency, making retrieval behavior an explicit design dimension.

fallback-degradation (resilience)

Invalid PDFs fall back to image conversion and Textract processing rather than being dropped.

Tradeoffs

Choose vector indexing technology

Chose A vector database/index with acceptable latency and recall over Staying with PostgreSQL/pgvector IVF-flat despite poor latency.

Rationale: The team valued open-source database benefits, but latency results pushed them toward a different vector search route.

Pick embedding models by use-case fit

Chose Evaluate context length, benchmark metric, and training data style over Defaulting automatically to Ada embeddings.

Rationale: Ada has long context and good performance, but other models may fit the data type or metric better.

Chunk by formatting at scale

Chose Formatting-based chunking over Pure conceptual-boundary chunking.

Rationale: Conceptual chunking is ideal, but formatting is the realistic signal available at scale.

Gotchas (what bit them)

PDF ingestion is unreliable even when files claim to be PDFs.

Root cause: PDFs include decades of extensions, invalid generator output, and poorly photocopied documents.

Resolution: Use Textract where possible and convert problematic PDFs to JPEG for backup OCR.

Website scraping tools may return poorly manicured data.

Root cause: HTML data is not well structured across varied frameworks and WYSIWYG editors.

Resolution: Inspect each website and choose the relevant HTML elements for that site.

Formatting-based chunks only approximate conceptual boundaries.

Root cause: The realistic chunking signal at scale is often Markdown or document formatting, not true semantic separation.

Resolution: Use formatting as a proxy while recognizing it can differ from actual conceptual boundaries.

Evaluation

The talk does not describe a formal production evaluation harness; it describes practical comparison dimensions such as latency, recall/accuracy, MTEB ranking, context length, and data fit.

Metrics: recall or accuracy versus latency · MTEB leaderboard performance · embedding context length · vector dimensionality and storage size

Outcomes

• The team identified data preprocessing as the dominant practical determinant of RAG quality
• The design lessons cover PDF repair, HTML scraping, vector database selection, embedding model choice, chunking, and future-proof vector storage

No deployment adoption numbers are given; the strongest scale evidence is the speaker's statement that formatting-based chunking is what they have seen at scale.