LLM-Powered Investment Document Analysis and Processing

angellist · Finance · 2023 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Investment document extraction and investor updates

AngelList needed to extract terms, entities, and update signals from investment and legal documents so operations teams and investors did not have to read entire documents manually.

Constraints:

Investment documents and company updates arrive in varied email/document formats.
Legal documents remain the source of truth, so extraction must be auditable against source text.
Prompt quality needed domain expert input from operations and legal staff.
Scaling depended on OpenAI/Azure rate limits and model availability.

Architecture

AngelList packaged its internal document-classification and extraction stack into Relay. Incoming emails/documents are securely stored, classified, processed through LangChain-structured prompt pipelines, and converted into metadata such as company, investor, amount, valuation, terms, and company-update metrics. Humans review outputs during prompt development, backtests compare extracted values against structured system-of-record data, and Relay exposes parsed dashboards to investors.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  Document_intake["Document intake"] --> Secure_docs["Secure document infrastructure"]
  Secure_docs --> Classifier["Document classifier"]
  Classifier --> Prompt_pipeline["Prompt extraction pipeline"]
  Prompt_pipeline --> Human_review["Human review workspace"]
  Human_review --> Backtesting["Backtesting corpus"]
  Prompt_pipeline --> Backtesting
  Backtesting --> Relay_dashboard["Relay investor dashboard"]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Document intake`	Receives investment documents, legal documents, and company updates through forwarding or upload.	`Secure document infrastructure`
`Secure document infrastructure`	Stores and analyzes incoming documents before extraction.	`Document intake`, `Document classifier`
`Document classifier`	Determines whether a document type supports terms extraction or update parsing.	`Secure document infrastructure`, `Prompt extraction pipeline`
`Prompt extraction pipeline`	LangChain-structured prompts that parse key terms and flag legal language requiring investor attention.	`Document classifier`, `Human review workspace`, `Backtesting corpus`
`Human review workspace`	Internal operations interface where domain experts upload documents, run prompts, and review outputs.	`Prompt extraction pipeline`, `Backtesting corpus`
`Backtesting corpus`	Historical documents matched to structured AngelList system-of-record data for prompt validation.	`Prompt extraction pipeline`, `Relay investor dashboard`
`Relay investor dashboard`	Customer-facing product that organizes extracted investments, terms, updates, and anomalies.	`Backtesting corpus`

Data flow

Documents flow into AngelList's document infrastructure, are classified to determine whether terms extraction is supported, pass through iterated prompt chains for key term extraction and legal-language flagging, then become metadata attached to each customer document and visible in the investor portal.

3 supporting quotes

“we have a really solid document infrastructure that allows us to you know store securely uh store and analyze uh this information”transcript, stated

“we use length chain as the library to structure this and the document comes in we classify it”transcript, stated

“once that's all done you know it becomes meta data that's attached to the customer's document and that they can then retrieve in their portal”transcript, stated

Why this architecture

AngelList treated investment documents as a good LLM fit because they are text-heavy, structured enough to verify, and backed by historical system-of-record data. The architecture keeps legal documents as ground truth while letting domain experts tune prompts and engineers focus on production deployment and scale.

3 supporting quotes

“it is possible to just double check that the extracted text is actually present in the document right”transcript, stated

“the legal documents are the source of truth right so if there's ever a question with you know what the allocation in the deal should be”transcript, stated

“the prompt engineering can be done by the people who know the business the best or the domain the best”transcript, stated

Models used: GPT-4 · OpenAI models · Azure OpenAI · LangChain

Scale

Relay offered a free tier for five investment documents or company updates per month.
“there's a free tier so you can go and try it like on five investment documents per month or five company updates per month”transcript, stated
AngelList claimed 99% extraction accuracy in Relay based on backtesting.
“we're claiming you know 99 accuracy in relay this has been verified like with all of the back testing that we've done”transcript, stated
Scaling pressure centered on GPT-4 / GPT-4-32k access and higher rate limits.
“the biggest struggle for me has been spending time trying to figure out how to get you know access to gpt4 to GPC 432k how to get higher rate limits”transcript, stated

Alternatives considered (roads not taken)

Manual contractor or operations review of documents

Rejected because: Manual review was slower and more expensive, and the team reports models made fewer mistakes on repetitive legal text.

“we could help operations you know move faster so they didn't have to review and read through like entire documents all the time”transcript, stated

“we found that actually the models are more accurate and make fewer mistakes than uh you know when you have people even like experts reading through those documents”transcript, stated

Fully automated prompt development with no human loop

Rejected because: The speaker says goal-setting and domain judgment still required human work.

“for now we're still doing the goal setting you know I haven't seen models that actually tell you this is what you should do”transcript, stated

Integrating additional commercial model providers immediately

Rejected because: The team judged extra provider integration complexity unnecessary while OpenAI met current needs.

“having to integrate with another model would just you know increase the complexity of our system right”transcript, stated

Patterns

`human-in-the-loop-gate` (safety)

AngelList keeps reviewers in the prompt and output loop during development and lets users verify extracted terms against documents.

“you upload the document you apply the prompt to it get the data and then you can review the output then and there”transcript, stated

`eval-driven-development` (evaluation)

Historical documents and structured system-of-record entries provide a backtesting set for checking extracted values at scale.

“we naturally had a training set like or a validation set to say that our extraction actually matches you know what is being inputted into our system”transcript, stated

“we can write the prompt and check that what it extracts actually matches what's in our system and do that at scale like with a lot of examples”transcript, stated

`prompt-chaining` (orchestration)

The document flow classifies incoming documents first, then applies extraction prompts for supported document types.

“the document comes in we classify it then from the classification we determine whether it's one of the documents where we support terms extraction”transcript, stated

`model-routing` (reliability)

AngelList routed calls between Azure and OpenAI to increase rate limits and flexibility.

“we now like you know have calls going to both Azure and open AI directly and like we're able to like increase our rate limits by having that routing happening”transcript, stated

Tradeoffs

Use best-in-class OpenAI capability before deep cost optimization

Chose OpenAI/Azure OpenAI breadth and fast product coverage over early provider diversification or self-hosted custom models.

Rationale: The team wanted to prove broad product value before optimizing pennies or taking on integration complexity.

“we wanted to First go with breadth try to apply this to as many use cases as possible not thinking too hard about optimization just yet”transcript, stated

“we decided to just go use it for now learn as much as we could you know in the space with the Best in Class capability”transcript, stated

Let business/legal experts tune prompts

Chose domain experts writing and iterating natural-language prompts over engineers owning all prompt semantics.

Rationale: AngelList saw the highest leverage when lawyers and operations staff encoded domain expectations while engineers handled deployment.

“if the lawyers on on the team can actually help tune and improve the props right that's ideal for us because this is where we get the highest leverage”transcript, stated

Gotchas (what bit them)

Rate limits became a scaling bottleneck.

Root cause: The production workload depended on GPT-4/GPT-4-32k API access and enough throughput for document volume.

Resolution: Use both Azure and OpenAI directly and route calls to increase limits.

“we failed you know eventually found partners that no could help us with this we also found that we could leverage Azure directly”transcript, stated

Prompt work was not the whole engineering problem.

Root cause: Documents needed formatting, standardization, text extraction, chunking, and pipeline work before prompts could run.

Resolution: Treat prompt engineering as one part of a broader production data-pipeline effort.

“there's still a lot of Plumbing a lot of engineering work around the prompts right to get for example the documents you know formatted in the right way”transcript, stated

Regression testing for prompts was not yet fully structured.

Root cause: The team was still relying on internal review and backtesting rather than a formal repeatable regression suite.

Resolution: Build regression capability to rerun known documents and catch prompt-quality regressions.

“we don't have uh fully structured regression testing capability in the sense that you could you know rerun a bunch of documents”transcript, stated

Evaluation

AngelList backtested extraction prompts against historical documents and structured system-of-record data, while using human review during prompt iteration.

Metrics: 99% claimed Relay accuracy · five documents or company updates per month in free tier · historical documents matched to structured data for validation

“we're claiming you know 99 accuracy in relay this has been verified like with all of the back testing that we've done”transcript, stated

“that data has been doubled triple checked by you know the user by our team like the operations team”transcript, stated

Outcomes

AngelList released Relay for investment document and company update parsing.
The system extracts terms and creates investor dashboards from emailed or uploaded documents.
The team reports contractor-cost savings and 99% accuracy based on backtesting.

Relay processes investment documents/company updates and is exposed publicly with a free tier of five documents or updates per month.

“that's a product called angelisk relay that we shipped and opened to the public”transcript, stated

“relay will parse those extract all the key terms organize them by companies right and build a dashboard for you of all your Investments”transcript, stated