Enterprise LLM Playground Development for Internal AI Experimentation

thomson_reuters · Media & Entertainment · 2023 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise LLM experimentation platform

Thomson Reuters needed a safe, secure, user-friendly enterprise playground where employees without coding backgrounds could explore in-house and open-source LLMs, combine them with proprietary company data, discover use cases, and later productionize promising ideas.

Constraints:

The platform had to expose generative AI safely inside a global enterprise.
Non-technical employees needed non-programmatic access to LLM capabilities.
The architecture had to integrate multiple open-source and in-house models quickly.
Fine-tuning data needed to stay encrypted and inside the VPC.
The platform needed to support content-grounded experiences and operational observability.

Architecture

Open Arena is a serverless AWS LLM playground built in under six weeks. It uses CloudFront/S3 for the front end and SSO, API Gateway and Lambda for APIs and pre/postprocessing, SageMaker endpoints with Hugging Face containers for models and fine-tuning, DynamoDB for operational data, CloudWatch for monitoring, OpenSearch-backed RAG/re-ranking for content-based experiences, and preset UI tiles for chat, document QA, summarization, and experimentation.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  open-arena-web-ui["Open Arena web UI"] --> cloudfront-s3-front-end["CloudFront S3 front end"]
  cloudfront-s3-front-end --> api-gateway["API Gateway"]
  api-gateway --> lambda-api-layer["Lambda API layer"]
  lambda-api-layer --> sagemaker-model-endpoints["SageMaker model endpoints"]
  lambda-api-layer --> dynamodb-operations-store["DynamoDB operations store"]
  lambda-api-layer --> rag-retrieval-pipeline["RAG retrieval pipeline"]
  rag-retrieval-pipeline --> opensearch-embeddings-store["OpenSearch embeddings store"]
  rag-retrieval-pipeline --> sagemaker-model-endpoints["SageMaker model endpoints"]
  dynamodb-operations-store --> cloudwatch-monitoring["CloudWatch monitoring"]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Open Arena web UI`	Tile-based interface for experimenting with LLM experiences.	`CloudFront S3 front end`
`CloudFront S3 front end`	Hosts the static site and integrates with company SSO.	`API Gateway`
`API Gateway`	Secure API entry point for playground requests.	`Lambda API layer`
`Lambda API layer`	Performs preprocessing and postprocessing around model calls.	`SageMaker model endpoints`, `DynamoDB operations store`, `RAG retrieval pipeline`
`SageMaker model endpoints`	Deploys and serves open-source and in-house LLMs with Hugging Face containers.	`Lambda API layer`
`RAG retrieval pipeline`	Chunks documents, creates embeddings, stores them in OpenSearch, retrieves and re-ranks context.	`SageMaker model endpoints`, `OpenSearch embeddings store`
`OpenSearch embeddings store`	Stores embeddings for content-based retrieval.	`RAG retrieval pipeline`
`DynamoDB operations store`	Stores user queries, responses, response times, and user metadata.	`CloudWatch monitoring`
`CloudWatch monitoring`	Provides dashboards and logging for infrastructure operation.	—

Data flow

Employees use Open Arena tiles in the web UI; CloudFront/S3 and SSO secure access; API Gateway invokes Lambda preprocessing; SageMaker/Hugging Face endpoints run model inference or fine-tuned models; RAG paths split documents into chunks, embed them into OpenSearch, retrieve/re-rank matches, and pass context plus the query to the LLM; DynamoDB and CloudWatch store operational data and logs.

“Lambda functions, triggered by Amazon API Gateway, managed the APIs, ensuring meticulous preprocessing and postprocessing of the data.”article, stated

“documents are split into chunks and then embeddings are created and stored in OpenSearch.”article, stated

Why this architecture

Thomson Reuters chose managed AWS services and modular serverless design to move quickly, keep experimentation controlled and secure, support multiple models and RAG experiences, and adapt as new LLMs and trends emerged.

“pre-built Hugging Face Deep Learning Containers (DLCs), contributed to the pace of innovation.”article, stated

“This is all facilitated by the modular serverless AWS architecture that underpins the solution.”article, stated

Models used: Amazon SageMaker · Hugging Face Deep Learning Containers · Flan-T5-XL · Open Assistant · MPT · Falcon · bitsandbytes · OpenSearch · Amazon Bedrock · SageMaker JumpStart

Scale

Open Arena was developed in under six weeks.
“developed into production using AWS services in under 6 weeks with support from AWS.”article, stated
The platform reached over 1,000 monthly internal users within under a month.
“In under a month, the Open Arena catered to over 1,000 monthly internal users from TR’s global footprint”article, stated
Average interaction time was five minutes per user.
“averaging an interaction time of 5 minutes per user.”article, stated
Thomson Reuters has used AI/NLP in professional information products since the 1992 WIN system.
“A key milestone was the launch of Westlaw Is Natural (WIN) in 1992.”article, stated

Alternatives considered (roads not taken)

Let teams experiment only through coding or ad hoc model access.

Rejected because: Open Arena intentionally provided non-programmatic access for employees without coding backgrounds.

“This provides non-programmatic access for Thomson Reuters employees who don’t have a background in coding”article, stated

Use a single fixed model backend.

Rejected because: The platform was designed to integrate multiple LLMs quickly as new models emerged.

“designed to integrate seamlessly with multiple LLMs through REST APIs.”article, stated

Fine-tune models outside the enterprise security boundary.

Rejected because: The team required fine-tuning data to stay encrypted and inside the VPC.

“all data used to fine-tune LLMs remains encrypted and does not leave the VPC”article, stated

Send content queries directly to an LLM without retrieval/reranking.

Rejected because: Content-based experiences use RAG with chunking, embeddings, retrieval, and re-ranking.

“we use the retrieval/re-ranker approach based on bi-encoder and cross-encoder models.”article, stated

Patterns

`model-routing` (model-ops)

Open Arena exposes multiple open-source and in-house LLMs through REST APIs and use-case-specific tiles.

“Open Arena has been designed to integrate seamlessly with multiple LLMs through REST APIs.”article, stated

`rag-retrieval` (retrieval)

Content-based experiences split documents, embed chunks, retrieve/re-rank matches, and pass context to the LLM.

“The retrieved best match is then passed as an input to the LLM along with the query to generate the best response.”article, stated

`privacy-preserving-processing` (governance)

Fine-tuning happens in a secure VPC with encrypted data.

“Using Hugging Face models on SageMaker allowed the team to fine-tune models in a secure environment because all data is encrypted”article, stated

`observability-tracing` (operations)

DynamoDB stores operational interaction data and CloudWatch provides dashboards/logging.

“DynamoDB, our chosen NoSQL database service, efficiently stored and managed a wide variety of data, including user queries, responses, response times, and user data.”article, stated

Tradeoffs

Build on managed serverless AWS services.

Chose Lambda, DynamoDB, SageMaker, API Gateway, S3, CloudFront, CodeBuild, CodePipeline, and CloudWatch. over A slower custom infrastructure stack..

Rationale: Managed services enabled fast buildout, scalability, manageability, cost-effectiveness, and secure controls.

“We saw the potential of these services not only to provide scalability and manageability but also to ensure cost-effectiveness.”article, stated

Offer a playground rather than only productionized applications.

Chose Open Arena for controlled internal experimentation and use-case discovery. over Waiting to identify only preapproved production use cases..

Rationale: The platform crowdsourced ideas and helped teams identify/refine applications before productization.

“With a goal to foster internal TR LLM experimentation and crowdsource creation of LLM use cases”article, stated

Evaluate multiple model families and optimization techniques.

Chose Flan-T5-XL, Open Assistant, MPT, Falcon, fine-tuning, PEFT, quantization, and model-performance assessment. over Assuming one model is best for all use cases..

Rationale: Thomson Reuters needed model choices that fit NLP relevance, engineering aspects, cost, long documents, and performance.

“We’ve been asking questions like is the higher cost of larger models justified by significant performance gains? Can these models handle long documents?”article, stated

Gotchas (what bit them)

Enterprise experimentation can leak data if fine-tuning is not isolated.

Root cause: Employees may test proprietary content and internal datasets against LLMs.

Resolution: Use SageMaker/Hugging Face inside an encrypted VPC boundary.

“all data is encrypted and doesn’t leave the virtual private cloud (VPC)”article, stated

A playground can become stale if it cannot absorb new models quickly.

Root cause: Generative AI models and trends evolve rapidly.

Resolution: Use REST integrations and a modular serverless architecture.

“flexible enough to react and integrate quickly as new state-of-the art-models were developed and released”article, stated

Non-technical users need guided experiences rather than raw model endpoints.

Root cause: The goal was broad employee experimentation, not only developer access.

Resolution: Provide preset tiles for chat, document questions, summarization, and other experiences.

“These tiles serve as pre-set interactions that cater to the specific requirements of the users.”article, stated

Evaluation

Thomson Reuters evaluated models on open-source legal datasets and internal datasets, assessed model performance on relevant NLP tasks and engineering constraints, tracked usage/response operational data, and used Open Arena to surface use cases for productionization.

Metrics: Over 1,000 monthly internal users · Average 5-minute interaction time · Model performance on relevant NLP tasks · Open-source legal datasets · Thomson Reuters internal datasets · User queries, responses, response times, and metadata

“We have been evaluating these models on the preceding aspects on open-source legal datasets and Thomson Reuters internal datasets”article, stated