LLM-Ready Knowledge Base
A company knowledge base built so an AI system can cite real answers from it — sourced from documents, tickets, code, conversations, and structured records; chunked, embedded, permissioned, evaluated, and kept fresh on AWS.
What 'LLM-ready' actually means
A knowledge base is LLM-ready when an agent retrieving against it can answer a real question, attribute the answer to a specific chunk, respect the requesting user's permissions, and be measured for quality against a real eval set. That requires more than uploading documents to a vector store: it requires source contracts, chunking tuned per content type, refresh policies, hybrid retrieval, reranking, citations, permission propagation, and a feedback loop.
- Real answers with verifiable citations
- Permissions enforced at query time
- Freshness tracked per source
- Quality measured against a real eval set
Sources we build from
Documents (SharePoint, internal docs in S3, product documentation sites), support tickets and existing knowledge bases, code and engineering docs, structured records from analytical tables and operational stores, approved conversation transcripts from the channels you authorize. Each source gets an owner, a freshness target, a classification, and a retention rule before it enters the pipeline.
How it runs on AWS
S3 is the source of truth for raw and processed content with versioning and lifecycle policies. AWS Glue or Lambda handles ingestion, chunking, and PII redaction. Amazon Bedrock provides embeddings (Titan or Cohere via Bedrock) and the reasoning models that answer questions. OpenSearch Serverless holds vector indexes alongside a BM25 keyword leg for hybrid retrieval. Bedrock Knowledge Bases is the managed option when it fits the shape of the workload; a custom pipeline on the same primitives — S3, Bedrock, OpenSearch — covers the cases it does not. Lake Formation and IAM enforce the permission boundary. KMS encrypts at rest. PrivateLink keeps inference traffic in the VPC when the data class requires it.
- S3 source-of-truth, AWS Glue / Lambda ingestion
- Bedrock embeddings (Titan, Cohere via Bedrock)
- OpenSearch Serverless vector + BM25 hybrid retrieval
- Bedrock Knowledge Bases when it fits; custom pipeline when it doesn't
- Lake Formation + IAM permissions, KMS at rest, PrivateLink in transit
What gets evaluated
An eval set is built from real questions — pulled from support tickets, conversation logs, and user research — labeled with the right answer and the citation chunks that should support it. The harness runs retrieval and answering against the set on every change to chunking, embeddings, retrieval weights, or rerank depth. Quality is measured as: did the right chunks come back, did the answer cite them correctly, did it refuse when no chunk supported the claim. Promotion gates block changes that regress.
Closed loop with the work
Resolved threads from Conversation Intelligence propose draft entries; source owners review and approve; the knowledge base picks up the change at the next refresh; retrieval and eval re-run on the new corpus. Freshness is a tracked metric, not an assumption, and the knowledge base grows with the work the organization actually does instead of waiting for someone to write a wiki page.
What it is not
Not a vector store. Not a chatbot. Not a one-time content migration. The knowledge base is the substrate; the agent is the surface; the eval set is the contract that keeps them honest.