Chunking strategies

• Fixed-size: Simple, uniform documents. Chunk size suggestion: 500–1,000 tokens.
• Sentence-based: Good for prose and articles (3–5 sentences).
• Paragraph-based: For well-structured docs (1–3 paragraphs).
• Semantic: For mixed-format sources; dynamic sizes based on topic boundaries.
• Header-based: Documentation with clear headings (split by section).
• Code-aware: For source code — split by function or class rather than paragraphs.

Notes: include small overlaps (50–200 tokens) to reduce boundary artifacts and keep chunk IDs and metadata for traceability.

Embedding models (2026 guidance)

• Current recommended OpenAI choices: text-embedding-3-small and text-embedding-3-large remain strong, production-ready options for most RAG pipelines. Always confirm the provider docs for any newer embedding releases before migrating.
• Key rule: use the same embedding model (and same preprocessing) for documents and queries — mixing models invalidates similarity scores.
• Open / alternative options: Hugging Face / SentenceTransformers, Cohere Embed v4, and other lightweight models frequently perform well on MTEB-style tasks. Benchmark on your task (recall/precision/latency) before switching.

Workflow

Step 1: Document chunking

• Preserve metadata (source, document id, section heading, URL, language).
• Apply normalized preprocessing (lowercasing optional, Unicode normalization, consistent whitespace trimming) and the same tokenization pre-step used in embedding generation.
• Add chunk overlap (50–200 tokens) to mitigate split-claim issues.

Step 2: Generate embeddings

• Batch embeddings to improve throughput and manage rate limits (example batch sizes: 50–500 depending on token length and client limits).
• Ensure deterministic preprocessing for both documents and queries.

Example (TypeScript / OpenAI SDK style):

// Embedding examples — use the provider SDK you prefer
const openai = new OpenAI();

async function embedChunks(chunks: Chunk[]) {
  const batchSize = 100; // tune for throughput
  const results = [];
  for (let i = 0; i < chunks.length; i += batchSize) {
    const batch = chunks.slice(i, i + batchSize);
    const r = await openai.embeddings.create({
      model: "text-embedding-3-small",
      input: batch.map(c => c.text),
    });
    for (let j = 0; j < batch.length; j++) {
      results.push({ ...batch[j], embedding: r.data[j].embedding });
    }
  }
  return results;
}

async function embedQuery(query: string) {
  const r = await openai.embeddings.create({ model: "text-embedding-3-small", input: query });
  return r.data[0].embedding;
}

Step 3: Store in a vector database

• Common options: Pinecone, Supabase (pgvector), Qdrant, Weaviate, Milvus.
• Include metadata for filters (document id, source, language, published date) and deduplicate before upsert.
• For large corpora (10K+ docs) consider partitioning or sharding by topic, date, or tenant.

Step 4: Retrieval and reranking

• Hybrid search (semantic + keyword) often improves recall. Combine rankings with Reciprocal Rank Fusion (RRF) or a small learning-to-rank model.
• Reranking approaches:
  • Cross-encoder reranker (best accuracy; higher latency and cost).
  • LLM-based reranker (flexible, instruction-following; measure cost/latency carefully).
  • Lightweight open-model rerankers (now viable in many workloads — benchmark accuracy vs latency).
• Practical pattern: retrieve a broad candidate set (top-50–top-200), then rerank to produce the final top-K for prompt assembly.

Step 5: Build the RAG prompt

• Instruct: "Only use the provided context. If the answer is not contained there, say 'I don't have enough information.'"
• Reserve tokens: account for model context window, system prompt, retrieved context, and expected output size. Prefer rank-based selection over absolute similarity-score thresholds.

Step 6: Generation and agent runtimes (Responses API)

• Use the Responses API when you need agent-style runtimes, tool execution, or a hosted execution environment (OpenAI's Responses API now supports an isolated computer environment and shell tool for running commands, accessing a workspace, and producing artifacts). See provider docs for security and cost considerations.
• Choose a model matching your latency and cost needs (providers publish model family updates; as of early 2026 frontier models such as GPT-5.4 are available — benchmark them against your SLOs).

Example: end-to-end query (Responses API-ish)

async function ragQuery(userQuestion: string) {
  const queryEmbedding = await embedQuery(userQuestion);
  const candidates = await queryVectors(queryEmbedding, 50); // top-N
  const reranked = await rerankCandidates(userQuestion, candidates);
  const topChunks = reranked.slice(0, 5);
  const prompt = buildRagPrompt(userQuestion, topChunks);

  const response = await openai.responses.create({
    model: "gpt-5.4",
    instructions: "You are a helpful assistant. Use only the context provided and cite sources.",
    input: prompt,
  });

  return { answer: response.output_text, sources: topChunks.map(c => ({ id: c.id, score: c.score })) };
}

Notes: some deployments benefit from running rerankers or heavy retrieval steps asynchronously (see orchestration patterns below) to reduce user-perceived latency for interactive flows.

Orchestration and long-running work

• When reranking or retrieval is expensive (large candidate sets, cross-encoder cost), consider async subagent patterns: launch background tasks to compute expensive signals, then return incremental results to the user. Frameworks such as LangChain's Deep Agents now support async subagents and remote agent protocols — useful for long-running document analysis or cross-index aggregations.

Examples

Hybrid search (semantic + keyword)

• Retrieve separately from the vector index and a full-text index; combine with RRF; rerank the merged set.
• Tune RRF constants and top-K sizes for your dataset and recall/precision tradeoffs.

Context window management

• Prefer rank-based selection of chunks until the reserved token budget is reached; keep the highest-ranked chunks and drop the rest.
• For very large outputs, generate in multiple steps: (1) short factual answer, (2) extended explanation on demand.

Incremental ingestion

• Idempotent ingestion with stable chunk IDs.
• Reindex updated documents (partial re-embedding or incremental upserts). Keep a reindex plan for frequently-changing content.

Decision tree

Do you need external knowledge in the LLM response?
├── No → Standard prompting (no RAG needed)
└── Yes
    ├── How much data?
    │   ├── < 50 docs / < 100K tokens → Put it all in context (no vector DB)
    │   ├── 50-10K docs → Single vector index with metadata filters
    │   └── 10K+ docs → Partitioned indexes or hybrid search
    ├── Data freshness?
    │   ├── Static → One-time ingestion
    │   ├── Weekly updates → Batch re-ingestion
    │   └── Real-time → Incremental ingestion + cache invalidation
    ├── Search type?
    │   ├── Semantic similarity → Vector search only
    │   ├── Exact keyword match → Full-text search only
    │   └── Both → Hybrid search with RRF and reranking
    └── Vector store?
        ├── Already using Supabase → pgvector extension
        ├── Need managed service → Pinecone
        ├── Need open source → Qdrant, Weaviate, Milvus
        └── Prototyping → In-memory with cosine similarity

Edge cases and gotchas

• Chunk boundary artifacts: use overlap to reduce split claims.
• Embedding model mismatch: query and document embeddings must use the same model and preprocessing.
• Metadata filtering: filter before vector search where possible to avoid irrelevant candidates in top-N.
• Stale embeddings: reindex when docs change; consider storing doc version hashes to detect drift.
• Score calibration: cosine scores can vary by query — prefer ranks and reranking for comparability.
• Context window overflow: ensure retrieved chunks + system prompt + output fit the model context window.
• Hallucination despite RAG: enforce "use only provided context" instructions; validate citations against retrieved chunks.
• Multi-language: pick a multilingual embedding model if you must support many languages.
• Code vs. prose: use code-aware chunking and preserve function/class boundaries.
• Duplicate chunks: deduplicate before ingestion; use near-duplicate clustering for cleanup.
• Reranker selection: cross-encoders are accurate; lightweight open-model rerankers are increasingly viable — benchmark before full adoption.

Evaluation criteria

• Retrieval precision@K — % of top-K results relevant
• Retrieval recall — % of relevant docs found in top-K
• Answer accuracy — human or LLM-judged correctness vs ground truth
• Citation accuracy — % of claims backed by retrieved chunks
• Hallucination rate — % of unsupported claims
• Latency — end-to-end time (interactive target: < 3s)
• Ingestion throughput — docs/sec for bulk ingestion
• Cost per query — embedding + vector search + LLM generation

Operational notes (2026 signals)

• Use the Responses API for agent-style runtimes, tool execution, and hosted execution environments (OpenAI published a description of the Responses API computer environment and shell tool to enable safe, repeatable agent workflows).
• Frontier models (e.g., GPT-5.4 family) are available; choose and benchmark models by latency, cost, and safety needs before switching production workloads.
• LangChain and related frameworks now support async subagents and LangSmith Fleet skills for orchestrating background work and shareable skills across teams — useful when retrieval/ reranking steps are long-running.
• Hugging Face / SentenceTransformers and other open embedding providers (Cohere, Milvus benchmarks, community benchmarks) continue to be strong options for on-prem or cost-sensitive pipelines — evaluate on task-specific metrics (MTEB-like tests, precision@K, latency).

Short checklist for production readiness

• Use a single embedding model for documents + queries
• Deduplicate and normalize content before chunking
• Add overlap to reduce chunk boundary information loss
• Cache frequent query embeddings and reranker outputs
• Monitor retrieval precision@K and hallucination rate in production
• Maintain a reindexing plan for updated documents
• Benchmark open-model rerankers and generators against closed models for cost/latency tradeoffs

Experiments and next steps

• Compare cross-encoder reranker vs LLM-based reranker on a held-out test set
• Benchmark open-model generators for reranking and low-latency generation (cost vs latency vs accuracy)
• Evaluate async subagent patterns for long-running retrieval/reranking jobs (background tasking to reduce interactive latency)