Chunking strategies

• Fixed-size: Simple, uniform documents. Chunk size suggestion: 500–1,000 tokens.
• Sentence-based: Good for prose and articles (3–5 sentences).
• Paragraph-based: For well-structured docs (1–3 paragraphs).
• Semantic: For mixed-format sources; dynamic sizes based on topic boundaries.
• Header-based: Documentation with clear headings (split by section).
• Code-aware: For source code — split by function or class rather than paragraphs.

Notes: include small overlaps (50–200 tokens) to reduce boundary artifacts and keep chunk IDs and metadata for traceability.

Embedding models (2026 guidance)

• Production-ready OpenAI embedding models (as of early 2026): text-embedding-3-small and text-embedding-3-large. Always confirm the provider docs for newer embedding releases before migrating (see OpenAI models/docs).
• Generation models: frontier models such as the GPT-5.4 family are available for generation/agent workloads; benchmark latency, cost, and safety before switching production models (see OpenAI GPT-5.4 docs).
• Multimodal embeddings and rerankers: Hugging Face / SentenceTransformers v5.4 added first-class multimodal embedding and reranker support (text, image, audio, video) — useful for visual or cross-modal RAG pipelines; expect additional dependencies and higher compute for multimodal models (see Hugging Face blog, Apr 9 2026).
• Key rule: use the same embedding model (and same preprocessing) for documents and queries — mixing models invalidates similarity scores.
• Open / alternative options: Hugging Face / SentenceTransformers, Cohere Embed v4, and other lightweight models frequently perform well on MTEB-style tasks. Benchmark on your task (recall/precision/latency) before switching.

Workflow

Step 1: Document chunking

• Preserve metadata (source, document id, section heading, URL, language).
• Apply normalized preprocessing (Unicode normalization, consistent whitespace trimming) and the same tokenization pre-step used in embedding generation.
• Add chunk overlap (50–200 tokens) to mitigate split-claim issues.

Step 2: Generate embeddings

• Batch embeddings to improve throughput and manage rate limits (example batch sizes: 50–500 depending on token length and client limits).
• Ensure deterministic preprocessing for both documents and queries.

Example (TypeScript / OpenAI SDK style):

// Embedding examples — use the provider SDK you prefer
const openai = new OpenAI();

async function embedChunks(chunks: Chunk[]) {
  const batchSize = 100; // tune for throughput
  const results = [];
  for (let i = 0; i < chunks.length; i += batchSize) {
    const batch = chunks.slice(i, i + batchSize);
    const r = await openai.embeddings.create({
      model: "text-embedding-3-small",
      input: batch.map(c => c.text),
    });
    for (let j = 0; j < batch.length; j++) {
      results.push({ ...batch[j], embedding: r.data[j].embedding });
    }
  }
  return results;
}

async function embedQuery(query: string) {
  const r = await openai.embeddings.create({ model: "text-embedding-3-small", input: query });
  return r.data[0].embedding;
}

Step 3: Store in a vector database

• Common options: Pinecone, Supabase (pgvector), Qdrant, Weaviate, Milvus.
• Include metadata for filters (document id, source, language, published date) and deduplicate before upsert.
• For large corpora (10K+ docs) consider partitioning or sharding by topic, date, or tenant.

Step 4: Retrieval and reranking

• Hybrid search (semantic + keyword) often improves recall. Combine rankings with Reciprocal Rank Fusion (RRF) or a small learning-to-rank model.
• Reranking approaches:
  • Cross-encoder reranker (best accuracy; higher latency and cost).
  • LLM-based reranker (flexible, instruction-following; measure cost/latency carefully).
  • Lightweight open-model rerankers (now viable in many workloads — benchmark accuracy vs latency).
• Practical pattern: retrieve a broad candidate set (top-50–top-200), then rerank to produce the final top-K for prompt assembly.
• Multimodal reranking: if your documents or queries include images/audio/video, use a multimodal reranker that scores mixed-modality pairs (Hugging Face SentenceTransformers v5.4 provides this capability).

Step 5: Build the RAG prompt

• Instruct: "Only use the provided context. If the answer is not contained there, say 'I don't have enough information.'"
• Reserve tokens: account for model context window, system prompt, retrieved context, and expected output size. Prefer rank-based selection over absolute similarity-score thresholds.

Step 6: Generation and agent runtimes (Responses API)

• Use the Responses API when you need agent-style runtimes, tool execution, or a hosted execution environment — OpenAI now documents an isolated computer environment and shell tool that can safely run constrained commands and host a workspace for agent workflows.
• Choose a model matching your latency and cost needs (OpenAI's GPT-5.4 family includes variants for high-throughput and low-latency workloads); always benchmark against your SLOs.

Example: end-to-end query (Responses API-ish)

async function ragQuery(userQuestion: string) {
  const queryEmbedding = await embedQuery(userQuestion);
  const candidates = await queryVectors(queryEmbedding, 50); // top-N
  const reranked = await rerankCandidates(userQuestion, candidates);
  const topChunks = reranked.slice(0, 5);
  const prompt = buildRagPrompt(userQuestion, topChunks);

  const response = await openai.responses.create({
    model: "gpt-5.4",
    instructions: "You are a helpful assistant. Use only the context provided and cite sources.",
    input: prompt,
  });

  return { answer: response.output_text, sources: topChunks.map(c => ({ id: c.id, score: c.score })) };
}

Notes: some deployments benefit from running rerankers or heavy retrieval steps asynchronously (see orchestration patterns below) to reduce user-perceived latency for interactive flows.

Multimodal RAG (new)

• When your corpus or queries include images, audio, or video, treat those items as first-class documents: extract metadata, optionally OCR text, and store modality-aware representations in your vector store or alongside text embeddings.
• Use multimodal embedding models (e.g., SentenceTransformers v5.4 multimodal models) to encode mixed inputs into a shared vector space for cross-modal similarity.
• For reranking, use multimodal cross-encoders or multimodal rerankers capable of scoring text↔image or text↔video pairs. Expect higher latency and GPU requirements; evaluate cost vs accuracy.

Orchestration and long-running work

• When reranking or retrieval is expensive (large candidate sets, cross-encoder cost), consider async subagent patterns: launch background tasks to compute expensive signals, then return incremental results to the user.
• Frameworks such as LangChain's Deep Agents now support async (non-blocking) subagents and expanded multi-modal filesystem support — useful for long-running document analysis, background reranking, or cross-index aggregations.

Examples

Hybrid search (semantic + keyword)

• Retrieve separately from the vector index and a full-text index; combine with RRF; rerank the merged set.
• Tune RRF constants and top-K sizes for your dataset and recall/precision tradeoffs.

Context window management

• Prefer rank-based selection of chunks until the reserved token budget is reached; keep the highest-ranked chunks and drop the rest.
• For very large outputs, generate in multiple steps: (1) short factual answer, (2) extended explanation on demand.

Incremental ingestion

• Idempotent ingestion with stable chunk IDs.
• Reindex updated documents (partial re-embedding or incremental upserts). Keep a reindex plan for frequently-changing content.

Decision tree

Do you need external knowledge in the LLM response?
├── No → Standard prompting (no RAG needed)
└── Yes
    ├── How much data?
    │   ├── < 50 docs / < 100K tokens → Put it all in context (no vector DB)
    │   ├── 50-10K docs → Single vector index with metadata filters
    │   └── 10K+ docs → Partitioned indexes or hybrid search
    ├── Data freshness?
    │   ├── Static → One-time ingestion
    │   ├── Weekly updates → Batch re-ingestion
    │   └── Real-time → Incremental ingestion + cache invalidation
    ├── Search type?
    │   ├── Semantic similarity → Vector search only
    │   ├── Exact keyword match → Full-text search only
    │   └── Both → Hybrid search with RRF and reranking
    └── Vector store?
        ├── Already using Supabase → pgvector extension
        ├── Need managed service → Pinecone
        ├── Need open source → Qdrant, Weaviate, Milvus
        └── Prototyping → In-memory with cosine similarity

Edge cases and gotchas

• Chunk boundary artifacts: use overlap to reduce split claims.
• Embedding model mismatch: query and document embeddings must use the same model and preprocessing.
• Metadata filtering: filter before vector search where possible to avoid irrelevant candidates in top-N.
• Stale embeddings: reindex when docs change; consider storing doc version hashes to detect drift.
• Score calibration: cosine scores can vary by query — prefer ranks and reranking for comparability.
• Context window overflow: ensure retrieved chunks + system prompt + output fit the model context window.
• Hallucination despite RAG: enforce "use only provided context" instructions; validate citations against retrieved chunks.
• Multi-language: pick a multilingual embedding model if you must support many languages.
• Code vs. prose: use code-aware chunking and preserve function/class boundaries.
• Duplicate chunks: deduplicate before ingestion; use near-duplicate clustering for cleanup.
• Reranker selection: cross-encoders are accurate; lightweight open-model rerankers are increasingly viable — benchmark before full adoption.

Security and dependency notes (urgent)

• pgvector security update: pgvector 0.8.2 (Feb 26, 2026) fixes a buffer overflow in parallel HNSW index builds (CVE-2026-3172). If you run pgvector in production, plan an upgrade to 0.8.2+ and test index rebuilds. (See PostgreSQL announcement linked from pgvector issue tracker.)
• Dependency management: pin extension and model versions, test upgrades in a staging environment, and include index rebuild procedures in your runbook.

Evaluation criteria

• Retrieval precision@K — % of top-K results relevant
• Retrieval recall — % of relevant docs found in top-K
• Answer accuracy — human or LLM-judged correctness vs ground truth
• Citation accuracy — % of claims backed by retrieved chunks
• Hallucination rate — % of unsupported claims
• Latency — end-to-end time (interactive target: < 3s)
• Ingestion throughput — docs/sec for bulk ingestion
• Cost per query — embedding + vector search + LLM generation

Operational notes (2026 signals)

• Use the Responses API for agent-style runtimes, tool execution, and hosted execution environments — OpenAI documents an isolated computer environment and shell tool for safe agent workflows.
• Frontier generation models (GPT-5.4 family) are available; choose and benchmark models by latency, cost, and safety needs before switching production workloads (see OpenAI model docs).
• Hugging Face / SentenceTransformers v5.4 provides multimodal embedding and reranker models for cross-modal RAG; expect GPU requirements and extra dependencies for training and inference.
• LangChain's Deep Agents supports async subagents and expanded multi-modal filesystem support — useful when retrieval/ reranking steps are long-running.
• For vector stores: pgvector remains a strong Postgres extension, but upgrade if you rely on HNSW parallel builds (see pgvector 0.8.2 release notes for CVE fix).

Short checklist for production readiness

• Use a single embedding model for documents + queries
• Deduplicate and normalize content before chunking
• Add overlap to reduce chunk boundary information loss
• Cache frequent query embeddings and reranker outputs
• Monitor retrieval precision@K and hallucination rate in production
• Maintain a reindexing plan for updated documents
• Benchmark open-model rerankers and generators against closed models for cost/latency tradeoffs
• Pin vector-extension versions and have index-rebuild runbooks for upgrades

Experiments and next steps

• Compare cross-encoder reranker vs LLM-based reranker on a held-out test set
• Benchmark multimodal embedding + reranker pipelines (text↔image, text↔video) using SentenceTransformers v5.4 on task-specific metrics
• Evaluate async subagent patterns for long-running retrieval/reranking jobs (background tasking to reduce interactive latency)