Chunking

What is it?

Replaceable strategies that split extracted document text into embeddable chunks before storage.

Why does it exist?

Embedding quality depends on chunk boundaries. Markdown headings need different splits than prose or logs.

How does it work?

import { createChunkingStrategy } from "wolbarg";

createChunkingStrategy("fixed")
createChunkingStrategy("sentence")   // default when no markdown headings
createChunkingStrategy("paragraph")
createChunkingStrategy("markdown")   // auto-inferred when headings present
createChunkingStrategy("heading")

Per-call options:

await ctx.ingest({
  agent: "docs",
  source: { path: "./guide.md" },
  chunking: {
    strategy: "markdown",
    chunkSize: 800,
    overlap: 100,
  },
});

Set a default on the constructor with chunking: createChunkingStrategy("markdown").

When should it be used?

• markdown / heading for docs sites and READMEs
• paragraph for long articles
• sentence for dense prose
• fixed for uniform token budgets

Performance notes

• Smaller chunks improve precision, increase storage and recall noise
• Overlap reduces boundary drops for multi-sentence facts
• Batch embedding amortizes network cost during ingest

Related pages

• Document Ingestion
• Provider Architecture
• Performance