Wolbarg · v0.2.1 · Benchmarks
Measured like infrastructure.Dual-backend. Reproducible. No invented competitor timings.
Full SDK path on SQLite and local Docker PostgreSQL — startup, insert, search, hybrid, filters, rerank, concurrency, compression, memory, and disk. Storage suite uses mock embeddings so you see the database ceiling, not the API quota.
SQLite search @ 1k
2.02 ms
Storage suite · mock embeddings
SQLite insert @ 1k
1.72k/s
remember() throughput
SQLite cold start
7.91 ms
ready() on clean store
PG 16 writers
1.12k/s
Local Docker pgvector
Compression @ 200
99.5%
Active-set reduction
Fastest search
893 µs
SQLite semantic search @ 100 memories
Largest insert advantage
6×
SQLite 1.72k/s vs Postgres 289/s @ 1k (local, same machine)
Leanest disk footprint
2.98 MB
SQLite file size @ 1,000 memories
Strongest concurrency (16 writers)
1.12k/s
PostgreSQL scales writers better than SQLite at this level
Credibility · Two suites
Never confuse storage latency with provider latency
Transparency is the product. Wolbarg ships two benchmark suites on purpose.
Storage Benchmarks
- Embeddings
- Mock OpenAI-compatible embeddings
- Measures
- SDK + database performance (I/O, indexes, concurrency)
- Use for
- Comparing SQLite vs PostgreSQL inside Wolbarg
Production / LIVE Benchmarks
- Embeddings
- Real embedding providers (OpenAI, etc.)
- Measures
- End-to-end latency including network + provider time
- Use for
- Estimating real app wait times — not storage ceiling
Numbers on this page are from the Storage Benchmarks suite (mock · quick). The suite artifact recorded wolbarg@0.2.0; this release packages the same production-hardened path as wolbarg@0.2.1.
SQLite
Local-first memory that feels instantaneous
WAL, prepared statements, in-process vectors, FTS5 hybrid — the default for single-node agents.
Cold start
7.91 ms
Warm start
4.33 ms
Insert @ 1k
1.72k/s
Search @ 1k
2.02 ms
Retrieval top-5
2.16 ms
Hybrid @ 1k
2.81 ms
16 writers
907/s
DB @ 1k
2.98 MB
Compression @ 200
99.5%
Heap @ 1k
26.8 MB
Full SQLite table
| Benchmark | Dataset | Result |
|---|---|---|
| Startup | Cold | 7.91 ms |
| Startup | Warm | 4.33 ms |
| Insert | 100 | 1.85k ops/sec |
| Insert | 1000 | 1.72k ops/sec |
| Search | 100 | 893.1 µs |
| Search | 1000 | 2.02 ms |
| Retrieval top-5 | 1000 | 2.16 ms |
| Retrieval top-10 | 1000 | 1.56 ms |
| Retrieval top-20 | 1000 | 1.84 ms |
| Hybrid · semantic | 1000 | 2.30 ms |
| Hybrid · default | 1000 | 2.81 ms |
| Hybrid · 0.5/0.5 | 1000 | 2.22 ms |
| Filter · unfiltered | 1000 | 2.26 ms |
| Filter · agent | 1000 | 1.41 ms |
| Filter · meta.eq | 1000 | 2.60 ms |
| Filter · meta.and/or | 1000 | 1.53 ms |
| MMR / Rerank · baseline | 1000 | 5.10 ms |
| MMR / Rerank · mmr | 1000 | 2.33 ms |
| MMR / Rerank · λ=0.7 | 1000 | 2.78 ms |
| Concurrency | 2 writers | 1.70k ops/sec |
| Concurrency | 4 writers | 1.60k ops/sec |
| Concurrency | 8 writers | 1.58k ops/sec |
| Concurrency | 16 writers | 907.05 ops/sec |
| Compression | 50 | 98.00% |
| Compression | 200 | 99.50% |
| Memory heap | after 1000 inserts | 26.82 MB |
| Memory RSS | after 1000 inserts | 105.88 MB |
| Database size | 100 | 376.00 KB |
| Database size | 1000 | 2.98 MB |
Production notes
- Best default for local agents, CLI tools, and single-node apps.
- WAL + NORMAL synchronous is the production-safe default.
- File size stays small (~3 MB @ 1k memories in this suite).
- At very high writer fan-out, expect write serialization (see 16-writer drop).
Environment
- Backend
- SQLite + WAL
- Platform
- win32/arm64 · Node v24.13.1 · 8 CPUs · 15.61 GB RAM
- Embeddings
- local-mock-openai-compatible · mock-embed · 384 dims
PostgreSQL
Multi-tenant memory that scales writers
pgvector HNSW, prepared statements, insert coalescing, adaptive overfetch — for shared agent fleets.
Cold start
52.95 ms
Warm start
63.64 ms
Insert @ 1k
289/s
Search @ 1k
4.70 ms
Retrieval top-5
4.47 ms
Hybrid @ 1k
17.97 ms
16 writers
1.12k/s
DB @ 1k
37.51 MB
Compression @ 200
99.5%
Heap @ 1k
39.2 MB
Full PostgreSQL table
| Benchmark | Dataset | Result |
|---|---|---|
| Startup | Cold | 52.95 ms |
| Startup | Warm | 63.64 ms |
| Insert | 100 | 258.16 ops/sec |
| Insert | 1000 | 288.52 ops/sec |
| Search | 100 | 5.97 ms |
| Search | 1000 | 4.70 ms |
| Retrieval top-5 | 1000 | 4.47 ms |
| Retrieval top-10 | 1000 | 5.41 ms |
| Retrieval top-20 | 1000 | 7.49 ms |
| Hybrid · semantic | 1000 | 4.66 ms |
| Hybrid · default | 1000 | 17.97 ms |
| Hybrid · 0.5/0.5 | 1000 | 10.31 ms |
| Filter · unfiltered | 1000 | 4.50 ms |
| Filter · agent | 1000 | 2.54 ms |
| Filter · meta.eq | 1000 | 4.20 ms |
| Filter · meta.and/or | 1000 | 4.26 ms |
| MMR / Rerank · baseline | 1000 | 4.64 ms |
| MMR / Rerank · mmr | 1000 | 3.30 ms |
| MMR / Rerank · λ=0.7 | 1000 | 3.19 ms |
| Concurrency | 2 writers | 295.88 ops/sec |
| Concurrency | 4 writers | 438.97 ops/sec |
| Concurrency | 8 writers | 554.12 ops/sec |
| Concurrency | 16 writers | 1.12k ops/sec |
| Compression | 50 | 98.00% |
| Compression | 200 | 99.50% |
| Memory heap | after 1000 inserts | 39.20 MB |
| Memory RSS | after 1000 inserts | 147.60 MB |
| Database size | 100 | 33.83 MB |
| Database size | 1000 | 37.51 MB |
Production notes
- Best default for multi-tenant SaaS and shared agent fleets.
- Always benchmark against local Docker — remote Neon/Supabase RTTs hide Wolbarg cost.
- HNSW is deferred until first search so bulk inserts stay fast.
- Hybrid keyword path is heavier than SQLite FTS5 in this suite (~18 ms vs ~2.8 ms).
Environment
- Backend
- Postgres + pgvector
- Host
- local Docker pgvector/pg17 (benchmark/docker-compose.yml)
- Platform
- win32/arm64 · Node v24.13.1 · 8 CPUs · 15.61 GB RAM
Charts
SQLite vs PostgreSQL — same machine, same suite
Bar, line, radar, and horizontal comparisons from the published dual-backend run.
Latency comparison
Cold/warm ready() and search/retrieval at 1k ΓÇö milliseconds, lower is better.
Insert throughput
remember() ops/sec at 100 and 1k. Scale axis stops at publicly measured sizes.
Search scaling
Corpus sizes measured in this run: 100 and 1k. 10k / 100k not published here.
Overall performance profile
Radar normalized within this dual-backend run (higher = better).
SQLite vs PostgreSQL
Horizontal head-to-head for the metrics that matter most in this suite.
Concurrency
Parallel writers on one Wolbarg client. Postgres overtakes SQLite at 16 writers.
Retrieval latency @ 1k
Top-5 / top-10 / top-20 recall latency.
Hybrid search
Semantic-only vs hybrid fusion overhead at 1k.
Metadata filtering
Agent and metadata filters at 1k corpus.
Memory heap
Process heap across workload stages (MB).
Database size
On-disk / relation size. Postgres base relation overhead dominates at small corpora.
Compression ┬╖ active-set reduction
Working-set shrink after compress(). Disk size does not shrink.
Competitors
Fair comparisons only
We only place a number in a competitor latency cell when the publisher measured the same class of workload (Wolbarg-style storage/SDK path with mock or fixed embeddings). Accuracy benchmarks that include LLM judges (LoCoMo, LongMemEval, BEAM) and raw ANN engine suites (Qdrant, Weaviate) are different experiments — we cite them in footnotes but never paste them into Startup / Insert / Search cells.
Capability matrix
✅ Supported · ⚠️ Partial · ❌ No · ❓ Unknown
| Feature | Wolbarg | Mem0 | Zep | Letta | Chroma | Qdrant | Weaviate | LlamaIndex | LangGraph Memory | Mastra |
|---|---|---|---|---|---|---|---|---|---|---|
| SQLite-based | ✅ | ⚠️ | ❌ | ⚠️ | ⚠️ | ❌ | ❌ | ⚠️ | ⚠️ | ⚠️ |
| PostgreSQL | ✅ | ⚠️ | ⚠️ | ⚠️ | ❌ | ❌ | ❌ | ⚠️ | ✅ | ⚠️ |
| Local-first | ✅ | ⚠️ | ⚠️ | ⚠️ | ✅ | ⚠️ | ⚠️ | ⚠️ | ⚠️ | ⚠️ |
| Framework Agnostic | ✅ | ✅ | ✅ | ⚠️ | ✅ | ✅ | ✅ | ⚠️ | ❌ | ❌ |
| Model Agnostic | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Memory Compression | ✅ | ✅ | ⚠️ | ✅ | ❓ | ❌ | ❌ | ⚠️ | ⚠️ | ❓ |
| Semantic Search | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ⚠️ | ⚠️ |
| Hybrid Search | ✅ | ⚠️ | ✅ | ❓ | ⚠️ | ✅ | ✅ | ✅ | ❓ | ❓ |
| Open Source | ✅ | ✅ | ⚠️ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Runs Offline | ✅ | ⚠️ | ⚠️ | ⚠️ | ✅ | ✅ | ✅ | ⚠️ | ⚠️ | ⚠️ |
| Multi-tenant scoping | ✅ | ⚠️ | ✅ | ❓ | ⚠️ | ✅ | ✅ | ❓ | ⚠️ | ❓ |
| Public Storage Benchmarks | ✅ | ❌ | ❌ | ❌ | ⚠️ | ✅ | ✅ | ❌ | ❌ | ❌ |
Storage-path latency (apples-to-apples only)
| Framework | Startup | Insert | Search | Retrieval | Hybrid | Concurrency | Storage | Open Source | Production | License | Architecture |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Wolbarg | 7.91 ms cold (SQLite) | 1.72k ops/s @ 1k (SQLite) | 2.02 ms @ 1k (SQLite) | 2.16 ms top-5 @ 1k | 2.81 ms @ 1k (SQLite) | 1.12k ops/s · 16 writers (PG) | SQLite / PostgreSQL | Yes (MIT) | Yes | MIT | Memory SDK |
| Mem0 | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked* | Not publicly benchmarked | Not publicly benchmarked | Pluggable | Yes (Apache-2.0) | Yes | Apache-2.0 | Memory platform |
| Zep | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked* | Not publicly benchmarked | Not publicly benchmarked | Graph + vectors | Partial | Yes | Mixed | Temporal graph memory |
| Letta | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Agent tiers | Yes | Yes | Apache-2.0 | Agent runtime |
| LangGraph Memory | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Pluggable | Yes | Yes | MIT | Agent framework memory |
| LlamaIndex | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Pluggable | Yes | Yes | MIT | RAG framework |
| Mastra | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Framework stores | Yes | Partial | Apache-2.0 | TS agent framework |
| Chroma | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked* | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Embedding DB | Yes | Yes | Apache-2.0 | Vector database |
| Qdrant | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked* | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Vector DB | Yes | Yes | Apache-2.0 | Vector search engine |
| Weaviate | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked* | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Vector DB | Yes | Yes | BSD-3-Clause | Vector database |
| Supermemory | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked* | Not publicly benchmarked | Not publicly benchmarked | Hosted memory API | Partial | Yes | Proprietary / mixed | Hosted memory |
| CrewAI Memory | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Framework-integrated | Partial | Partial | Mixed | Multi-agent framework |
| AutoGen | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Framework-dependent | Yes | Partial | MIT | Multi-agent framework |
| OpenMemory | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Varies | Unknown | Unknown | Unknown | Memory layer (ecosystem) |
| Continue.dev Memory | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | Not publicly benchmarked | IDE-integrated | Yes | Partial | Apache-2.0 | Coding agent memory |
* Cells marked “Not publicly benchmarked*” may have related numbers for a different workload (accuracy evals or ANN engines). Sources below — never treated as Wolbarg storage latency.
- Mem0 accuracy / tokens / p50 — LoCoMo, LongMemEval, BEAM — LLM-judge memory quality, not storage ops/sec.
- Mem0 evaluation docs — Token-efficient algorithm methodology.
- Zep LongMemEval claims — Accuracy + retrieval latency with temporal graph — not Wolbarg remember/recall.
- Qdrant ANN benchmarks — Vector engine RPS/latency on ANN datasets.
- Weaviate ANN benchmarks — ANN QPS / latency / recall trade-offs.
Why
Why Wolbarg performs well
Numbers without mechanism are marketing. Here is the mechanism — kept simple.
SQLite
WAL mode
Readers never block writers. Crash recovery stays safe under concurrent agent loads.
Prepared statements
Hot paths parse SQL once and reuse bindings for remember / recall / forget.
In-process vector index
sqlite-vec + blob index keep ANN inside the same process — no network hop.
Batched ACID transactions
Bulk inserts share a transaction so throughput stays high without sacrificing durability.
FTS5 hybrid
Keyword BM25 lives beside vectors; archived rows are dropped from FTS on archive.
Single-file ops
Backup, copy, and ship memory like any other SQLite database.
PostgreSQL
pgvector + HNSW
Approximate nearest neighbor with HNSW built lazily before the first KNN query.
Named prepared statements
Parse/plan once per connection — insert, search, and filter stay predictable.
Insert coalescing
Concurrent remember() calls collapse into unnest batches; large ingest uses COPY.
Adaptive overfetch
Filtered ANN overfetches just enough to preserve recall after org / metadata filters.
Multi-tenant correctness
Organization scoping is enforced on ANN paths so shared clusters do not leak memories.
Writer scaling
At 16 parallel writers, Postgres throughput overtakes SQLite on the same machine.
Methodology
What we measured — and what we did not
What
Wolbarg.ready, remember, recall (semantic / hybrid / filter / MMR), compress, concurrency writers, heap/RSS, on-disk size.
Why mock
Live API quotas fail before storage contention. Mock OpenAI-compatible embeddings isolate SDK + DB cost.
How
Fixed datasets (100 / 1k), controlled writer counts, reported averages from the published suite artifact.
Limitations
- This published run is scale=quick (100 / 1k corpora). 10k / 100k are not in this artifact — say so rather than invent a curve.
- Mock embeddings isolate storage. Real OpenAI/Ollama latency dominates end-to-end recall in LIVE mode.
- Postgres numbers are local Docker pgvector — not managed cloud round-trips.
- Compression % is active-set reduction; archived rows remain on disk.
- Competitor storage latencies are not published apples-to-apples — we refuse to invent them.
Hardware · published run
- Date
- 2026-07-15T13:59:16.872Z
- SDK
- wolbarg@0.2.1
- Suite
- mock · quick
- Platform
- win32/arm64 · Node v24.13.1 · 8 CPUs · 15.61 GB RAM
- Postgres
- local Docker pgvector/pg17 (benchmark/docker-compose.yml)
- Wall clock
- 34.87s · 92 rows
- Are these storage benchmarks or end-to-end agent latency?
- Primary numbers are Storage Benchmarks: mock embeddings isolating the SDK + database. A separate LIVE suite measures real provider latency. Never mix the two.
- Why don’t competitor latency cells have numbers?
- We only fill a cell when a publisher measured the same workload class. Mem0/Zep publish accuracy (LoCoMo/LongMemEval); Qdrant/Weaviate publish ANN engine suites. Those are different experiments — we cite them instead of inventing Wolbarg-style ops/sec.
- Why is mock mode the default?
- API rate limits fail long before SQLite/Postgres contention. Mock embeddings reveal the real storage ceiling. Use LIVE spots for provider-inclusive latency.
- Do you have 10k / 100k scaling charts?
- This published artifact is scale=quick (100 and 1k). Larger corpora are not listed as measured — we do not invent a curve.
- What does compression % mean?
- Active working-set reduction after compress(). Archived rows remain on disk, so file size does not shrink.
Downloads & reproduce
Raw reports and the suite you can run
Beautiful markdown and machine-readable JSON from the same run that powers these charts.
git clone https://github.com/Atharvmunde11/agentorc-benchmarks.git
cd Wolbarg-benchmarks
npm install
npm run benchmark