summa-core

summa-core is the storage, indexing, and query engine behind Summa. It is an embeddable Rust library with native and WebAssembly build profiles. For SDL syntax, see the schema guide.

Quick start

With summa-core, tokio (macros, rt-multi-thread), and serde_json dependencies:

use summa_core::{Index, IndexConfig, RamDirectory, index_json_document, parse_single_index};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let schema = parse_single_index(r#"
        index articles { field title: text<en_stem> [indexed, stored] }
    "#)?.to_schema();
    let index = Index::create(RamDirectory::new(), schema, IndexConfig::default()).await?;
    let mut writer = index.writer();
    index_json_document(&writer, &serde_json::json!({"title": "Hybrid search"})).await?;
    writer.commit().await?;
    for hit in index.query("hybrid", 10).await?.hits {
        println!("{} {:?}", hit.score, index.get_document(&hit.address).await?);
    }
    Ok(())
}

Module map

Module	Responsibility
directories	Async storage abstraction plus RAM, mmap, filesystem, HTTP, and slice-caching implementations
dsl	Schema and query-language parsing
index	Public index, reader, writer, and searcher orchestration
merge	Segment lifecycle, metadata publication, merge policy, and cleanup
query	Query planning, scoring, collection, fusion, and reranking
segment	Immutable segment construction, loading, vector data, and format code
structures	SSTables, posting-list codecs, fast fields, SIMD kernels, and vector indexes
tokenizer	Built-in tokenizers and optional Hugging Face tokenizer support

The important ownership boundary is that SegmentManager is the sole writer of index metadata. Indexing, merging, reordering, publication, and cleanup must go through its lifecycle protocol; see segment lifecycle and recovery.

Features

Feature	Purpose
sync (default)	Synchronous search paths; implies native
native	Filesystem/mmap directories, native writer, parallel builders, and native tokenizers
wasm	Browser-compatible writer/reader components and tokenizer backend
http	HTTP-backed directory access
cjk-dict	Embedded Japanese/Korean morphology dictionaries for the morph tokenizer option
metrics	Runtime metrics emission
diagnostics	Additional build diagnostics
query-diagnostics	Opt-in per-query work accounting; implies native
fst-index	FST-backed SSTable block indexes; enabled by native and wasm

Common validation profiles are:

# Default native + synchronous API
cargo test -p summa-core

# Native async API without synchronous query paths
cargo check -p summa-core --no-default-features --features native

# Feature set consumed by summa-wasm
cargo check -p summa-core --no-default-features --features wasm,http \
  --target wasm32-unknown-unknown

Row deletion and upserts

The native writer supports delete_primary_key, upsert_document, compact, and compact_segment. Initialize primary-key deduplication, stage mutations, then commit; reload readers to observe the new generation. Compaction preserves indexed-only fields and trained ANN codes. Old searchers retain their snapshots. force_merge() retains tombstones; force_merge_with_compaction(true) additionally compacts the final output, including a single-segment index. See the deletion design for budgets and compatibility. Multiple staged replacements of one key are supported; commit publishes the latest accepted version. Optional content hashes skip unchanged upserts.

Benchmarks

See the benchmark guide for the complete target list, filtered Criterion runs, and retrieval-quality datasets.

Maintenance rules

• Keep storage access behind Directory/DirectoryWriter; query and segment code must not assume local files.
• Keep on-disk decoding shared between point lookup, scans, and iteration. Every decoder change needs malformed-input and round-trip coverage.
• Treat synchronous and asynchronous search implementations as one behavior contract. Add parity coverage when changing either path.
• Route all SegmentManager construction through the index-level config adapter so every create/open path receives the same resource policy.
• Preserve public re-exports in lib.rs when moving implementation code between modules.

Run cargo fmt --package summa-core -- --check, strict rustdoc, and the narrowest relevant test target before the full crate suite:

RUSTDOCFLAGS="-D warnings" cargo doc -p summa-core --no-deps

Format and serialization changes should also run their module tests and any regression test under summa-core/tests.