Summa Tool

Command-line utilities for building, inspecting, optimizing, and preprocessing Summa indexes.

Build and help

cargo build --release -p summa-tool
target/release/summa-tool --help
target/release/summa-tool <command> --help

Set RUST_LOG=summa_tool=debug for additional diagnostics.

Index lifecycle

Create an index from a schema file:

summa-tool create --index ./my-index --schema ./schema.sdl

Or initialize one from inline SDL:

summa-tool init \
  --index ./my-index \
  --sdl 'index docs { field title: text<simple> [indexed, stored] }'

Index JSON Lines from a file or standard input:

summa-tool index \
  --index ./my-index \
  --documents ./documents.jsonl

zstdcat documents.jsonl.zst |
  summa-tool index --index ./my-index --stdin

index commits and waits for background merges before returning. Inspect or maintain the index:

summa-tool info --index ./my-index
summa-tool diagnose --index ./my-index
summa-tool search --index ./my-index --query 'title:summa' --limit 10
summa-tool merge --index ./my-index
summa-tool reorder --index ./my-index
summa-tool heatmap --index ./my-index --field sparse_embedding
summa-tool warmup --index ./my-index --cache-size 67108864

heatmap requires a BMP sparse-vector field; the example name assumes your schema defines sparse_embedding.

index accepts memory, indexing-thread, compression-thread, and optimization controls. Run summa-tool index --help before tuning them; the defaults are chosen for general-purpose ingestion.

JSONL preprocessing

simhash, sort, and term-stats read JSON objects from standard input and write their primary output to standard output, so they can be composed:

zstdcat documents.jsonl.zst |
  summa-tool simhash --field title --output title_simhash |
  summa-tool sort --field title_simhash --numeric \
  > ordered.jsonl

zstdcat documents.jsonl.zst |
  summa-tool term-stats --field title --field body \
  > term-stats.json

The external sorter writes bounded chunks to a temporary directory. Use --chunk-size to control memory and --temp-dir to choose a volume with sufficient free space.

Development

From the repository root:

cargo fmt --all -- --check
cargo clippy -p summa-tool --all-targets -- -D warnings
cargo test -p summa-tool

Keep the clap help in src/main.rs and this command overview aligned whenever commands or defaults change.

Delete, upsert, and compact rows

delete and upsert require a primary-key field and commit their changes:

summa-tool merge -i ./my_index --compact
summa-tool delete -i ./my_index --key obsolete-id --key another-id
summa-tool upsert -i ./my_index --document '{"id":"existing-id","title":"replacement"}'
summa-tool compact -i ./my_index --memory-budget-mb 256
summa-tool compact -i ./my_index --segment SEGMENT_HEX_ID --memory-budget-mb 256

upsert supplies a complete replacement document and inserts a missing key. compact removes deleted rows from each dirty segment, including a singleton. merge combines segments and retains deletion masks; merge --compact physically removes deleted rows after merging. Indexed-only fields are preserved. Only the current metadata format opens; older indexes must be rebuilt (compatibility). See row deletion for compatibility and diagnostics for health checks.