Documentation
Home GitHub

Search and Retrieval

Understand how KuraDB prepares queries, runs keyword and semantic retrieval, and shapes safe API results.

Strategy selection

GET /api/search accepts an optional target parameter:

target Work performed Response keys
Omitted or empty Keyword and semantic branches concurrently keyword, semantic
keyword Keyword branch only keyword
semantic Semantic branch only semantic

Both branches use the same loaded database and request context. If either selected branch returns an error, the handler returns HTTP 500 rather than partial data.

Keyword retrieval

The keyword branch calls segmenter.Tokenize, which:

  1. Trims the complete query.
  2. Uses the embedded gse zh_s dictionary and stop-word data.
  3. Runs search-mode segmentation.
  4. Trims tokens, lowercases them, removes empty values, and deduplicates in encounter order.

SearchKeyword builds one case-insensitive SQLite LIKE predicate per token. Rows must satisfy dismiss = FALSE and match at least one token.

Ranking is deterministic:

  1. Descending count of matched tokens.
  2. Ascending row ID.
  3. The request limit, defaulting to 10.

Semantic query embedding

The semantic branch first checks openai.Cache with the exact query string. On a cache miss it sends a one-item batch to OpenAI. A successful vector is inserted into memory and asynchronously persisted to global.db through the cache's OnSet callback.

Startup preloads query-cache blobs only when their byte length equals openai.Dim() * 4. Preload bypasses OnSet, so it does not rewrite the same entry.

Each database bucket stores chunk vectors and one derived vector per source.

Stage 1: source candidates

KuraDB computes cosine similarity between the query and every same-dimension source vector. The candidate count is:

min(max(number of sources / 20, 20), number of sources)

This keeps about 5% of a large source set while retaining at least 20 when available.

Stage 2: chunk ranking

KuraDB collects chunk IDs from candidate sources and calculates same-dimension cosine similarity for each chunk. Work is split across at most CPU count - 1 workers, with a target minimum of 200 chunks per worker. Hits are sorted by descending score and truncated to topK.

If no source vectors exist, search falls back to ranking every chunk vector directly.

Score filtering and hydration

The API layer removes semantic hits whose score is below 0.3. It then fetches the remaining IDs from SQLite with dismiss = FALSE, restores vector-ranking order, and drops any ID that no longer resolves to an active row.

This final hydration keeps SQLite authoritative even when in-memory state changes concurrently.

Grouping

Both flat result sets are grouped by first-seen source. The external representation is:

[
  {
    "source": "/path/to/document.md",
    "matches": [
      {"chunk": 1, "content": "..."},
      {"chunk": 2, "content": "..."}
    ]
  }
]

An empty selected branch is represented as [].

Private ranking data

The handler intentionally strips internal fields before serialization:

Internal field Why it stays private
Row ID Storage implementation detail
Semantic score Threshold and ranking may evolve
Keyword hit count Internal ranking signal
Chunk total Not part of the consumer contract
Overall total The API does not provide pagination metadata

Consumers should depend only on source, matches, chunk, and content.

Consistency behavior

中文