Indexing Pipeline

Follow a file from a watched inbox through parsing, SQLite persistence, OpenAI embedding, and vector-cache rebuild.

Pipeline stages

inbox → watcher diff → parser → Upsert → pending rows → OpenAI → UpdateEmbedding → vector rebuild

Each loaded database runs one watcher goroutine and one embedding-scheduler goroutine under the daemon's shared cancellation context.

Watcher snapshots

Every 10 seconds, runWatcher calls filesystem.WalkFiles for the database inbox. The walker recursively builds a snapshot containing:

Field	Purpose
`Size`	Detect file-size changes
`ModTime`	Detect content or direct-entry changes
`IsDir`	Distinguish files from recursive directory nodes
`Children`	Preserve the prior subtree for comparison and removal detection

After a successful scan, the daemon asynchronously writes the snapshot to {db}/record.json. A saved snapshot is loaded on the next startup.

The watcher depends on direct directory entry mtime behavior and is designed only for supported local macOS and Linux filesystems.

Parser dispatch

A changed non-directory entry is selected by extension or text sniffing:

Input	Parser path	Chunking behavior
`.pdf`	`go_pkg_parser.PDF`	Document parser chunks
`.docx`	`go_pkg_parser.Docx`	Document parser chunks
`.pptx`	`go_pkg_parser.PPTX`	Presentation parser chunks
`.csv`, `.tsv`	`go_pkg_parser.CSV` through `parseTabular`	Header-aware groups of five data rows
`.xlsx`	`go_pkg_parser.XLSX` through `parseTabular`	Header-aware groups of five data rows
Other valid UTF-8 text	`go_pkg_parser.Markdown`	Markdown/text chunks

Text sniffing reads at most 8,192 bytes, rejects NUL-containing or invalid UTF-8 samples, and rejects empty files.

Skipped inputs

.DS_Store is skipped by name. Known image, video, audio, archive, executable, database, design, font, and other binary extensions are skipped before parsing. This keeps image base64 and unsupported binary data out of the text embedding pipeline.

Transactional upsert

databaseHandler.Upsert is the sole content-write entry point used by the filesystem layer. In one SQLite transaction it:

Marks all currently active rows for the source as dismiss = TRUE.
Inserts each current chunk or updates the (source, chunk) conflict.
Restores current chunks to dismiss = FALSE.
Preserves the embedding only when chunk content is unchanged.
Clears changed embeddings and resets is_embed = FALSE.

This sequence handles shortened files without returning stale trailing chunks and avoids re-embedding unchanged chunks.

Removed files

After scanning current entries, the walker compares the previous snapshot. Missing file nodes are recursively passed to databaseHandler.Dismiss, which soft-deletes active rows by setting dismiss = TRUE. Data remains in SQLite but is excluded by indexing and query reads.

Embedding scheduler

Every five seconds, runEmbedder selects up to 64 rows where:

is_embed = FALSE AND dismiss = FALSE

The scheduler sends their content to EmbedBatch. Oversized input is truncated to 8,000 Unicode runes before the request. The client requests text-embedding-3-small with 512 dimensions and a one-minute HTTP timeout.

The batch is rejected if OpenAI returns the wrong number of vectors or any vector has the wrong dimension.

Race-safe application

UpdateEmbedding updates each row only when all of these still match:

The row ID is unchanged.
dismiss = FALSE.
The stored content equals the content that was embedded.

This prevents a slow OpenAI response from attaching an obsolete vector after a file changes. The function returns only IDs whose updates were applied.

Vector-cache update

For applied IDs, KuraDB stores each vector in the database bucket, tracks the source-to-chunk relationship, and rebuilds every affected source vector. Source vectors are normalized sums of valid, same-dimension chunk vectors.

On daemon startup, the cache is reconstructed from active embedded rows in SQLite, then all source vectors are rebuilt. SQLite remains authoritative if the in-memory cache is lost.

Operational timing

The pipeline is asynchronous. Under normal conditions, a changed file can wait up to one watcher interval before parsing and up to one embedding interval before selection, plus OpenAI request time. Keyword retrieval can use parsed content before embedding completes; semantic retrieval requires an applied vector.