rebuild_indices

Rebuild one or more ofa RAG collections from collections.toml.

The go-to script to run after git pull-ing a code repo, dropping new PDFs into a papers directory, or adding a new collection entry.

Usage:

python3 $OFA_ROOT/src/rebuild_indices.py                 # all collections
python3 $OFA_ROOT/src/rebuild_indices.py --collection X  # just one
python3 $OFA_ROOT/src/rebuild_indices.py --list          # show configured
python3 $OFA_ROOT/src/rebuild_indices.py --dry-run       # preview only
python3 $OFA_ROOT/src/rebuild_indices.py --force         # ignore mtime cache

Design:

• Config lives at $OFA_ROOT/collections.toml (declarative — see the file’s own header comment for the schema).

• Chunk IDs are sha256(collection + relpath + chunk_index) truncated, so re-runs upsert rather than duplicate. Chunks whose source file is gone from disk are removed from the collection at the end of the pass (“orphan sweep”).

• Per-source-file mtime is remembered in $OFA_VECTORDB/.rebuild_state.json so unchanged files aren’t re-embedded (embedding is by far the slowest step). --force ignores the cache and re-embeds everything.

• PDFs are extracted via pdf_extract (pdfplumber under the hood); each PDF page becomes a chunk unit, further split to PAPER_CHUNK characters if a page is long.

• Code files use chunk_text with the same size/overlap constants as build_index_v2.py (kept in sync manually — see the CHUNK_* constants below).

The script is deliberately self-contained: it does NOT import from build_index_v2.py to keep responsibilities separate. If chunk sizes need to change, update both files.

Functions

chunk_text(text, size, overlap)	Split text into overlapping chunks, preferring newline breaks.
list_collections(cfg)
load_config([path])
load_state()
main()
process_code_file(path, root, collection)	Read a text/code file, return list of (chunk_id, doc, metadata) tuples.
process_pdf_file(path, root, collection, *)	Extract a PDF, return list of (chunk_id, doc, metadata) tuples.
rebuild_collection(name, cinfo, embed_model, ...)	Rebuild one collection according to its config entry.
save_state(state)
stable_id(collection, source_relpath, chunk_idx)	Deterministic chunk ID so re-runs upsert instead of duplicate.
walk_code(root, extensions)	Yield source files under root matching any of extensions.
walk_pdfs(root)

rebuild_indices.chunk_text(text, size, overlap)

Split text into overlapping chunks, preferring newline breaks.

Parameters:

• text (str)

• size (int)

• overlap (int)

Return type:

list[str]

rebuild_indices.stable_id(collection, source_relpath, chunk_idx)

Deterministic chunk ID so re-runs upsert instead of duplicate.

Parameters:

• collection (str)

• source_relpath (str)

• chunk_idx (int)

Return type:

str

rebuild_indices.load_config(path=PosixPath('/home/runner/work/onfield-assistant/onfield-assistant/collections.toml'))

Parameters:

path (Path)

Return type:

dict

rebuild_indices.list_collections(cfg)

Parameters:

cfg (dict)

Return type:

None

rebuild_indices.load_state()

Return type:

dict

rebuild_indices.save_state(state)

Parameters:

state (dict)

Return type:

None

rebuild_indices.walk_code(root, extensions)

Yield source files under root matching any of extensions.

Parameters:

• root (Path)

• extensions (list[str])

Return type:

Iterable[Path]

rebuild_indices.walk_pdfs(root)

Parameters:

root (Path)

Return type:

Iterable[Path]

rebuild_indices.process_code_file(path, root, collection)

Read a text/code file, return list of (chunk_id, doc, metadata) tuples.

Parameters:

• path (Path)

• root (Path)

• collection (str)

Return type:

list[tuple[str, str, dict]]

rebuild_indices.process_pdf_file(path, root, collection, *, page_ranges=None, ocr='off')

Extract a PDF, return list of (chunk_id, doc, metadata) tuples.

Each PDF page becomes at least one chunk. Long pages are split further with PAPER_CHUNK / PAPER_OVERLAP. ocr is forwarded to extract_pages (“off” | “auto” | “force”).

Parameters:

• path (Path)

• root (Path)

• collection (str)

• page_ranges (dict | None)

• ocr (str)

Return type:

list[tuple[str, str, dict]]

rebuild_indices.rebuild_collection(name, cinfo, embed_model, chroma_client, state, *, dry_run=False, force=False, clear=False, incremental=False)

Rebuild one collection according to its config entry.

Prints per-source progress; returns nothing (side effects on chroma_client and state).

Parameters:

• name (str)

• cinfo (dict)

• state (dict)

• dry_run (bool)

• force (bool)

• clear (bool)

• incremental (bool)

Return type:

None

rebuild_indices.main()