pdf_extract

PDF text extraction for RAG ingestion.

Small wrapper around pdfplumber (MIT-licensed, pure Python) that returns per-page text plus a bit of structure. Kept as its own module so rebuild_indices.py stays framework-agnostic and so a different backend (pdftotext, pymupdf, etc.) could be swapped in later.

Design notes:

  • We keep one chunk per PDF page as the unit passed on to the downstream chunker. Multi-page documents that need finer splitting are handled by the fixed-size character chunker in rebuild_indices.py, which will further split any page whose extracted text exceeds the target chunk size.

  • Empty / near-empty pages (< 20 chars after strip) are skipped — typically figure-only or fully-image pages.

  • Extraction failures on individual pages are logged and skipped, not propagated, so one bad page doesn’t kill the whole ingestion.

Functions

extract_all(pdf_path, *[, ocr])

Convenience: return the whole PDF as one text blob with page markers so downstream chunking can still surface page numbers via regex if needed.

extract_pages(pdf_path, *[, start_page, ...])

Yield (page_number, text) tuples for each non-empty page of pdf_path.
pdf_extract.extract_pages(pdf_path, *, start_page=1, end_page=None, ocr='off')[source]

Yield (page_number, text) tuples for each non-empty page of pdf_path.

page_number is 1-indexed to match how humans (and citations) refer to PDF pages. text is UTF-8 with layout roughly preserved (pdfplumber’s default extract_text output).

start_page (1-based, inclusive) skips pages before it. end_page (1-based, inclusive; None = read to end) stops after it. Use these to ingest only a chapter or appendix of a large textbook without embedding its front-matter, table of contents, or unrelated chapters.

ocr controls vision-OCR via the local ofa model (nothing leaves the node):

  • "off" (default) — text-layer extraction only.

  • "auto" — OCR only pages whose text layer looks degraded (_needs_ocr: many unmapped glyphs, or almost no text). Best for scientific PDFs where most pages are fine but equation-dense ones aren’t.

  • "force" — OCR every page. Slow; use for scanned documents with no usable text layer at all.

Raises ImportError if pdfplumber isn’t installed. Raises FileNotFoundError if the PDF doesn’t exist. Per-page extraction errors are printed to stderr and the page is skipped.

Parameters:

  • pdf_path (Path)

  • start_page (int)

  • end_page (int | None)

  • ocr (str)

Return type:

Iterator[tuple[int, str]]

pdf_extract.extract_all(pdf_path, *, ocr='off')[source]

Convenience: return the whole PDF as one text blob with page markers so downstream chunking can still surface page numbers via regex if needed.

Parameters:
Return type:

str