📄 Document Collector

Download Bangladesh Bank circulars and collect documents from any URL

Files are saved under All_Downloads/BB/<department>/. Source: bb.org.bd

Matches the dropdown on the BB circulars page.

Collect documents from any URL. The system analyzes the page structure and determines the optimal scraping strategy with optional LLM assistance.

Scans a documents folder and detects duplicates by content, not by filename. Each file's text is extracted, normalized (Bangla Bijoy→Unicode, conjunct repair, spelling unification) and hashed; copies that differ only by page headers/footers are still matched.

Relative to the project workspace (e.g. All_Downloads, All_Downloads/BB/Finance Company Regulation and Policy Department).
Documents with less normalized text are skipped.
Runs Tesseract OCR; much slower on large folders.

Upload a single document and index it through the full ingestion pipeline (extraction, OCR, Bangla normalization, chunking, embedding).

PDF, TXT, MD, HTML or JSON. Scanned PDFs are OCR'd automatically.

Batch-index every supported document in a workspace directory. Files are extracted, chunked, translated, and embedded in the background; per-file task progress is tracked live.

Relative to the project workspace.
Leave empty or 0 to index all.