Resolve
Feed it a messy batch
CSV, BibTeX, RIS, or a plain DOI list all resolve through the same concurrent batch runner with a priority reranker for ambiguous matches.
The input doesn’t need to be clean first.
CLI / DESKTOP GUI / HTTP SERVER · IN DEVELOPMENT
documentcrawler
Resolve a batch of document references and pull full text from open-access and library sources.
PythonFTS5UnpaywallOpenAlex
AT A GLANCE
Batch
DOI/Title/ISBN/Keyword
FTS5
Full-Text Search
5 Sources
Open Access Default
CLI + GUI
Plus HTTP Server
THE PROBLEM
Chasing down full text for a reading list is still mostly manual
A batch of references — DOIs, rough citations, keywords — usually means opening each one individually across several sites to find a downloadable copy, with no systematic way to dedupe or audit what’s already been collected.
HOW IT WAS BUILT
A batch pipeline with a real library behind it
documentcrawler resolves a batch of references — DOI, title/author/year, ISBN, or keywords — against a configurable set of sources, open-access ones like Unpaywall, OpenAlex, arXiv, PubMed Central, and DOAJ enabled by default.
It ships as a Python CLI with FTS5 full-text indexing, a deduplication engine, and a metadata health auditor, plus an optional desktop GUI and HTTP server for use outside the terminal.
INSIDE THE PRODUCT
Core surfaces that define the experience
Retrieve
Open access sources by default
Unpaywall, OpenAlex, arXiv, PubMed Central, and DOAJ are queried by default; other sources require explicit opt-in via config.
Legitimate sources first, always.
Library
A searchable, deduplicated archive
FTS5 indexing, similarity-based deduplication, and a metadata health scanner keep the resulting library actually usable, not just a folder of PDFs.
Built to be searched, not just stored.
KEY FEATURES
What makes it work
Batch Resolution From Loose References
A DOI, a title/author/year guess, an ISBN, or plain keywords all resolve through the same pipeline, so a messy reading list doesn’t need to be cleaned up before it can be processed.
Open Access First
Unpaywall, OpenAlex, arXiv, PubMed Central, and DOAJ are enabled by default; shadow-library sources are off unless explicitly opted into via config.
A Real Library Underneath
FTS5 full-text search, deduplication with multi-field similarity scoring, and metadata health auditing turn a folder of PDFs into an actually searchable library.
WORKFLOW
Current status
documentcrawler is a mature CLI tool with an existing HTTP server, but has no public web front end yet — turning it into a hosted webapp is scoped work, not a quick redeploy.
01
Browser UI
A web front end needs to be built against the existing HTTP server’s API.
02
Service hosting
The Python HTTP server needs a production host — not a static-asset deploy like the rest of this portfolio.
03
Source configuration
Default source configuration and rate limits need review before opening the service up publicly.