v0.2.0

This is a rewrite rather than a release with features in it.

v0.1.0 had four commands, read one surface, and returned nine fields per paper. This has thirty, reads twelve, and every record says which surface answered and which one stands behind each field.

Read the breaking changes before you upgrade. The JSON keys on a paper have moved.

The category search bug is fixed

In v0.1.0 every category-restricted search returned nothing:

$ arxiv search "attention" --cat cs.CL --limit 5
searching for "attention"...
ERROR  Exit 3.

Exit 3 is "no results". There were 11,831 of them.

The query was assembled by gluing terms together with the literal string +AND+, and then url.Values.Encode percent-encoded that plus into %2B. In an arXiv query URL a + means a space and a %2B means a plus sign, so the moment a term carried cat:, au: or ti: the whole query matched nothing. A plain two word search looked like it worked and quietly returned 17,802 hits where the correct query returns 18,258.

There is now one type that holds a query as plain text with nothing escaped, escaping happens once in the request builder, and nowhere else can reach it. A test asserts the URL and fails on a %2B.

Twelve surfaces, not one

arXiv publishes the same paper in twelve places and they do not carry the same fields.

arxiv surfaces

That prints the table from the values the code uses, so it cannot drift. The export API is one row of it. The OAI-PMH endpoint is where the version history, the sizes, the licence, the report number and the structured author names live. The abstract page has the subject class names and says whether a rendering exists. The LaTeXML rendering has affiliations and the section tree. The RSS feed is the only place that says whether an announcement is new, a cross list or a replacement. BibTeX is arXiv's own bytes, passed through unchanged.

Two planes, two paces

arxiv planes

export.arxiv.org, oaipmh.arxiv.org and rss.arxiv.org are read one request every three seconds, which is what arXiv's terms of use ask for. arxiv.org is read one request every fifteen seconds, because robots.txt says Crawl-delay: 15.

The html pace is a floor rather than a default. --html-rate can make the tool slower and cannot make it faster, and asking for less exits 2 with the reason. Going faster is not theoretical: it returns HTTP 429 with a fourteen byte "Rate exceeded." body and then stalls for about 45 seconds.

Pacing is chosen by host, so a URL on a host with no plane is an error rather than an unpaced request.

Depth is a knob on cost

arxiv paper 1706.03762 --depth quick   # 1 request
arxiv paper 1706.03762                 # 2, the default
arxiv paper 1706.03762 --depth full    # 4, one of them on the slow plane
arxiv paper 1706.03762 --depth text    # 5

Each depth says what it did not look at, in sentences, and names the command that would.

Every record says where it came from

{
  "kind": "paper",
  "depth": "full",
  "surfaces": ["s1", "s2", "s3"],
  "sources": ["https://export.arxiv.org/api/query?...", "..."],
  "retrieved_at": "2026-08-14T07:41:15Z",
  "via": {"submitter": "s2", "license": "s2", "title": "s1"},
  "missed": ["affiliations, the licence name and the section tree were not read; ..."]
}

sources are the URLs actually fetched, so any record can be rebuilt by hand. via names the surface behind each field, which matters because two surfaces disagree: the OAI created date is the day arXiv filed the record and not the day the paper was submitted, and the export API is the one to believe.

A field the read did not look at is absent rather than zero. An empty string in a record used to be indistinguishable from a field nobody asked for.

via earns its place on withdrawn. arXiv removing a record and an author withdrawing a paper are two different events, published in two different places, and only one of them ever happens: 40,000 OAI headers sampled on 2026-08-15 contained no status="deleted" at all, while thousands of papers carry the abstract page's (withdrawn) marker. Both feed the field and via says which answered.

Walking past ten thousand results

arXiv will not page past 10,000 results for any query.

arxiv search cat:cs.CL --all -n 25000 -o jsonl | jq -r .id

--all cuts the query into date slices that each fit and walks them in submission order, and -vv names each slice as it reaches it.

116325 results in 18 slices, 35 count requests to plan and about 1170 to walk
slice 1 of 18, 199108010000 to 200902061159, 1585 results
slice 2 of 18, 200902061200 to 201711110559, 6000 results

That run returned 25,000 distinct ids in 287 requests over about fifty minutes. The export API answered 429 three times along the way, and the walk finished anyway, because the hold is on the plane rather than on the walk. Submission order rather than relevance order is not a preference: relevance is recomputed on every request, so a walk ordered by it both repeats and skips papers and you would never know which.

The graph, the store and RDF

Every read can be turned into claims, and the claims into a store you can query.

arxiv edges 1706.03762
arxiv crawl --search "cat:cs.CL" --max 100 --depth meta --budget 20
arxiv query "select count(*) from claims" --db arxiv.db
arxiv rdf 1706.03762 --format turtle

Twenty predicates, arxiv predicates prints them with what may be at either end. The store is SQLite through a pure Go driver, so there is still nothing to install alongside the binary. A crawl takes a budget in requests per plane, writes a manifest, and resumes where it stopped.

One registration, three surfaces

Every read is declared once and shows up as a CLI subcommand, an HTTP route and an MCP tool.

arxiv serve --addr :8080
arxiv mcp

That is one registration rather than three implementations kept in step by hand, which is why a command cannot exist on the CLI and be missing from the server. A policy test holds that no route writes to arXiv.

Breaking changes

The paper record is not the v0.1.0 one.

v0.1.0	v0.2.0
summary	abstract
published	first_submitted
updated	last_updated
category	primary_category, with categories beside it
pdf	pdf_url
authors as a list of strings	a list of objects with name and via

Anything reading those keys needs updating. jq -r '.[].summary' now prints nulls rather than failing, which is the worst way to find out, so check before you upgrade a script.

arxiv categories reads arXiv's taxonomy instead of printing a list compiled into the binary. It costs two requests, it returns all 155 categories with arXiv's own descriptions, and a code arXiv does not have is refused before any request goes out.

arxiv author <name> is still a search over the author field. arxiv author <id> --id is the new one: it reads an author's own arXiv page, with the ORCID and the paper list.

--sort date still works and is now spelled --sort submitted.

The library package is not source compatible with v0.1.0. arxiv.Paper carries twenty six fields instead of nine, Client takes a Config, and every read takes a context and returns records carrying an envelope.

Install

go install github.com/tamnd/arxiv-cli/cmd/[email protected]

Or download a prebuilt binary from the releases page.