Scrapivore
Docs menuPython client
Reference

Python client

A small client class covering the main endpoints. Copy it into your project. It only needs requests.

Methods

scrape(url, **opts)
Calls POST /api/scrape/ and returns its JSON. opts takes any scrape option.
batch(urls, concurrency=20, **opts)
Calls POST /api/scrape/batch/ and returns the results list, in input order.
best_proxies(domain, n=10)
Calls GET /api/proxies/best/ and returns the proxies list.
add_proxies(lines)
Calls POST /api/proxies/ with one proxy per line.
bans(domain)
Calls GET /api/bans/ for the domain.
stats(domain)
Calls GET /api/stats/domain/ for the domain.

All methods raise requests.HTTPError on a 4xx or 5xx status.

Save the file as proxydockerbee.py and run it to try the client: it scrapes one URL, then a batch of 20.