Open Source at Scrapinghub

Leading open source technology for web scraping

Scrapinghub was built on the success of Scrapy, an open source web crawling framework our founders released in 2008. We’ve been managing Scrapy with the same commitment and enthusiasm ever since.

11 years later our commitment and enthusiasm to open source is still the same with a few dozen more open source projects under our belt. They’re all crafted and maintained with the same love and passion. Open source is in our DNA.

Our open source projects

Portia

Portia is our tool for building spiders through a friendly, visual user interface. No programming knowledge required.

Eli5

A library for debugging machine learning classifiers and explaining their predictions.

Scrapely

Scrapely is a library for generating parsers for web pages.

Scrapyjs

ScrapyJS is our middleware for Splash, making it easy to use Splash in your Scrapy projects.

Frontera

Frontera is a framework for managing your crawl logic and policies.

Formasaurus

Formasaurus figures out the type of an HTML form using machine learning. Is it a login, search, sign up, password recovery, contact form, etc?

W3lib

w3lib provides a number of useful web-related functions for your web scraping projects.

ScrapyRT

ScrapyRT let’s you reuse your spider’s logic to extract data from web pages through a single HTTP request.

Loginform

Loginform is a library that detects and fills login forms on specified URLs.

Webstruct

Webstruct is our library for building NER systems that work with HTML.

Queuelib

Queuelib lets you create disk-based queues in Python.

adblockparser

adblockparser is a library for parsing and matching against Adblock Plus filters.

MDR

MDR is a library for detecting and extracting list data from web pages.

Webpager

Webpager is a library for classifying whether a link on a web page is a pagination link or not.

Skinfer

Skinfer is a tool we developed to infer schemas from a sample of JSON data.

Scrapy-StreamItem

Scrapy-StreamItem provides support for working with streamcorpus’ StreamItems.

Wappalyzer-Python

Wappalyzer-Python is a Python based wrapper for Wappalyzer.

We're hiring

Come join our self-motivated, progressive, multi-cultural team.

Open roles

What Web Data Do You Need?

Use Cases

Data APIs

Proxy Management

Developer Tools

Useful Information

How can we help?

AUTOEXTRACT

USEFUL INFORMATION

HOW CAN WE HELP?

Open Source at Scrapinghub

Leading open source technology for web scraping

Our open source projects

SCRAPY

SPLASH

DATEPARSER

SPIDERMON