If you were looking for a web scraper that actually works, you just found it. It's called Scrapper! Downloading a page from the Internet and then extracting an article from it in a structured format is not as easy as it may seem at first glance. Sometimes it can be damn hard. But Scrapper will help you with this, I hope :) To do this, Scrapper has plenty of features.
Scrapper is a free and open-source product. It combines the power and experience of related open-source projects, so give it a try first.
Quick start
Quick start a scrapper instance:
docker run -d -p 3000:3000 --name scrapper amerkurev/scrapper:latestScrapper will be available at http://localhost:3000/. For more details, see Usage
Watch a 30-second demo reel showcasing the web interface of Scrapper.
demo1280x720.mp4
The main features of Scrapper are:
- The headless browser is already built-in. Modern websites actively use JavaScript on their pages, fight against crawlers, and sometimes require user actions, such as agreeing to the use of cookies on the site. All of these tasks are solved by the Scrapper using the excellent Playwright project. The Scrapper container image is based on the Playwright image, which already includes all modern browsers.
- The Read mode in the browser is used for parsing. Millions of people use the "Read" mode in the browser to display only the text of the article on the screen while hiding the other elements of the page. Scrapper follows the same path by using the excellent Readability.js library from Mozilla. The parsing result will be the same as in the Read mode of your favorite browser.
- A simple and beautiful web interface. Working with Scrapper is easy and enjoyable because you can do it right in your browser. The simple web interface allows you to debug your query, experiment with each parameter in the API, and see the result in HTML, JSON, or a screenshot. The beautiful design helps achieve an excellent Pico project. A dark theme for comfortable reading is also available.
- The Scrapper REST API is incredibly simple to use as it only requires a single call and just a few parameters making it easy to integrate into any project. Furthermore, the web interface offers a visual query-builder that simplifies the learning process for the user.
- Scrapper can search for news links on the main pages of websites. This is difficult because there may be not only links to news but also links to other parts of the website. However, Scrapper can distinguish between them and select only links to news articles.
And many other features:
- Stealth mode. Various methods are used to make it difficult for websites to detect a Headless browser and bypass web scraping protection.
- Caching results. All parsing results are saved to disk, and you can access them later by API without repeating the whole request.
- Page screenshots. Headless browsers don't have a window, but screenshots allow you to see the page as it appears to the parser. This is very useful!
- Incognito mode or persistent sessions. You can configure the browser to work in incognito mode or without it. In this case, the browser will save session data such as cookies and local storage to disk. To use them again.
- Proxy support. HTTP/SOCKS4/SOCKS5 proxy work is supported.
- Fully customizable. You can control a lot through the API: additional HTTP headers, viewport for device emulation, Readability parser settings, and much more.
- Delivered as a Docker image. Scrapper is built and delivered as a Docker image, which is very easy to deploy for testing and production. No dependencies or installations on the host. All you need to run Scrapper is Docker.
- Free license. Scrapper doesn't ask for money, insert ads, or track your actions ever. And if you want to help the project develop further, just give us a star on GitHub ⭐
The Scrapper Docker image is based on the Playwright image, which includes all the dependencies needed to run a browser in Docker and also includes the browsers themselves. As a result, the image size is quite large, around 2 GB. Make sure you have enough free disk space, especially if you plan to take and store screenshots frequently.
To get the latest version of Scrapper, run:
docker pull amerkurev/scrapper:latestScrapper uses two directories on the disk. The first one is the user_data directory. This directory contains browser session data such as cookies and local storage.
Additionally, the cache of Scrapper's own results (including screenshots) is stored in this directory.
The second directory is user_scripts. In this directory, you can place your own JavaScript scripts, which you can then embed on pages through the Scrapper API.
For example, to remove ads blocks or click the "Accept Cookies" button (see the user-scripts parameter in the API Reference section for more information).
Scrapper does not work from the root user inside the container. Instead, it uses a user with UID 1001.
Since you will be mounting the user_data and user_scripts directories from the host using Bind Mount, you will need to set write permissions for UID 1001 on these directories on the host.
Here is an example of how to do this:
mkdir -p user_data user_scripts
chown 1001:1001 user_data/ user_scripts/
ls -lThe last command (ls -l) should output a result similar to this:
drwxr-xr-x 2 1001 1001 4096 Mar 17 23:23 user_data
drwxr-xr-x 2 1001 1001 4096 Mar 17 23:23 user_scripts
Over time, the Scrapper cache will grow in size, especially if you are making frequent requests with screenshots.
The scrapper's cache is stored in the user_data/_res directory. You will need to set up automatic clearing of this directory yourself.
For example, you could add the following task to your cron jobs:
find /path/to/user_data/_res -ctime +7 -deleteThis command will use the find utility to locate all files in the cache that were created more than 7 days ago. All such files will be deleted because the find utility accepts the -delete option.
This is just an example of how you might deal with the scrapper's cache growing over time. You can come up with other strategies for this and implement them yourself.
The main thing to remember is where Scrapper stores its cache data - it's in the user_data/_res.
Once the directories have been created and write permissions have been set, you can run Scrapper using the following command:
docker run -d -p 3000:3000 -v $(pwd)/user_data:/home/user/user_data -v $(pwd)/user_scripts:/home/user/user_scripts --name scrapper amerkurev/scrapper:latestThe Scrapper web interface should now be available at http://localhost:3000/. Use any modern browser to access it.
To connect to Scrapper logs, use the following command:
docker logs -f scrapperThe Scrapper API is very simple. Essentially, it is just one call that can easily be demonstrated using the cURL:
curl -X GET "localhost:3000/api/article?url=https://en.wikipedia.org/wiki/web_scraping"Use the GET method on the /api/article endpoint, passing one required parameter url. This is the full URL of the webpage on the Internet that contains an article.
Scrapper will load the webpage in a browser, extract the article text, and return it in JSON format in the response.
All other request parameters are optional and have default values. However, you can customize them to your liking. The table below lists all the parameters that you can use, along with their descriptions and default values. To make it easier to build requests, use the web interface where the final request link is generated in real-time as you configure the parameters.
| Parameter | Description | Default |
|---|---|---|
url |
Page URL. The page should contain the text of the article that needs to be extracted. | |
cache |
All results of the parsing process will be cached in the user_data directory. Cache can be disabled by setting the cache option to false. In this case, the page will be fetched and parsed every time. Cache is enabled by default. |
true |
full-content |
If this option is set to true, the result will have the full HTML contents of the page (fullContent field in the response). |
false |
stealth |
Stealth mode allows you to bypass anti-scraping techniques. It is disabled by default. | false |
screenshot |
If this option is set to true, the result will have the link to the screenshot of the page (screenshot field in the response). Important implementation details: Initially, Scrapper attempts to take a screenshot of the entire scrollable page. If it fails because the image is too large, it will only capture the currently visible viewport. |
false |
user-scripts |
To use your JavaScript scripts on a webpage, put your script files into the user_scripts directory. Then, list the scripts you need in the user-scripts parameter, separating them with commas. These scripts will run after the page loads but before the article parser starts. This means you can use these scripts to do things like remove ad blocks or automatically click the cookie acceptance button. Keep in mind, script names cannot include commas, as they are used for separation.For example, you might pass remove-ads.js, click-cookie-accept-button.js.If you plan to run asynchronous long-running scripts, check user-scripts-timeout parameter. |
|
user-scripts-timeout |
Waits for the given timeout in milliseconds after users scripts injection. For example if you want to navigate through page to specific content, set a longer period (higher value). The default value is 0, which means no sleep. | 0 |
| Parameter | Description | Default |
|---|---|---|
incognito |
Allows creating incognito browser contexts. Incognito browser contexts don't write any browsing data to disk. |
true |
timeout |
Maximum operation time to navigate to the page in milliseconds; defaults to 60000 (60 seconds). Pass 0 to disable the timeout. | 60000 |
wait-until |
When to consider navigation succeeded, defaults to domcontentloaded. Events can be either:load - consider operation to be finished when the load event is fired.domcontentloaded - consider operation to be finished when the DOMContentLoaded event is fired.networkidle - consider operation to be finished when there are no network connections for at least 500 ms.commit - consider operation to be finished when network response is received and the document started loading. |
domcontentloaded |
sleep |
Waits for the given timeout in milliseconds before parsing the article, and after the page has loaded. In many cases, a sleep timeout is not necessary. However, for some websites, it can be quite useful. Other waiting mechanisms, such as waiting for selector visibility, are not currently supported. The default value is 0, which means no sleep. | 0 |
resource |
List of resource types allowed to be loaded on the page. All other resources will not be allowed, and their network requests will be aborted. By default, all resource types are allowed. The following resource types are supported: document, stylesheet, image, media, font, script, texttrack, xhr, fetch, eventsource, websocket, manifest, other. Example: document,stylesheet,fetch. |
|
viewport-width |
The viewport width in pixels. It's better to use the device parameter instead of specifying it explicitly. |
|
viewport-height |
The viewport height in pixels. It's better to use the device parameter instead of specifying it explicitly. |
|
screen-width |
The page width in pixels. Emulates consistent window screen size available inside web page via window.screen. Is only used when the viewport is set. | |
screen-height |
The page height in pixels. | |
device |
Simulates browser behavior for a specific device, such as user agent, screen size, viewport, and whether it has touch enabled. Individual parameters like user-agent, viewport-width, and viewport-height can also be used; in such cases, they will override the device settings.List of available devices. |
iPhone 12 |
scroll-down |
Scroll down the page by a specified number of pixels. This is particularly useful when dealing with lazy-loading pages (pages that are loaded only as you scroll down). This parameter is used in conjunction with the sleep parameter. Make sure to set a positive value for the sleep parameter, otherwise, the scroll function won't work. |
0 |
ignore-https-errors |
Whether to ignore HTTPS errors when sending network requests. The default setting is to ignore HTTPS errors. | true |
user-agent |
Specific user agent. It's better to use the device parameter instead of specifying it explicitly. |
|
locale |
Specify user locale, for example en-GB, de-DE, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting rules. | |
timezone |
Changes the timezone of the context. See ICU's metaZones.txt for a list of supported timezone IDs. | |
http-credentials |
Credentials for HTTP authentication (string containing username and password separated by a colon, e.g. username:password). |
|
extra-http-headers |
Contains additional HTTP headers to be sent with every request. Example: X-API-Key:123456;X-Auth-Token:abcdef. |
| Parameter | Description | Default |
|---|---|---|
proxy-server |
Proxy to be used for all requests. HTTP and SOCKS proxies are supported, for example http://myproxy.com:3128 or socks5://myproxy.com:3128. Short form myproxy.com:3128 is considered an HTTP proxy. | |
proxy-bypass |
Optional comma-separated domains to bypass proxy, for example .com, chromium.org, .domain.com. |
|
proxy-username |
Optional username to use if HTTP proxy requires authentication. | |
proxy-password |
Optional password to use if HTTP proxy requires authentication. |
| Parameter | Description | Default |
|---|---|---|
max-elems-to-parse |
The maximum number of elements to parse. The default value is 0, which means no limit. | 0 |
nb-top-candidates |
The number of top candidates to consider when analysing how tight the competition is among candidates. | 5 |
char-threshold |
The number of characters an article must have in order to return a result. | 500 |
The response to the /api/article request returns a JSON object that contains fields, which are described in the table below.
| Parameter | Description | Type |
|---|---|---|
byline |
author metadata | null or str |
content |
HTML string of processed article content | null or str |
dir |
content direction | null or str |
excerpt |
article description, or short excerpt from the content | null or str |
fullContent |
full HTML contents of the page | null or str |
id |
unique result ID | str |
url |
page URL after redirects, may not match the query URL | str |
domain |
page's registered domain | str |
lang |
content language | null or str |
length |
length of extracted article, in characters | null or int |
date |
date of extracted article in ISO 8601 format | str |
query |
request parameters | object |
meta |
social meta tags (open graph, twitter) | object |
resultUri |
URL of the current result, the data here is always taken from cache | str |
screenshotUri |
URL of the screenshot of the page | null or str |
siteName |
name of the site | null or str |
textContent |
text content of the article, with all the HTML tags removed | null or str |
title |
article title | null or str |
publishedTime |
article publication time | null or str |
If an error (or multiple errors) occurs during the execution of a request, the response structure will be as follows:
{
"detail": [
{
"type": "error_type",
"msg": "some message"
}
]
}Some errors do not have a detailed description in the response to the request. In this case, you should refer to the log of the Docker container to investigate the cause of the error.
To collect links to news articles on the main pages of websites, use a different query on the /api/links endpoint. The query parameters are similar, but the Readability settings are not required for this query because no text is extracted.
Instead, the Link parser is used, which has its own set of parameters. A description of these parameters is provided below.
curl -X GET "localhost:3000/api/links?url=https://www.cnet.com/"| Parameter | Description | Default |
|---|---|---|
text-len-threshold |
The median (middle value) of the link text length in characters. The default value is 40 characters. Hyperlinks must adhere to this criterion to be included in the results. However, this criterion is not a strict threshold value, and some links may ignore it. | 40 |
words-threshold |
The median (middle value) of the number of words in the link text. The default value is 3 words. Hyperlinks must adhere to this criterion to be included in the results. However, this criterion is not a strict threshold value, and some links may ignore it. | 3 |
The response to the /api/links request returns a JSON object that contains fields, which are described in the table below.
| Parameter | Description | Type |
|---|---|---|
fullContent |
full HTML contents of the page | str |
id |
unique result ID | str |
url |
page URL after redirects, may not match the query URL | str |
domain |
page's registered domain | str |
date |
date when the links were collected in ISO 8601 format | str |
query |
request parameters | object |
meta |
social meta tags (open graph, twitter) | object |
resultUri |
URL of the current result, the data here is always taken from cache | str |
screenshotUri |
URL of the screenshot of the page | str |
links |
list of collected links | list |
title |
page title | str |
Enhance the security of your Scrapper deployment with HTTPS and Basic Authentication by integrating Caddy server.
This approach is recommended for instances exposed to the internet and can be configured with minimal effort using Docker Compose.
Caddy handles SSL certificate issuance and renewal through Let's Encrypt and supports Basic Authentication for added security.
To configure Caddy with Scrapper:
-
Customize the Caddyfile: Update
scrapper.localhostto your domain name. For Basic Authentication, generate a secure hashed password withcaddy hash-passwordand update the Caddyfile with this hash.To generate a password hash:
caddy hash-password -plaintext 'your_new_password'Replace
your_new_passwordwith a strong password, then insert the hashed result into the Caddyfile. -
Launch with Docker Compose: With your
docker-compose.ymland edited Caddyfile ready, deploy the services:docker compose up -d
Once deployed, access Scrapper at https://your_domain. You'll be asked for the username and password specified in the Caddyfile.
Caddy automatically renews SSL certificates before they expire, requiring no action from the user. Enjoy uninterrupted HTTPS protection for your Scrapper instance without manual intervention.
- linux/amd64
- linux/arm64
The project is under active development and may have breaking changes till v1 is released.
However, we are trying our best not to break things unless there is a good reason. As of version v0.8.0, Scrapper is considered good enough for real-life usage, and many setups are running it in production.