mirror of https://github.com/lorien/awesome-web-scraping.git synced 2024-11-24 08:32:19 +02:00

List of libraries, tools and APIs for web scraping and data processing.

Go to file

Ondra Urban c437605b7b Replace Apify SDK with Crawlee, its successor The crawling part of Apify SDK is now named Crawlee and its new version is out with a bunch of improvements.		2022-08-17 22:48:10 +02:00
.gitignore	add html5-parser to python	2020-10-19 21:48:59 +03:00
console_tools.md	Add item to Other Lists to Console Tools list	2019-11-12 00:55:55 +03:00
CONTRIBUTING.md	Update CONTRIBUTING.md	2022-03-16 12:57:23 +03:00
golang.md	fix url to caddy proxy server repository	2022-03-15 00:41:39 +02:00
java.md	Update java.md	2018-09-06 16:09:58 +03:00
javascript.md	Replace Apify SDK with Crawlee, its successor	2022-08-17 22:48:10 +02:00
LICENSE	Initial commit	2015-08-13 02:27:49 +06:00
Makefile	New stuff	2015-08-13 02:49:22 +06:00
perl.md	Update perl.md	2018-09-06 16:01:37 +03:00
php.md	Add crwlr packages and Symfony DomCrawler	2022-06-02 23:09:46 +02:00
python.md	Fix spaces	2022-06-24 14:47:34 +03:00
README.md	Update README.md	2022-03-02 00:39:52 +03:00
ruby.md	Add arachnid2 to Ruby scrapers	2020-06-25 16:52:15 +01:00
web_services.md	Add Scraping Fish API to web services	2022-06-27 12:30:17 +02:00

🇷🇺 Awesome Web Scraping

The list of tools, programming libraries and web services used in web scraping and data processing.

Web scraping chats: @grablab (English) and @grablab_ru (Russian)

Programming Libraries

Web Scraping Services
Console tools
dhamaniasad / HeadlessBrowsers - a list of (almost) all headless web browsers in existence
DNS over HTTPS providers - list of DNS over HTTPs providers

These two links point to same captcha service, it is just a different language versions

See Contributing document.

The list is based initially on some data from these sources awesome-python, awesome-php, awesome-ruby, ruby-nlp, awesome-javascript