mirror of https://github.com/lorien/awesome-web-scraping.git synced 2024-11-28 08:48:58 +02:00

List of libraries, tools and APIs for web scraping and data processing.

Go to file

otsch 394e973b96 Add crwlr packages and Symfony DomCrawler		2022-06-02 23:09:46 +02:00
.gitignore	add html5-parser to python	2020-10-19 21:48:59 +03:00
console_tools.md	Add item to Other Lists to Console Tools list	2019-11-12 00:55:55 +03:00
CONTRIBUTING.md	Update CONTRIBUTING.md	2022-03-16 12:57:23 +03:00
golang.md	fix url to caddy proxy server repository	2022-03-15 00:41:39 +02:00
java.md	Update java.md	2018-09-06 16:09:58 +03:00
javascript.md	Added Playwright	2020-09-11 10:54:07 +05:45
LICENSE	Initial commit	2015-08-13 02:27:49 +06:00
Makefile	New stuff	2015-08-13 02:49:22 +06:00
perl.md	Update perl.md	2018-09-06 16:01:37 +03:00
php.md	Add crwlr packages and Symfony DomCrawler	2022-06-02 23:09:46 +02:00
python.md	Add Dude to python web scraping frameworks	2022-03-18 01:55:37 +01:00
README.md	Update README.md	2022-03-02 00:39:52 +03:00
ruby.md	Add arachnid2 to Ruby scrapers	2020-06-25 16:52:15 +01:00
web_services.md	Update web_services.md	2022-03-16 11:39:57 +02:00

🇷🇺 Awesome Web Scraping

The list of tools, programming libraries and web services used in web scraping and data processing.

Web scraping chats: @grablab (English) and @grablab_ru (Russian)

Programming Libraries

Web Scraping Services
Console tools
dhamaniasad / HeadlessBrowsers - a list of (almost) all headless web browsers in existence
DNS over HTTPS providers - list of DNS over HTTPs providers

These two links point to same captcha service, it is just a different language versions

See Contributing document.

The list is based initially on some data from these sources awesome-python, awesome-php, awesome-ruby, ruby-nlp, awesome-javascript