mirror of https://github.com/lorien/awesome-web-scraping.git synced 2024-11-28 08:48:58 +02:00

List of libraries, tools and APIs for web scraping and data processing.

Go to file

lorien 7221b9bed5 Add manuals.md link to README.md		2023-08-07 13:25:07 +06:00
.gitignore	add html5-parser to python	2020-10-19 21:48:59 +03:00
console_tools.md	order fix	2022-10-10 21:58:50 +03:00
CONTRIBUTING.md	Update CONTRIBUTING.md	2022-03-16 12:57:23 +03:00
golang.md	fix url to caddy proxy server repository	2022-03-15 00:41:39 +02:00
java.md	Update java.md	2018-09-06 16:09:58 +03:00
javascript.md	Replace Apify SDK with Crawlee, its successor	2022-08-17 22:48:10 +02:00
LICENSE	Initial commit	2015-08-13 02:27:49 +06:00
Makefile	New stuff	2015-08-13 02:49:22 +06:00
manuals.md	Create manuals.md	2023-08-07 13:22:39 +06:00
perl.md	Update perl.md	2018-09-06 16:01:37 +03:00
php.md	fix format	2022-10-08 18:53:12 +03:00
python.md	Add cloudpickle to python list	2023-04-18 00:59:47 +07:00
README.md	Add manuals.md link to README.md	2023-08-07 13:25:07 +06:00
ruby.md	Add arachnid2 to Ruby scrapers	2020-06-25 16:52:15 +01:00

Awesome Web Scraping

The list of tools, programming libraries and web services used for web scraping and data processing.

Feel free to give feedback or ask web scraping questions in Telegram groups: @grablab (English) and @grablab_ru (Russian).

Programming Libraries

Web Scraping Manuals - list of articles and books teaching web scraping
Console tools
dhamaniasad / HeadlessBrowsers - a list of (almost) all headless web browsers in existence
DNS over HTTPS providers - list of DNS over HTTPs providers

These two links point to same captcha service, it is just a different language versions

Largest marketplaces in the world which contain offers from hundreds sellers and services:

The list is based initially on some data from these sources awesome-python, awesome-php, awesome-ruby, ruby-nlp, awesome-javascript