mirror of https://github.com/lorien/awesome-web-scraping.git synced 2025-01-05 22:53:47 +02:00

List of libraries, tools and APIs for web scraping and data processing.

Go to file

lorien 36748ce07e Merge pull request #177 from hilmanski/patch-2 Add nokolexbor		2024-12-27 12:17:08 +03:00
.gitignore	add html5-parser to python	2020-10-19 21:48:59 +03:00
cli.md	add trafilatura and courlan to cli.md	2024-10-29 17:24:53 +01:00
CONTRIBUTING.md	Update CONTRIBUTING.md	2022-03-16 12:57:23 +03:00
golang.md	Update golang.md	2024-09-13 16:02:47 +05:30
java.md	Update java.md	2018-09-06 16:09:58 +03:00
javascript.md	Replace Apify SDK with Crawlee, its successor	2022-08-17 22:48:10 +02:00
LICENSE
Makefile
manuals.md	Create manuals.md	2023-08-07 13:22:39 +06:00
perl.md	Update perl.md	2018-09-06 16:01:37 +03:00
php.md	Merge pull request #168 from 3xploiton3/patch-3	2024-08-23 10:07:40 +03:00
python.md	add courlan to python.md	2024-10-29 17:26:47 +01:00
README.md	Update README with a link to cli.md	2024-10-15 15:29:22 +03:00
ruby.md	Update description	2024-12-27 16:21:23 +08:00

Awesome Web Scraping

Lists of packages, services and manuals related to web scraping.

Topics

Python - Python packages
PHP - PHP packages
Ruby - Ruby packages
JavaScript - JavaScript packages
Go - Go packages
Command Line Tools - tools with a command line interface
Web Scraping Manuals - list of articles and books teaching web scraping
dhamaniasad / HeadlessBrowsers - list of (almost) all headless web browsers in existence
DNS over HTTPS providers - list of DNS over HTTPs providers
Awesome Pastebins - list of pastebin sites

The list is based initially on some data from these sources awesome-python, awesome-php, awesome-ruby, ruby-nlp, awesome-javascript