mirror of https://github.com/lorien/awesome-web-scraping.git synced 2024-11-24 08:32:19 +02:00

List of libraries, tools and APIs for web scraping and data processing.

captcha-bypass captcha-recaptcha crawler crawling crawling-framework crawling-python crawling-tool scraping scraping-framework scraping-python scraping-tool spider web-scraping webscraping

Go to file

Gregory Petukhov 3f44997732 Delete proxy_services.md		2020-05-13 00:48:23 +03:00
.gitignore	Refactor markup	2019-07-09 03:04:16 +03:00
console_tools.md	Add item to Other Lists to Console Tools list	2019-11-12 00:55:55 +03:00
CONTRIBUTING.md	New stuff	2015-08-13 02:49:22 +06:00
golang.md	Add geziyor scraper&crawler	2019-09-11 18:54:18 +03:00
java.md	Update java.md	2018-09-06 16:09:58 +03:00
javascript.md	Add web scraping framework pjscrape to javascript.md	2020-03-05 12:20:20 +05:30
LICENSE	Initial commit	2015-08-13 02:27:49 +06:00
Makefile	New stuff	2015-08-13 02:49:22 +06:00
new_language_template.md	Create new_language_template.md	2015-08-21 21:17:28 +05:00
perl.md	Update perl.md	2018-09-06 16:01:37 +03:00
php.md	Replace an abandoned package with the supported one	2019-12-23 12:33:28 +04:00
python.md	Use github links for some of packages in the list	2020-03-27 17:01:52 +03:00
README.md	Update README.md	2020-05-13 00:43:56 +03:00
ruby.md	added kimuraframework	2018-12-21 00:48:14 +03:00
web_services.md	Merge pull request #77 from jancurn/master	2019-01-28 13:58:48 +03:00

README.md

Awesome Web Scraping

The list of tools, programming libraries and web services used in web scraping.

Web scraping chats: @grablab (English) and @grablab_ru (Russian)

Programming Libraries

Other Things

Web Scraping Services
Console tools
dhamaniasad / HeadlessBrowsers - a list of (almost) all headless web browsers in existence
lorien / awesome-python-dev - a list of tools for debugging, profiling and analyzing python programs.

Captcha Solving Services

2captcha.com - 2captcha.com
anti-gate.com - anti-gate.com

Proxy Services

proxyrack.com - Over 1,250,000 Unique Private IPs With Instant Setup
webshare.io - 5000+ HTTP server proxies
proxium.ru - extremely cheap small proxy lists
scrapinghub.com/crawlera - Smart Proxy Network that automatically rotates IPs

About awesome-web-scraping

Make this list better! Your contributions are always welcome! See contributing how-to. To add new language to Programming Libraries section use new_language_template as starting point.

The list is based initially on some data from these sources awesome-python, awesome-php, awesome-ruby, ruby-nlp, awesome-javascript