mirror of https://github.com/lorien/awesome-web-scraping.git synced 2025-02-15 13:33:11 +02:00

Go to file

Gregory Petukhov 4dc74fc216

Merge pull request #105 from samnissen/add-ruby-arachnid2

Add arachnid2 to Ruby scrapers

2020-06-25 19:46:31 +03:00

.gitignore

Refactor markup

2019-07-09 03:04:16 +03:00

console_tools.md

Add item to Other Lists to Console Tools list

2019-11-12 00:55:55 +03:00

CONTRIBUTING.md

New stuff

2015-08-13 02:49:22 +06:00

golang.md

Add geziyor scraper&crawler

2019-09-11 18:54:18 +03:00

java.md

Update java.md

2018-09-06 16:09:58 +03:00

javascript.md

Add web scraping framework pjscrape to javascript.md

2020-03-05 12:20:20 +05:30

LICENSE

Initial commit

2015-08-13 02:27:49 +06:00

Makefile

New stuff

2015-08-13 02:49:22 +06:00

new_language_template.md

Create new_language_template.md

2015-08-21 21:17:28 +05:00

perl.md

Update perl.md

2018-09-06 16:01:37 +03:00

php.md

Adding new libraries

2020-06-09 13:39:09 +04:00

python.md

Use github links for some of packages in the list

2020-03-27 17:01:52 +03:00

README.md

Update README.md

2020-05-13 00:43:56 +03:00

ruby.md

Add arachnid2 to Ruby scrapers

2020-06-25 16:52:15 +01:00

web_services.md

Merge pull request #77 from jancurn/master

2019-01-28 13:58:48 +03:00

README.md

Awesome Web Scraping

The list of tools, programming libraries and web services used in web scraping.

Web scraping chats: @grablab (English) and @grablab_ru (Russian)

Programming Libraries

Other Things

Web Scraping Services
Console tools
dhamaniasad / HeadlessBrowsers - a list of (almost) all headless web browsers in existence
lorien / awesome-python-dev - a list of tools for debugging, profiling and analyzing python programs.

Captcha Solving Services

2captcha.com - 2captcha.com
anti-gate.com - anti-gate.com

Proxy Services

proxyrack.com - Over 1,250,000 Unique Private IPs With Instant Setup
webshare.io - 5000+ HTTP server proxies
proxium.ru - extremely cheap small proxy lists
scrapinghub.com/crawlera - Smart Proxy Network that automatically rotates IPs

About awesome-web-scraping

Make this list better! Your contributions are always welcome! See contributing how-to. To add new language to Programming Libraries section use new_language_template as starting point.

The list is based initially on some data from these sources awesome-python, awesome-php, awesome-ruby, ruby-nlp, awesome-javascript