1
0
mirror of https://github.com/lorien/awesome-web-scraping.git synced 2024-11-24 08:32:19 +02:00
awesome-web-scraping/console_tools.md

28 lines
1.9 KiB
Markdown
Raw Normal View History

2015-08-13 15:08:38 +02:00
# Console Tools
2015-08-13 15:53:52 +02:00
Console tools related to web scraping and data processing
2015-08-13 15:08:38 +02:00
2015-08-14 22:15:19 +02:00
* [Console Tools](#console-tools)
* [HTTP Clients](#http-clients)
* [Specific Formats Processing](#specific-formats-processing)
## HTTP Clients
* [curl](https://github.com/bagder/curl) - supports SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, proxies, HTTP/2, cookies, user+password authentication (Basic, Plain, Digest, CRAM-MD5, NTLM, Negotiate and Kerberos), file transfer resume, proxy tunneling and more.
* [httpie](https://github.com/jkbrzt/httpie) - sending arbitrary HTTP requests using a simple and natural syntax, and displays colorized output. HTTPie can be used for testing, debugging, and generally interacting with HTTP servers.
* [wget](http://www.gnu.org/software/wget/) - package for retrieving files using HTTP, HTTPS and FTP, the most widely-used Internet protocols. It is a non-interactive commandline tool, so it may easily be called from scripts, cron jobs, terminals without X-Windows support, etc.
2015-08-13 15:08:38 +02:00
## Specific Formats Processing
2015-08-13 15:18:37 +02:00
* Office
2019-06-21 00:03:26 +02:00
* [unoconv](https://github.com/dagwieers/unoconv) - convert between any document format supported by LibreOffice/OpenOffice.
2015-08-13 15:08:38 +02:00
* CSV
2019-06-21 00:03:26 +02:00
* [csvkit](https://github.com/onyxfish/csvkit) - utilities for converting to and working with CSV.
## Proxy Wrappers
* [proxychains](https://github.com/haad/proxychains) - a tool that forces any TCP connection made by any given application to follow through proxy
* [proxychains-ng](https://github.com/rofl0r/proxychains-ng) - a preloader which hooks calls to sockets in dynamically linked programs and redirects it through one or more socks/http proxies. continuation of the unmaintained proxychains project
## Other Lists
* [structured-text-tools](https://github.com/dbohdan/structured-text-tools) - A list of command line tools for manipulating structured text data