2018-09-18 22:42:38 +02:00
# Ferret
2018-09-27 18:05:52 +02:00
[![Build Status ](https://travis-ci.com/MontFerret/ferret.svg?branch=master )](https://travis-ci.com/MontFerret/ferret)
2018-09-26 00:12:12 +02:00
![ferret ](https://raw.githubusercontent.com/MontFerret/ferret/master/assets/intro.jpg )
2018-09-22 05:48:35 +02:00
## What is it?
2018-10-02 17:58:22 +02:00
```ferret``` is a web scraping system aiming to simplify data extraction from the web for such things like ui testing, machine learning and analytics.
2018-09-23 08:34:26 +02:00
Having it's own declarative language, ```ferret``` abstracts away technical details and complexity of the underlying technologies, helping to focus on the data itself.
2018-09-22 05:48:35 +02:00
It's extremely portable, extensible and fast.
2018-09-19 03:41:16 +02:00
2018-09-28 06:34:24 +02:00
## Show me some code
2018-09-28 19:43:29 +02:00
The following example demonstrates the use of dynamic pages.
First of all, we load the main Google Search page, type search criteria into an input box and then click a search button.
The click action triggers a redirect, so we wait till its end.
Once the page gets loaded, we iterate over all elements in search results and assign output to a variable.
2018-09-28 01:43:33 +02:00
The final for loop filters out empty elements that might be because of inaccurate use of selectors.
2018-09-27 19:37:37 +02:00
```aql
2018-10-02 19:32:22 +02:00
LET google = DOCUMENT("https://www.google.com/", true)
2018-09-28 01:43:33 +02:00
2018-10-02 19:32:22 +02:00
INPUT(google, 'input[name="q"]', "ferret")
CLICK(google, 'input[name="btnK"]')
2018-09-28 01:43:33 +02:00
2018-10-02 19:32:22 +02:00
WAIT_NAVIGATION(google)
2018-09-28 01:43:33 +02:00
LET result = (
2018-10-02 19:32:22 +02:00
FOR result IN ELEMENTS(google, '.g')
2018-09-28 01:43:33 +02:00
RETURN {
title: ELEMENT(result, 'h3 > a'),
description: ELEMENT(result, '.st'),
url: ELEMENT(result, 'cite')
}
)
RETURN (
2018-10-02 19:32:22 +02:00
FOR page IN result
FILTER page.title != NONE
RETURN page
2018-09-28 01:43:33 +02:00
)
2018-09-27 19:37:37 +02:00
```
2018-09-19 03:41:16 +02:00
## Features
* Declarative language
2018-09-22 05:48:35 +02:00
* Support of both static and dynamic web pages
2018-09-19 03:41:16 +02:00
* Embeddable
* Extensible
2018-09-18 22:42:38 +02:00
## Motivation
2018-09-19 03:47:54 +02:00
Nowadays data is everything and who owns data - owns the world.
I have worked on multiple data-driven projects where data was an essential part of a system and I realized how cumbersome writing tons of scrapers is.
After some time looking for a tool that would let me to not write a code, but just express what data I need, decided to come up with my own solution.
2018-09-23 08:34:26 +02:00
```ferret``` project is an ambitious initiative trying to bring universal platform for writing scrapers without any hassle.
2018-09-18 22:42:38 +02:00
## Inspiration
2018-09-19 03:47:54 +02:00
FQL (Ferret Query Language) is heavily inspired by [AQL ](https://www.arangodb.com/ ) (ArangoDB Query Language).
But due to the domain specifics, there are some differences in how things work.
2018-09-18 22:42:38 +02:00
## WIP
2018-09-19 03:47:54 +02:00
Be aware, the the project is under heavy development. There is no documentation and some things may change in the final release.
2018-10-03 00:10:11 +02:00
For query syntax, you may go to [ArangoDB web site ](https://docs.arangodb.com/3.3/AQL/index.html ) and use AQL docs as docs for FQL - since they are identical.
2018-09-18 22:42:38 +02:00
2018-09-22 05:48:35 +02:00
## Installation
### Prerequisites
* Go >=1.6
* GoDep
* GNU Make
2018-09-27 22:07:36 +02:00
* Chrome or Docker (optional)
2018-09-22 05:48:35 +02:00
```sh
2018-10-05 04:16:03 +02:00
go get github.com/MontFerret/ferret
2018-09-22 05:48:35 +02:00
```
2018-09-29 05:49:05 +02:00
You can use your local copy of Google Chrome / Chromium, but for ease of use it's recommended to run it inside a Docker container:
2018-09-29 05:47:40 +02:00
```sh
docker pull alpeware/chrome-headless-trunk
docker run -d -p=0.0.0.0:9222:9222 --name=chrome-headless -v /tmp/chromedata/:/data alpeware/chrome-headless-trunk
```
2018-10-04 22:19:26 +02:00
But if you want to see what's happening during query execution, just start your Chrome with remote debugging port:
```sh
chrome.exe --remote-debugging-port=9222
```
2018-09-19 03:41:16 +02:00
## Quick start
2018-09-18 22:42:38 +02:00
### Browserless mode
2018-09-19 03:41:16 +02:00
If you want to play with ```fql``` and check its syntax, you can run CLI with the following commands:
2018-09-18 22:42:38 +02:00
```
2018-10-05 04:16:03 +02:00
ferret
2018-09-18 22:42:38 +02:00
```
2018-09-19 03:41:16 +02:00
```ferret``` will run in REPL mode.
2018-09-18 22:42:38 +02:00
```shell
Welcome to Ferret REPL
Please use `Ctrl-D` to exit this program.
2018-09-23 08:34:26 +02:00
>%
>LET doc = DOCUMENT('https://news.ycombinator.com/')
>FOR post IN ELEMENTS(doc, '.storylink')
2018-09-18 22:42:38 +02:00
>RETURN post.attributes.href
2018-09-23 08:34:26 +02:00
>%
2018-09-18 22:42:38 +02:00
```
2018-09-23 08:34:26 +02:00
**Note:** symbol ```%``` is used to start and end multi line queries. You also can use heredoc format.
2018-09-18 22:42:38 +02:00
2018-09-19 03:41:16 +02:00
If you want to execute a query stored in a file, just pass a file name:
2018-09-18 22:42:38 +02:00
```
2018-10-05 04:17:33 +02:00
ferret ./docs/examples/static-page.fql
2018-09-18 22:42:38 +02:00
```
2018-09-23 08:34:26 +02:00
```
2018-10-05 04:17:33 +02:00
cat ./docs/examples/static-page.fql | ferret
2018-09-23 08:34:26 +02:00
```
```
2018-10-05 04:17:33 +02:00
ferret < . / docs / examples / static-page . fql
2018-09-23 08:34:26 +02:00
```
2018-09-18 22:42:38 +02:00
### Browser mode
2018-09-19 03:47:54 +02:00
By default, ``ferret`` loads HTML pages via http protocol, because it's faster.
But nowadays, there are more and more websites rendered with JavaScript, and therefore, this 'old school' approach does not really work.
For such cases, you may fetch documents using Chrome or Chromium via Chrome DevTools protocol (aka CDP).
First, you need to make sure that you launched Chrome with ```remote-debugging-port=9222``` flag.
Second, you need to pass the address to ```ferret``` CLI.
2018-09-18 22:42:38 +02:00
2018-09-19 03:41:16 +02:00
```
2018-10-05 04:16:03 +02:00
ferret --cdp http://127.0.0.1:9222
2018-09-18 22:42:38 +02:00
```
2018-09-19 03:47:54 +02:00
**NOTE:** By default, ```ferret``` will try to use this local address as a default one, so it makes sense to explicitly pass the parameter only in case of either different port number or remote address.
2018-09-18 22:42:38 +02:00
2018-09-19 03:41:16 +02:00
Alternatively, you can tell CLI to launch Chrome for you.
2018-09-18 22:42:38 +02:00
2018-09-19 03:41:16 +02:00
```shell
2018-10-05 04:16:03 +02:00
ferret --cdp-launch
2018-09-18 22:42:38 +02:00
```
2018-09-19 03:47:54 +02:00
**NOTE:** Launch command is currently broken on MacOS.
2018-09-19 03:41:16 +02:00
2018-09-23 10:33:20 +02:00
Once ```ferret``` knows how to communicate with Chrome, you can use a function ```DOCUMENT(url, isDynamic)``` with ```true``` boolean value for dynamic pages:
2018-09-18 22:42:38 +02:00
```shell
Welcome to Ferret REPL
Please use `exit` or `Ctrl-D` to exit this program.
2018-09-23 08:34:26 +02:00
>%
2018-09-18 22:42:38 +02:00
>LET doc = DOCUMENT('https://soundcloud.com/charts/top', true)
2018-09-23 10:33:20 +02:00
>WAIT_ELEMENT(doc, '.chartTrack__details', 5000)
2018-09-18 22:42:38 +02:00
>LET tracks = ELEMENTS(doc, '.chartTrack__details')
>FOR track IN tracks
> LET username = ELEMENT(track, '.chartTrack__username')
> LET title = ELEMENT(track, '.chartTrack__title')
> RETURN {
> artist: username.innerText,
> track: title.innerText
> }
2018-09-23 08:34:26 +02:00
>%
2018-09-19 03:41:16 +02:00
```
2018-09-26 00:12:12 +02:00
```shell
Welcome to Ferret REPL
Please use `exit` or `Ctrl-D` to exit this program.
>%
>LET doc = DOCUMENT("https://github.com/", true)
>LET btn = ELEMENT(doc, ".HeaderMenu a")
>CLICK(btn)
>WAIT_NAVIGATION(doc)
>WAIT_ELEMENT(doc, '.IconNav')
>FOR el IN ELEMENTS(doc, '.IconNav a')
> RETURN TRIM(el.innerText)
>%
```
2018-09-19 03:41:16 +02:00
### Embedded mode
2018-09-19 03:47:54 +02:00
```ferret``` is a very modular system and therefore, can be easily be embedded into your Go application.
2018-09-19 03:41:16 +02:00
```go
package main
import (
"context"
"encoding/json"
"fmt"
"github.com/MontFerret/ferret/pkg/compiler"
"os"
)
type Topic struct {
Name string `json:"name"`
Description string `json:"description"`
Url string `json:"url"`
}
func main() {
topics, err := getTopTenTrendingTopics()
if err != nil {
fmt.Println(err)
os.Exit(1)
}
for _, topic := range topics {
fmt.Println(fmt.Sprintf("%s: %s %s", topic.Name, topic.Description, topic.Url))
}
}
func getTopTenTrendingTopics() ([]*Topic, error) {
query := `
LET doc = DOCUMENT("https://github.com/topics")
FOR el IN ELEMENTS(doc, ".py-4.border-bottom")
LIMIT 10
LET url = ELEMENT(el, "a")
LET name = ELEMENT(el, ".f3")
LET desc = ELEMENT(el, ".f5")
RETURN {
name: TRIM(name.innerText),
description: TRIM(desc.innerText),
url: "https://github.com" + url.attributes.href
}
`
comp := compiler.New()
program, err := comp.Compile(query)
if err != nil {
return nil, err
}
out, err := program.Run(context.Background())
if err != nil {
return nil, err
}
res := make([]*Topic, 0, 10)
err = json.Unmarshal(out, & res)
if err != nil {
return nil, err
}
return res, nil
}
```
## Extensibility
2018-09-19 03:47:54 +02:00
That said, ```ferret``` is a very modular system which also allows not only embed it, but extend its standard library.
2018-09-19 03:41:16 +02:00
2018-09-19 03:49:26 +02:00
```go
2018-09-19 03:41:16 +02:00
package main
import (
"context"
"encoding/json"
"fmt"
"github.com/MontFerret/ferret/pkg/compiler"
"github.com/MontFerret/ferret/pkg/runtime/core"
"github.com/MontFerret/ferret/pkg/runtime/values"
"os"
)
func main() {
strs, err := getStrings()
if err != nil {
fmt.Println(err)
os.Exit(1)
}
for _, str := range strs {
fmt.Println(str)
}
}
func getStrings() ([]string, error) {
// function implements is a type of a function that ferret supports as a runtime function
transform := func(ctx context.Context, args ...core.Value) (core.Value, error) {
// it's just a helper function which helps to validate a number of passed args
err := core.ValidateArgs(args, 1)
if err != nil {
// it's recommended to return built-in None type, instead of nil
return values.None, err
}
// this is another helper functions allowing to do type validation
err = core.ValidateType(args[0], core.StringType)
if err != nil {
return values.None, err
}
// cast to built-in string type
str := args[0].(values.String)
return str.Concat(values.NewString("_ferret")).ToUpper(), nil
}
query := `
FOR el IN ["foo", "bar", "qaz"]
// conventionally all functions are registered in upper case
RETURN TRANSFORM(el)
`
comp := compiler.New()
comp.RegisterFunction("transform", transform)
program, err := comp.Compile(query)
if err != nil {
return nil, err
}
out, err := program.Run(context.Background())
if err != nil {
return nil, err
}
res := make([]string, 0, 3)
err = json.Unmarshal(out, & res)
if err != nil {
return nil, err
}
return res, nil
}
```
On top of that, you can completely turn off standard library, by passing the following option:
```go
comp := compiler.New(compiler.WithoutStdlib())
```
2018-09-19 03:47:54 +02:00
And after that, you can easily provide your own implementation of functions from standard library.
2018-09-19 03:41:16 +02:00
2018-09-19 03:47:54 +02:00
If you don't need a particular set of functions from standard library, you can turn off the entire ```stdlib``` and register separate packages from that:
2018-09-19 03:41:16 +02:00
```go
package main
import (
"github.com/MontFerret/ferret/pkg/compiler"
"github.com/MontFerret/ferret/pkg/stdlib/strings"
)
func main() {
comp := compiler.New(compiler.WithoutStdlib())
comp.RegisterFunctions(strings.NewLib())
}
```