Merge pull request #170 from kellyjonbrazil/dev

jc v1.17.0 from dev

CHANGELOG

@@ -1,5 +1,42 @@
 jc changelog
 
+20210923 v1.17.0
+- Note to Package Maintainers: please see note at 20210720 v1.16.0
+- Add wrapping of warning and error messages
+- Add vmstat parser tested on linux
+- Add support for streaming parsers
+- Add ls command streaming parser tested on linux, macOS, and freeBSD
+- Add ping command streaming parser tested on linux, macOS, and freeBSD
+- Add vmstat command streaming parser tested on linux
+- Add -u option to allow unbuffered output
+
+20210830 v1.16.2
+- Note to Package Maintainers: please see note at 20210720 v1.16.0
+- Update sfdisk parser to support the -F option and newer versions of sfdisk
+
+20210813 v1.16.1
+- Note to Package Maintainers: please see note at 20210720 v1.16.0
+- Fix issue with process substitution with the magic syntax
+- Fix issue with globs not including filenames with spaces with magic syntax
+- Fix stat parser to properly handle filenames with spaces on macOS/BSD
+
+20210720 v1.16.0
+- Note to Package Maintainers:
+    TL;DR: `/man/jc.1.gz` and `/jc/man/jc.1.gz` are deprecated and only `/man/jc.1` should be used.
+    
+    The Man page in the PyPi source packages will be moving from `/jc/man/jc.1.gz` to `/man/jc.1`
+    in version 1.17.0. For now the Man pages will be available in both locations, but be aware that
+    the Man page at `/jc/man/jc.1.gz` is now considered deprecated.
+
+    Also, starting in v1.17.0, the Man page will no longer be compressed in the source package,
+    therefore `/man/jc.1.gz` should also be considered deprecated and will no longer be available
+    after v1.17.0. Please use `/man/jc.1` and compress downstream if you would like.
+- Include CHANGELOG in source distribution
+- Fix Man page location in source packages
+- Add sfdisk command parser tested on linux
+- Update unit test files to change the timezone when needed (POSIX only)
+- Binaries and DEB/RPM/MSI packages now include Python 3.9.5 interpreter
+
 20210628 v1.15.6
 - Fix issue to only load local plugin parsers that have filenames that end in .py 
 

CONTRIBUTING.md

@@ -15,11 +15,12 @@ Pull requests are the best way to propose changes to the codebase (we use [Githu
 
 1. Open an issue to discuss the new feature, bug fix, or parser before opening a pull request. For new parsers, it is important to agree upon a schema before developing the parser.
 2. Fork the repo and create your branch from `dev`, if available, otherwise `master`.
-3. If you've added code that should be tested, add tests. All new parsers should have several sample outputs and tests.
-4. Documentation is auto-generated from docstrings, so ensure they are clear and accurate.
-5. Ensure the test suite passes. (Note: "**America/Los_Angeles**" timezone should be configured on the test system)
-6. Make sure your code lints.
-7. Issue that pull request!
+3. For new parsers: Use the [`jc/parsers/foo.py`](https://github.com/kellyjonbrazil/jc/blob/master/jc/parsers/foo.py) or [`jc/parsers/foo_s.py (streaming)`](https://github.com/kellyjonbrazil/jc/blob/master/jc/parsers/foo_s.py) parsers as a template to get started. You can even place a new parser python module file in the [parser plugin directory](https://github.com/kellyjonbrazil/jc#custom-parsers) to get started right away with just a standard `jc` installation.
+4. If you've added code that should be tested, add tests. All new parsers should have several sample outputs and tests.
+5. Documentation is auto-generated from docstrings, so ensure they are clear and accurate.
+6. Ensure the test suite passes. (Note: "**America/Los_Angeles**" timezone should be configured on the test system)
+7. Make sure your code lints.
+8. Issue that pull request!
 
 ## Parser Schema Guidelines
 - Try to keep the schema as flat as possible - typically a list of flat dictionaries
@@ -31,31 +32,31 @@ This will make it easier to use tools like `jq` without requiring escaping of sp
 **Examples**
 
 Bad:
-```
+```json
 {
   "Interface 1": [
-    192.168.1.1,
-    172.16.1.1
+    "192.168.1.1",
+    "172.16.1.1"
   ],
   "Wifi Interface 1": [
-    10.1.1.1
+    "10.1.1.1"
   ]
 }
 ```
 Good:
-```
+```json
 [
   {
     "interface": "Interface 1",
     "ip_addresses": [
-      192.168.1.1,
-      172.16.1.1
+      "192.168.1.1",
+      "172.16.1.1"
     ]
   },
   {
     "interface": "Wifi Interface 1",
     "ip_addresses": [
-      10.1.1.1
+      "10.1.1.1"
     ]
   }
 ]

EXAMPLES.md

@@ -2556,6 +2556,75 @@ rpm_qia | jc --rpm_qi -p          # or:  jc -p rpm -qia
   }
 ]
 ```
+### sfdisk
+```bash
+sfdisk -l | jc --sfdisk -p          # or jc -p sfdisk -l
+```
+```json
+[
+  {
+    "disk": "/dev/sda",
+    "cylinders": 2610,
+    "heads": 255,
+    "sectors_per_track": 63,
+    "units": "cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0",
+    "partitions": [
+      {
+        "device": "/dev/sda1",
+        "boot": true,
+        "start": 0,
+        "end": 130,
+        "cyls": 131,
+        "blocks": 1048576,
+        "id": "83",
+        "system": "Linux"
+      },
+      {
+        "device": "/dev/sda2",
+        "boot": false,
+        "start": 130,
+        "end": 2610,
+        "cyls": 2481,
+        "blocks": 19921920,
+        "id": "8e",
+        "system": "Linux LVM"
+      },
+      {
+        "device": "/dev/sda3",
+        "boot": false,
+        "start": 0,
+        "end": null,
+        "cyls": 0,
+        "blocks": 0,
+        "id": "0",
+        "system": "Empty"
+      },
+      {
+        "device": "/dev/sda4",
+        "boot": false,
+        "start": 0,
+        "end": null,
+        "cyls": 0,
+        "blocks": 0,
+        "id": "0",
+        "system": "Empty"
+      }
+    ]
+  },
+  {
+    "disk": "/dev/mapper/centos-root",
+    "cylinders": 2218,
+    "heads": 255,
+    "sectors_per_track": 63
+  },
+  {
+    "disk": "/dev/mapper/centos-swap",
+    "cylinders": 261,
+    "heads": 255,
+    "sectors_per_track": 63
+  }
+]
+```
 ### /etc/shadow file
 ```bash
 cat /etc/shadow | jc --shadow -p

MANIFEST.in

@@ -1 +1,3 @@
 include jc/man/jc.1.gz
+include man/jc.1
+include CHANGELOG

README.md

@@ -1,7 +1,7 @@
 ![Tests](https://github.com/kellyjonbrazil/jc/workflows/Tests/badge.svg?branch=master)
 ![Pypi](https://img.shields.io/pypi/v/jc.svg)
 
-> Try the new `jc` [web demo](https://jc-web-demo.herokuapp.com/)!
+> Try the `jc` [web demo](https://jc-web-demo.herokuapp.com/)
 
 > JC is [now available](https://galaxy.ansible.com/community/general) as an Ansible filter plugin in the `community.general` collection! See this [blog post](https://blog.kellybrazil.com/2020/08/30/parsing-command-output-in-ansible-with-jc/) for an example.
 
@@ -31,28 +31,13 @@ $ jc dig example.com | jq -r '.[].answer[].data'
 ```
 The `jc` parsers can also be used as python modules. In this case the output will be a python dictionary, or list of dictionaries, instead of JSON:
 ```python
+>>> import subprocess
 >>> import jc.parsers.dig
 >>> 
->>> data = '''; <<>> DiG 9.10.6 <<>> example.com
-... ;; global options: +cmd
-... ;; Got answer:
-... ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64612
-... ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
-... 
-... ;; OPT PSEUDOSECTION:
-... ; EDNS: version: 0, flags:; udp: 4096
-... ;; QUESTION SECTION:
-... ;example.com.           IN  A
-... 
-... ;; ANSWER SECTION:
-... example.com.        29658   IN  A   93.184.216.34
-... 
-... ;; Query time: 52 msec
-... ;; SERVER: 2600:1700:bab0:d40::1#53(2600:1700:bab0:d40::1)
-... ;; WHEN: Fri Apr 16 16:13:00 PDT 2021
-... ;; MSG SIZE  rcvd: 56'''
+>>> cmd_output = subprocess.check_output(['dig', 'example.com'], text=True)
+>>> data = jc.parsers.dig.parse(cmd_output)
 >>>
->>> jc.parsers.dig.parse(data)
+>>> data
 [{'id': 64612, 'opcode': 'QUERY', 'status': 'NOERROR', 'flags': ['qr', 'rd', 'ra'], 'query_num': 1, 'answer_num':
 1, 'authority_num': 0, 'additional_num': 1, 'opt_pseudosection': {'edns': {'version': 0, 'flags': [], 'udp':
 4096}}, 'question': {'name': 'example.com.', 'class': 'IN', 'type': 'A'}, 'answer': [{'name': 'example.com.',
@@ -75,6 +60,8 @@ See also:
 - [libxo on FreeBSD](http://juniper.github.io/libxo/libxo-manual.html)
 - [powershell](https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/convertto-json?view=powershell-7)
 - [blog: linux apps should have a json flag](https://thomashunter.name/posts/2012-06-06-linux-cli-apps-should-have-a-json-flag)
+- [Hacker News discussion](https://news.ycombinator.com/item?id=28266193)
+- [Reddit discussion](https://www.reddit.com/r/programming/comments/pa4cbb/bringing_the_unix_philosophy_to_the_21st_century/)
 
 Use Cases:
 - [Bash scripting](https://blog.kellybrazil.com/2021/04/12/practical-json-at-the-command-line/)
@@ -83,7 +70,7 @@ Use Cases:
 - [Nornir command output parsing](https://blog.kellybrazil.com/2020/12/09/parsing-command-output-in-nornir-with-jc/)
 
 ## Installation
-There are several ways to get `jc`. You can install via `pip`; other OS package repositories like `apt-get`, `dnf`, `zypper`, `pacman`, `nix-env`, `guix`, `brew`, or `portsnap`; via DEB, RPM, and MSI packaged binaries for linux and Windows; or by downloading the correct binary for your architecture and running it anywhere on your filesystem.
+There are several ways to get `jc`. You can install via `pip`, OS package repositories, via DEB/RPM/MSI packaged binaries for linux and Windows, or by downloading the correct binary for your architecture and running it anywhere on your filesystem.
 
 ### Pip (macOS, linux, unix, Windows)
 ```bash
@@ -100,7 +87,7 @@ pip3 install jc
 | Arch linux            | `pacman -S jc`                                                                |
 | NixOS linux           | `nix-env -iA nixpkgs.jc` or `nix-env -iA nixos.jc`                            |
 | Guix System linux     | `guix install jc`                                                             |
-| MacOS                 | `brew install jc`                                                             |
+| macOS                 | `brew install jc`                                                             |
 | FreeBSD               | `portsnap fetch update && cd /usr/ports/textproc/py-jc && make install clean` |
 | Ansible filter plugin | `ansible-galaxy collection install community.general`                         |
 
@@ -156,6 +143,7 @@ The JSON output can be compact (default) or pretty formatted with the `-p` optio
 - `--kv` enables the Key/Value file parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/kv))
 - `--last` enables the `last` and `lastb` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/last))
 - `--ls` enables the `ls` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ls))
+- `--ls-s` enables the `ls` command streaming parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ls_s))
 - `--lsblk` enables the `lsblk` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/lsblk))
 - `--lsmod` enables the `lsmod` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/lsmod))
 - `--lsof` enables the `lsof` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/lsof))
@@ -164,11 +152,13 @@ The JSON output can be compact (default) or pretty formatted with the `-p` optio
 - `--ntpq` enables the `ntpq -p` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ntpq))
 - `--passwd` enables the `/etc/passwd` file parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/passwd))
 - `--ping` enables the `ping` and `ping6` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ping))
+- `--ping-s` enables the `ping` and `ping6` command streaming parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ping_s))
 - `--pip-list` enables the `pip list` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/pip_list))
 - `--pip-show` enables the `pip show` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/pip_show))
 - `--ps` enables the `ps` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ps))
 - `--route` enables the `route` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/route))
 - `--rpm-qi` enables the `rpm -qi` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/rpm_qi))
+- `--sfdisk` enables the `sfdisk` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/sfdisk))
 - `--shadow` enables the `/etc/shadow` file parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/shadow))
 - `--ss` enables the `ss` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ss))
 - `--stat` enables the `stat` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/stat))
@@ -187,6 +177,8 @@ The JSON output can be compact (default) or pretty formatted with the `-p` optio
 - `--uname` enables the `uname -a` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/uname))
 - `--upower` enables the `upower` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/upower))
 - `--uptime` enables the `uptime` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/uptime))
+- `--vmstat` enables the `vmstat` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/vmstat))
+- `--vmstat-s` enables the `vmstat` command streaming parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/vmstat_s))
 - `--w` enables the `w` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/w))
 - `--wc` enables the `wc` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/wc))
 - `--who` enables the `who` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/who))
@@ -196,11 +188,12 @@ The JSON output can be compact (default) or pretty formatted with the `-p` optio
 ### Options
 - `-a` about `jc`. Prints information about `jc` and the parsers (in JSON, of course!)
 - `-d` debug mode. Prints trace messages if parsing issues are encountered (use `-dd` for verbose debugging)
-- `-h` `jc` help. Use `jc -h --parser_name` for parser documentation
+- `-h` help. Use `jc -h --parser_name` for parser documentation
 - `-m` monochrome JSON output
 - `-p` pretty format the JSON output
-- `-q` quiet mode. Suppresses parser warning messages
+- `-q` quiet mode. Suppresses parser warning messages (use `-qq` to ignore streaming parser errors)
 - `-r` raw output. Provides a more literal JSON output, typically with string values and no additional semantic processing
+- `-u` unbuffer output
 - `-v` version information
 
 ### Exit Codes
@@ -232,6 +225,64 @@ or
 JC_COLORS=default,default,default,default
 ```
 
+### Streaming Parsers
+Most parsers load all of the data from STDIN, parse it, then output the entire JSON document serially. There are some streaming parsers (e.g. `ls-s` and `ping-s`) that immediately start processing and outputing the data line-by-line as [JSON Lines](https://jsonlines.org/) (aka [NDJSON](http://ndjson.org/)) while it is being received from STDIN. This can significantly reduce the amount of memory required to parse large amounts of command output (e.g. `ls -lR /`) and can sometimes process the data more quickly. Streaming parsers have slightly different behavior than standard parsers as outlined below.
+
+> Note: Streaming parsers cannot be used with the "magic" syntax
+
+#### Ignoring Errors
+
+You may want to ignore parsing errors when using streaming parsers since these may be used in long-lived processing pipelines and errors can break the pipe. To ignore parsing errors, use the `-qq` cli option or the `ignore_exceptions=True` argument with the `parse()` function. This will add a `_jc_meta` object to the JSON output with a `success` attribute. If `success` is `true`, then there were no issues parsing the line. If `success` is `false`, then a parsing issue was found and `error` and `line` fields will be added to include a short error description and the contents of the unparsable line, respectively:
+
+Successfully parsed line with `-qq` option:
+```json
+{
+  "command_data": "data",
+  "_jc_meta": {
+    "success": true
+  }
+}
+```
+Unsuccessfully parsed line with `-qq` option:
+```json
+{
+  "_jc_meta": {
+    "success": false,
+    "error": "error message",
+    "line": "original line data"
+  }
+}
+```
+
+#### Unbuffering Output
+
+Most operating systems will buffer output that is being piped from process to process. The buffer is usually around 4KB. When viewing the output in the terminal the OS buffer is not engaged so output is immediately displayed on the screen. When piping multiple processes together, though, it may seem as if the output is hanging when the input data is very slow (e.g. `ping`):
+```
+$ ping 1.1.1.1 | jc --ping-s | jq
+<slow output>
+```
+This is because the OS engages the 4KB buffer between `jc` and `jq` in this example. To display the data on the terminal in realtime, you can disable the buffer with the `-u` (unbuffer) cli option:
+```
+$ ping 1.1.1.1 | jc --ping-s -u | jq
+{"type":"reply","pattern":null,"timestamp":null,"bytes":"64","response_ip":"1.1.1.1","icmp_seq":"1","ttl":"128","time_ms":"24.6","duplicate":false}
+{"type":"reply","pattern":null,"timestamp":null,"bytes":"64","response_ip":"1.1.1.1","icmp_seq":"2","ttl":"128","time_ms":"26.8","duplicate":false}
+...
+```
+> Note: Unbuffered output can be slower for large data streams.
+
+#### Using Streaming Parsers as Python Modules
+
+Streaming parsers accept any iterable object and return a generator iterator object allowing lazy processing of the data. The input data should iterate on lines of string data. Examples of good input data are `sys.stdin` or `str.splitlines()`.
+
+To use the generator object in your code, simply loop through it or use the [next()](https://docs.python.org/3/library/functions.html#next) builtin function:
+```python
+import jc.parsers.ls_s
+
+result = jc.parsers.ls_s.parse(ls_command_output.splitlines())
+for item in result:
+    print(item["filename"])
+```
+
 ### Custom Parsers
 Custom local parser plugins may be placed in a `jc/jcparsers` folder in your local **"App data directory"**:
 
@@ -246,7 +297,8 @@ Local plugin filenames must be valid python module names, therefore must consist
 > Note: The application data directory follows the [XDG Base Directory Specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html)
 
 ### Caveats
-**Locale:**
+
+#### Locale
 
 For best results set the `LANG` locale environment variable to `C` or `en_US.UTF-8`. For example, either by setting directly on the command-line:
 ```
@@ -257,7 +309,7 @@ or by exporting to the environment before running commands:
 $ export LANG=C
 ```
 
-**Timezones:**
+#### Timezones
 
 Some parsers have calculated epoch timestamp fields added to the output. Unless a timestamp field name has a `_utc` suffix it is considered naive. (i.e. based on the local timezone of the system the `jc` parser was run on).
 
@@ -291,7 +343,7 @@ Tested on:
 - Windows 2019 Server
 
 ## Contributions
-Feel free to add/improve code or parsers! You can use the [`jc/parsers/foo.py`](https://github.com/kellyjonbrazil/jc/blob/master/jc/parsers/foo.py) parser as a template and submit your parser with a pull request.
+Feel free to add/improve code or parsers! You can use the [`jc/parsers/foo.py`](https://github.com/kellyjonbrazil/jc/blob/master/jc/parsers/foo.py) or [`jc/parsers/foo_s.py (streaming)`](https://github.com/kellyjonbrazil/jc/blob/master/jc/parsers/foo_s.py) parsers as a template and submit your parser with a pull request.
 
 Please see the [Contributing Guidelines](https://github.com/kellyjonbrazil/jc/blob/master/CONTRIBUTING.md) for more information.
 

docs/parsers/kv.md

@@ -38,7 +38,7 @@ Examples:
     # but can be preserved with the -r argument
     occupation:"Engineer"
 
-    $ cat keyvalue.txt | jc --ini -p
+    $ cat keyvalue.txt | jc --kv -p
     {
       "name": "John Doe",
       "address": "555 California Drive",

docs/parsers/ls_s.md

@@ -0,0 +1,92 @@
+[Home](https://kellyjonbrazil.github.io/jc/)
+
+# jc.parsers.ls_s
+jc - JSON CLI output utility `ls` and `vdir` command output streaming parser
+
+This streaming parser requires the `-l` option to be used on `ls`. If there are newline characters in the filename, then make sure to use the `-b` option on `ls`.
+
+The `jc` `-q` option can be used to ignore parsing errors. (e.g. filenames with newline characters, but `-b` was not used)
+
+The `epoch` calculated timestamp field is naive (i.e. based on the local time of the system the parser is run on)
+
+The `epoch_utc` calculated timestamp field is timezone-aware and is only available if the timezone field is UTC.
+
+Usage (cli):
+
+    $ ls | jc --ls-s
+
+Usage (module):
+
+    import jc.parsers.ls_s
+    result = jc.parsers.ls_s.parse(ls_command_output.splitlines())    # result is an iterable object
+    for item in result:
+        # do something
+
+Schema:
+
+    {
+      "filename":       string,
+      "flags":          string,
+      "links":          integer,
+      "parent":         string,
+      "owner":          string,
+      "group":          string,
+      "size":           integer,
+      "date":           string,
+      "epoch":          integer,     # naive timestamp if date field exists and can be converted
+      "epoch_utc":      integer,     # timezone aware timestamp if date field is in UTC and can be converted
+      "_jc_meta":                    # This object only exists if using -qq or ignore_exceptions=True
+        {
+          "success":    booean,      # true if successfully parsed, false if error
+          "error":      string,      # exists if "success" is false
+          "line":       string       # exists if "success" is false
+        }
+    }
+
+Examples:
+
+    $ ls -l /usr/bin | jc --ls-s
+    {"filename":"2to3-","flags":"-rwxr-xr-x","links":4,"owner":"root","group":"wheel","size":925,"date":"Feb 22 2019"}
+    {"filename":"2to3-2.7","link_to":"../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/2to3-2.7","flags":"lrwxr-xr-x","links":1,"owner":"root","group":"wheel","size":74,"date":"May 4 2019"}
+    {"filename":"AssetCacheLocatorUtil","flags":"-rwxr-xr-x","links":1,"owner":"root","group":"wheel","size":55152,"date":"May 3 2019"}
+    ...
+
+    $ ls -l /usr/bin | jc --ls-s -r
+    {"filename":"2to3-","flags":"-rwxr-xr-x","links":"4","owner":"root","group":"wheel","size":"925","date":"Feb 22 2019"}
+    {"filename":"2to3-2.7","link_to":"../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/2to3-2.7","flags":"lrwxr-xr-x","links":"1","owner":"root","group":"wheel","size":"74","date":"May 4 2019"}
+    {"filename":"AssetCacheLocatorUtil","flags":"-rwxr-xr-x","links":"1","owner":"root","group":"wheel","size":"55152","date":"May 3 2019"}
+    ...
+
+
+## info
+```python
+info()
+```
+Provides parser metadata (version, author, etc.)
+
+## parse
+```python
+parse(data, raw=False, quiet=False, ignore_exceptions=False)
+```
+
+Main text parsing generator function. Returns an iterator object.
+
+Parameters:
+
+    data:              (iterable)  line-based text data to parse (e.g. sys.stdin or str.splitlines())
+    raw:               (boolean)   output preprocessed JSON if True
+    quiet:             (boolean)   suppress warning messages if True
+    ignore_exceptions: (boolean)   ignore parsing exceptions if True
+
+Yields:
+
+    Dictionary. Raw or processed structured data.
+
+Returns:
+
+    Iterator object
+
+## Parser Information
+Compatibility:  linux, darwin, cygwin, aix, freebsd
+
+Version 0.5 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/ping_s.md

@@ -0,0 +1,99 @@
+[Home](https://kellyjonbrazil.github.io/jc/)
+
+# jc.parsers.ping_s
+jc - JSON CLI output utility `ping` command output streaming parser
+
+Supports `ping` and `ping6` output.
+
+Usage (cli):
+
+    $ ping | jc --ping-s
+
+> Note: When piping `jc` converted ping output to other processes it may appear the output is hanging due to the OS pipe buffers. This is because `ping` output is too small to quickly fill up the buffer. Use the `-u` option to unbuffer the `jc` output if you would like immediate output. See the [readme](https://github.com/kellyjonbrazil/jc/tree/streaming#streaming-parsers) for more information.
+
+Usage (module):
+
+    import jc.parsers.ping_s
+    result = jc.parsers.ping_s.parse(ping_command_output.splitlines())    # result is an iterable object
+    for item in result:
+        # do something
+
+Schema:
+
+    {
+      "type":                        string,        # 'reply', 'timeout', 'summary', etc. See `_error_type.type_map` for all options.
+      "source_ip":                   string,
+      "destination_ip":              string,
+      "sent_bytes":                  integer,
+      "pattern":                     string,        # (null if not set)
+      "destination":                 string,
+      "timestamp":                   float,
+      "response_bytes":              integer,
+      "response_ip":                 string,
+      "icmp_seq":                    integer,
+      "ttl":                         integer,
+      "time_ms":                     float,
+      "duplicate":                   boolean,
+      "packets_transmitted":         integer,
+      "packets_received":            integer,
+      "packet_loss_percent":         float,
+      "duplicates":                  integer,
+      "round_trip_ms_min":           float,
+      "round_trip_ms_avg":           float,
+      "round_trip_ms_max":           float,
+      "round_trip_ms_stddev":        float,
+      "_jc_meta":                                  # This object only exists if using -qq or ignore_exceptions=True
+        {
+          "success":                 booean,       # true if successfully parsed, false if error
+          "error":                   string,       # exists if "success" is false
+          "line":                    string        # exists if "success" is false
+        }
+    }
+
+Examples:
+
+    $ ping 1.1.1.1 | jc --ping-s
+    {"type":"reply","destination_ip":"1.1.1.1","sent_bytes":56,"pattern":null,"response_bytes":64,"response_ip":"1.1.1.1","icmp_seq":0,"ttl":56,"time_ms":23.703}
+    {"type":"reply","destination_ip":"1.1.1.1","sent_bytes":56,"pattern":null,"response_bytes":64,"response_ip":"1.1.1.1","icmp_seq":1,"ttl":56,"time_ms":22.862}
+    {"type":"reply","destination_ip":"1.1.1.1","sent_bytes":56,"pattern":null,"response_bytes":64,"response_ip":"1.1.1.1","icmp_seq":2,"ttl":56,"time_ms":22.82}
+    ...
+
+    $ ping 1.1.1.1 | jc --ping-s -r
+    {"type":"reply","destination_ip":"1.1.1.1","sent_bytes":"56","pattern":null,"response_bytes":"64","response_ip":"1.1.1.1","icmp_seq":"0","ttl":"56","time_ms":"23.054"}
+    {"type":"reply","destination_ip":"1.1.1.1","sent_bytes":"56","pattern":null,"response_bytes":"64","response_ip":"1.1.1.1","icmp_seq":"1","ttl":"56","time_ms":"24.739"}
+    {"type":"reply","destination_ip":"1.1.1.1","sent_bytes":"56","pattern":null,"response_bytes":"64","response_ip":"1.1.1.1","icmp_seq":"2","ttl":"56","time_ms":"23.232"}
+    ...
+
+
+## info
+```python
+info()
+```
+Provides parser metadata (version, author, etc.)
+
+## parse
+```python
+parse(data, raw=False, quiet=False, ignore_exceptions=False)
+```
+
+Main text parsing generator function. Returns an iterator object.
+
+Parameters:
+
+    data:              (iterable)  line-based text data to parse (e.g. sys.stdin or str.splitlines())
+    raw:               (boolean)   output preprocessed JSON if True
+    quiet:             (boolean)   suppress warning messages if True
+    ignore_exceptions: (boolean)   ignore parsing exceptions if True
+
+Yields:
+
+    Dictionary. Raw or processed structured data.
+
+Returns:
+
+    Iterator object
+
+## Parser Information
+Compatibility:  linux, darwin, freebsd
+
+Version 0.5 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/route.md

@@ -25,7 +25,7 @@ Schema:
         "genmask":         string,
         "flags":           string,
         "flags_pretty": [
-                           string,
+                           string
         ]
         "metric":          integer,
         "ref":             integer,
@@ -76,7 +76,6 @@ Examples:
       }
     ]
 
-
     $ route -ee | jc --route -p -r
     [
       {
@@ -108,7 +107,6 @@ Examples:
     ]
 
 
-
 ## info
 ```python
 info()

docs/parsers/sfdisk.md

@@ -0,0 +1,228 @@
+[Home](https://kellyjonbrazil.github.io/jc/)
+
+# jc.parsers.sfdisk
+jc - JSON CLI output utility `sfdisk` command output parser
+
+Supports the following `sfdisk` options:
+- `-l`
+- `-F`
+- `-d`   (deprecated - only for older versions of util-linux)
+- `-uM`  (deprecated - only for older versions of util-linux)
+- `-uC`  (deprecated - only for older versions of util-linux)
+- `-uS`  (deprecated - only for older versions of util-linux)
+- `-uB`  (deprecated - only for older versions of util-linux)
+
+Usage (cli):
+
+    # sfdisk -l | jc --sfdisk
+
+    or
+
+    # jc sfdisk -l
+
+Usage (module):
+
+    import jc.parsers.sfdisk
+    result = jc.parsers.sfdisk.parse(sfdisk_command_output)
+
+Schema:
+
+    [
+      {
+        "disk":                     string,
+        "disk_size":                string,
+        "free_disk_size":           string,
+        "bytes":                    integer,
+        "free_bytes":               integer,
+        "sectors":                  integer,
+        "free_sectors":             integer,
+        "cylinders":                integer,
+        "heads":                    integer,
+        "sectors_per_track":        integer,
+        "units":                    string,
+        "logical_sector_size":      integer,
+        "physical_sector_size":     integer,
+        "min_io_size":              integer,
+        "optimal_io_size":          integer,
+        "disk_label_type":          string,
+        "disk_identifier":          string,
+        "disk_model":               string,
+        "partitions": [
+          {
+            "device":               string,
+            "boot":                 boolean,
+            "start":                integer,
+            "end":                  integer,
+            "size":                 string,    # Note: will be integer when using deprecated -d sfdisk option
+            "cyls":                 integer,
+            "mib":                  integer,
+            "blocks":               integer,
+            "sectors":              integer,
+            "id":                   string,
+            "system":               string,
+            "type":                 string
+          }
+        ]
+      }
+    ]
+
+Examples:
+
+    # sfdisk -l | jc --sfdisk -p
+    [
+      {
+        "disk": "/dev/sda",
+        "cylinders": 2610,
+        "heads": 255,
+        "sectors_per_track": 63,
+        "units": "cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0",
+        "partitions": [
+          {
+            "device": "/dev/sda1",
+            "boot": true,
+            "start": 0,
+            "end": 130,
+            "cyls": 131,
+            "blocks": 1048576,
+            "id": "83",
+            "system": "Linux"
+          },
+          {
+            "device": "/dev/sda2",
+            "boot": false,
+            "start": 130,
+            "end": 2610,
+            "cyls": 2481,
+            "blocks": 19921920,
+            "id": "8e",
+            "system": "Linux LVM"
+          },
+          {
+            "device": "/dev/sda3",
+            "boot": false,
+            "start": 0,
+            "end": null,
+            "cyls": 0,
+            "blocks": 0,
+            "id": "0",
+            "system": "Empty"
+          },
+          {
+            "device": "/dev/sda4",
+            "boot": false,
+            "start": 0,
+            "end": null,
+            "cyls": 0,
+            "blocks": 0,
+            "id": "0",
+            "system": "Empty"
+          }
+        ]
+      },
+      {
+        "disk": "/dev/mapper/centos-root",
+        "cylinders": 2218,
+        "heads": 255,
+        "sectors_per_track": 63
+      },
+      {
+        "disk": "/dev/mapper/centos-swap",
+        "cylinders": 261,
+        "heads": 255,
+        "sectors_per_track": 63
+      }
+    ]
+
+    # sfdisk -l | jc --sfdisk -p -r
+    [
+      {
+        "disk": "/dev/sda",
+        "cylinders": "2610",
+        "heads": "255",
+        "sectors_per_track": "63",
+        "units": "cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0",
+        "partitions": [
+          {
+            "device": "/dev/sda1",
+            "boot": "*",
+            "start": "0+",
+            "end": "130-",
+            "cyls": "131-",
+            "blocks": "1048576",
+            "id": "83",
+            "system": "Linux"
+          },
+          {
+            "device": "/dev/sda2",
+            "boot": null,
+            "start": "130+",
+            "end": "2610-",
+            "cyls": "2481-",
+            "blocks": "19921920",
+            "id": "8e",
+            "system": "Linux LVM"
+          },
+          {
+            "device": "/dev/sda3",
+            "boot": null,
+            "start": "0",
+            "end": "-",
+            "cyls": "0",
+            "blocks": "0",
+            "id": "0",
+            "system": "Empty"
+          },
+          {
+            "device": "/dev/sda4",
+            "boot": null,
+            "start": "0",
+            "end": "-",
+            "cyls": "0",
+            "blocks": "0",
+            "id": "0",
+            "system": "Empty"
+          }
+        ]
+      },
+      {
+        "disk": "/dev/mapper/centos-root",
+        "cylinders": "2218",
+        "heads": "255",
+        "sectors_per_track": "63"
+      },
+      {
+        "disk": "/dev/mapper/centos-swap",
+        "cylinders": "261",
+        "heads": "255",
+        "sectors_per_track": "63"
+      }
+    ]
+
+
+## info
+```python
+info()
+```
+Provides parser metadata (version, author, etc.)
+
+## parse
+```python
+parse(data, raw=False, quiet=False)
+```
+
+Main text parsing function
+
+Parameters:
+
+    data:        (string)  text data to parse
+    raw:         (boolean) output preprocessed JSON if True
+    quiet:       (boolean) suppress warning messages if True
+
+Returns:
+
+    List of Dictionaries. Raw or processed structured data.
+
+## Parser Information
+Compatibility:  linux
+
+Version 1.1 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/stat.md

@@ -193,4 +193,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux, darwin, freebsd
 
-Version 1.8 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.9 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/vmstat.md

@@ -0,0 +1,149 @@
+[Home](https://kellyjonbrazil.github.io/jc/)
+
+# jc.parsers.vmstat
+jc - JSON CLI output utility `vmstat` command output parser
+
+Options supported: `-a`, `-w`, `-d`, `-t`
+
+The `epoch` calculated timestamp field is naive (i.e. based on the local time of the system the parser is run on)
+
+The `epoch_utc` calculated timestamp field is timezone-aware and is only available if the timezone field is UTC.
+
+Usage (cli):
+
+    $ vmstat | jc --vmstat
+
+    or
+
+    $ jc vmstat
+
+Usage (module):
+
+    import jc.parsers.vmstat
+    result = jc.parsers.vmstat.parse(vmstat_command_output)
+
+Schema:
+
+    [
+      {
+        "runnable_procs":                    integer,
+        "uninterruptible_sleeping_procs":    integer,
+        "virtual_mem_used":                  integer,
+        "free_mem":                          integer,
+        "buffer_mem":                        integer,
+        "cache_mem":                         integer,
+        "inactive_mem":                      integer,
+        "active_mem":                        integer,
+        "swap_in":                           integer,
+        "swap_out":                          integer,
+        "blocks_in":                         integer,
+        "blocks_out":                        integer,
+        "interrupts":                        integer,
+        "context_switches":                  integer,
+        "user_time":                         integer,
+        "system_time":                       integer,
+        "idle_time":                         integer,
+        "io_wait_time":                      integer,
+        "stolen_time":                       integer,
+        "disk":                              string,
+        "total_reads":                       integer,
+        "merged_reads":                      integer,
+        "sectors_read":                      integer,
+        "reading_ms":                        integer,
+        "total_writes":                      integer,
+        "merged_writes":                     integer,
+        "sectors_written":                   integer,
+        "writing_ms":                        integer,
+        "current_io":                        integer,
+        "io_seconds":                        integer,
+        "timestamp":                         string,
+        "timezone":                          string,
+        "epoch":                             integer,     # naive timestamp if -t flag is used
+        "epoch_utc":                         integer      # aware timestamp if -t flag is used and UTC TZ
+      }
+    ]
+
+Examples:
+
+    $ vmstat | jc --vmstat -p
+    [
+      {
+        "runnable_procs": 2,
+        "uninterruptible_sleeping_procs": 0,
+        "virtual_mem_used": 0,
+        "free_mem": 2794468,
+        "buffer_mem": 2108,
+        "cache_mem": 741208,
+        "inactive_mem": null,
+        "active_mem": null,
+        "swap_in": 0,
+        "swap_out": 0,
+        "blocks_in": 1,
+        "blocks_out": 3,
+        "interrupts": 29,
+        "context_switches": 57,
+        "user_time": 0,
+        "system_time": 0,
+        "idle_time": 99,
+        "io_wait_time": 0,
+        "stolen_time": 0,
+        "timestamp": null,
+        "timezone": null
+      }
+    ]
+
+    $ vmstat | jc --vmstat -p -r
+    [
+      {
+        "runnable_procs": "2",
+        "uninterruptible_sleeping_procs": "0",
+        "virtual_mem_used": "0",
+        "free_mem": "2794468",
+        "buffer_mem": "2108",
+        "cache_mem": "741208",
+        "inactive_mem": null,
+        "active_mem": null,
+        "swap_in": "0",
+        "swap_out": "0",
+        "blocks_in": "1",
+        "blocks_out": "3",
+        "interrupts": "29",
+        "context_switches": "57",
+        "user_time": "0",
+        "system_time": "0",
+        "idle_time": "99",
+        "io_wait_time": "0",
+        "stolen_time": "0",
+        "timestamp": null,
+        "timezone": null
+      }
+    ]
+
+
+## info
+```python
+info()
+```
+Provides parser metadata (version, author, etc.)
+
+## parse
+```python
+parse(data, raw=False, quiet=False)
+```
+
+Main text parsing function
+
+Parameters:
+
+    data:        (string)  text data to parse
+    raw:         (boolean) output preprocessed JSON if True
+    quiet:       (boolean) suppress warning messages if True
+
+Returns:
+
+    List of Dictionaries. Raw or processed structured data.
+
+## Parser Information
+Compatibility:  linux
+
+Version 1.0 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/vmstat_s.md

@@ -0,0 +1,110 @@
+[Home](https://kellyjonbrazil.github.io/jc/)
+
+# jc.parsers.vmstat_s
+jc - JSON CLI output utility `vmstat` command output streaming parser
+
+Options supported: `-a`, `-w`, `-d`, `-t`
+
+The `epoch` calculated timestamp field is naive (i.e. based on the local time of the system the parser is run on)
+
+The `epoch_utc` calculated timestamp field is timezone-aware and is only available if the timezone field is UTC.
+
+Usage (cli):
+
+    $ vmstat | jc --vmstat-s
+
+Usage (module):
+
+    import jc.parsers.vmstat_s
+    result = jc.parsers.vmstat_s.parse(vmstat_command_output.splitlines())    # result is an iterable object
+    for item in result:
+        # do something
+
+Schema:
+
+    {
+      "runnable_procs":                    integer,
+      "uninterruptible_sleeping_procs":    integer,
+      "virtual_mem_used":                  integer,
+      "free_mem":                          integer,
+      "buffer_mem":                        integer,
+      "cache_mem":                         integer,
+      "inactive_mem":                      integer,
+      "active_mem":                        integer,
+      "swap_in":                           integer,
+      "swap_out":                          integer,
+      "blocks_in":                         integer,
+      "blocks_out":                        integer,
+      "interrupts":                        integer,
+      "context_switches":                  integer,
+      "user_time":                         integer,
+      "system_time":                       integer,
+      "idle_time":                         integer,
+      "io_wait_time":                      integer,
+      "stolen_time":                       integer,
+      "disk":                              string,
+      "total_reads":                       integer,
+      "merged_reads":                      integer,
+      "sectors_read":                      integer,
+      "reading_ms":                        integer,
+      "total_writes":                      integer,
+      "merged_writes":                     integer,
+      "sectors_written":                   integer,
+      "writing_ms":                        integer,
+      "current_io":                        integer,
+      "io_seconds":                        integer,
+      "timestamp":                         string,
+      "timezone":                          string,
+      "epoch":                             integer,     # naive timestamp if -t flag is used
+      "epoch_utc":                         integer      # aware timestamp if -t flag is used and UTC TZ
+      "_jc_meta":                                       # This object only exists if using -qq or ignore_exceptions=True
+        {
+          "success":                       booean,      # true if successfully parsed, false if error
+          "error":                         string,      # exists if "success" is false
+          "line":                          string       # exists if "success" is false
+        }
+    }
+
+Examples:
+
+    $ vmstat | jc --vmstat-s
+    {"runnable_procs":2,"uninterruptible_sleeping_procs":0,"virtual_mem_used":0,"free_mem":2794468,"buffer_mem":2108,"cache_mem":741208,"inactive_mem":null,"active_mem":null,"swap_in":0,"swap_out":0,"blocks_in":1,"blocks_out":3,"interrupts":29,"context_switches":57,"user_time":0,"system_time":0,"idle_time":99,"io_wait_time":0,"stolen_time":0,"timestamp":null,"timezone":null}
+    ...
+
+    $ vmstat | jc --vmstat-s -r
+    {"runnable_procs":"2","uninterruptible_sleeping_procs":"0","virtual_mem_used":"0","free_mem":"2794468","buffer_mem":"2108","cache_mem":"741208","inactive_mem":null,"active_mem":null,"swap_in":"0","swap_out":"0","blocks_in":"1","blocks_out":"3","interrupts":"29","context_switches":"57","user_time":"0","system_time":"0","idle_time":"99","io_wait_time":"0","stolen_time":"0","timestamp":null,"timezone":null}
+    ...
+
+
+## info
+```python
+info()
+```
+Provides parser metadata (version, author, etc.)
+
+## parse
+```python
+parse(data, raw=False, quiet=False, ignore_exceptions=False)
+```
+
+Main text parsing generator function. Returns an iterator object.
+
+Parameters:
+
+    data:              (iterable)  line-based text data to parse (e.g. sys.stdin or str.splitlines())
+    raw:               (boolean)   output preprocessed JSON if True
+    quiet:             (boolean)   suppress warning messages if True
+    ignore_exceptions: (boolean)   ignore parsing exceptions if True
+
+Yields:
+
+    Dictionary. Raw or processed structured data.
+
+Returns:
+
+    Iterator object
+
+## Parser Information
+Compatibility:  linux
+
+Version 0.5 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/readme.md

@@ -59,26 +59,13 @@ Module Example:
 
     >>> import jc.parsers.dig
     >>>
-    >>> data = '''; <<>> DiG 9.10.6 <<>> example.com
-    ... ;; global options: +cmd
-    ... ;; Got answer:
-    ... ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64612
-    ... ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
-    ...
-    ... ;; OPT PSEUDOSECTION:
-    ... ; EDNS: version: 0, flags:; udp: 4096
-    ... ;; QUESTION SECTION:
-    ... ;example.com.           IN  A
-    ...
-    ... ;; ANSWER SECTION:
-    ... example.com.        29658   IN  A   93.184.216.34
-    ...
-    ... ;; Query time: 52 msec
-    ... ;; SERVER: 2600:1700:bab0:d40::1#53(2600:1700:bab0:d40::1)
-    ... ;; WHEN: Fri Apr 16 16:13:00 PDT 2021
-    ... ;; MSG SIZE  rcvd: 56'''
+    >>> import subprocess
+    >>> import jc.parsers.dig
     >>>
-    >>> jc.parsers.dig.parse(data)
+    >>> cmd_output = subprocess.check_output(['dig', 'example.com'], text=True)
+    >>> data = jc.parsers.dig.parse(cmd_output)
+    >>>
+    >>> data
     [{'id': 64612, 'opcode': 'QUERY', 'status': 'NOERROR', 'flags': ['qr', 'rd', 'ra'], 'query_num': 1, 'answer_num':
     1, 'authority_num': 0, 'additional_num': 1, 'opt_pseudosection': {'edns': {'version': 0, 'flags': [], 'udp':
     4096}}, 'question': {'name': 'example.com.', 'class': 'IN', 'type': 'A'}, 'answer': [{'name': 'example.com.',

docs/utils.md

@@ -4,14 +4,16 @@ jc - JSON CLI output utility utils
 
 ## warning_message
 ```python
-warning_message(message)
+warning_message(message_lines)
 ```
 
-Prints a warning message for non-fatal issues
+Prints warning message for non-fatal issues. The first line is prepended with
+'jc:  Warning - ' and subsequent lines are indented. Wraps text as needed based
+on the terminal width.
 
 Parameters:
 
-    message:        (string) text of message
+    message:   (list) list of string lines
 
 Returns:
 
@@ -20,14 +22,16 @@ Returns:
 
 ## error_message
 ```python
-error_message(message)
+error_message(message_lines)
 ```
 
-Prints an error message for fatal issues
+Prints an error message for fatal issues. The first line is prepended with
+'jc:  Error - ' and subsequent lines are indented. Wraps text as needed based
+on the terminal width.
 
 Parameters:
 
-    message:        (string) text of message
+    message:   (list) list of string lines
 
 Returns:
 
@@ -117,6 +121,20 @@ Returns:
     True/False      False unless a 'truthy' number or string is found ('y', 'yes', 'true', '1', 1, -1, etc.)
 
 
+## stream_success
+```python
+stream_success(output_line, ignore_exceptions)
+```
+Add `_jc_meta` object to output line if `ignore_exceptions=True`
+
+## stream_error
+```python
+stream_error(e, ignore_exceptions, line)
+```
+Reraise the stream exception with annotation or print an error `_jc_meta`
+field if `ignore_exceptions=True`
+
+
 ## timestamp
 ```python
 timestamp(datetime_string)
@@ -131,7 +149,7 @@ Parameters:
 Attributes:
 
     string              (str)   the input datetime string
-    format              (int)   the format rule that was used to decode the datetime string
+    format              (int)   the format rule that was used to decode the datetime string. None if conversion fails
     naive               (int)   timestamp based on locally configured timezone. None if conversion fails
     utc                 (int)   aware timestamp only if UTC timezone detected in datetime string. None if conversion fails
 

jc/__init__.py

@@ -57,26 +57,13 @@ Module Example:
 
     >>> import jc.parsers.dig
     >>>
-    >>> data = '''; <<>> DiG 9.10.6 <<>> example.com
-    ... ;; global options: +cmd
-    ... ;; Got answer:
-    ... ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64612
-    ... ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
-    ...
-    ... ;; OPT PSEUDOSECTION:
-    ... ; EDNS: version: 0, flags:; udp: 4096
-    ... ;; QUESTION SECTION:
-    ... ;example.com.           IN  A
-    ...
-    ... ;; ANSWER SECTION:
-    ... example.com.        29658   IN  A   93.184.216.34
-    ...
-    ... ;; Query time: 52 msec
-    ... ;; SERVER: 2600:1700:bab0:d40::1#53(2600:1700:bab0:d40::1)
-    ... ;; WHEN: Fri Apr 16 16:13:00 PDT 2021
-    ... ;; MSG SIZE  rcvd: 56'''
+    >>> import subprocess
+    >>> import jc.parsers.dig
     >>>
-    >>> jc.parsers.dig.parse(data)
+    >>> cmd_output = subprocess.check_output(['dig', 'example.com'], text=True)
+    >>> data = jc.parsers.dig.parse(cmd_output)
+    >>>
+    >>> data
     [{'id': 64612, 'opcode': 'QUERY', 'status': 'NOERROR', 'flags': ['qr', 'rd', 'ra'], 'query_num': 1, 'answer_num':
     1, 'authority_num': 0, 'additional_num': 1, 'opt_pseudosection': {'edns': {'version': 0, 'flags': [], 'udp':
     4096}}, 'question': {'name': 'example.com.', 'class': 'IN', 'type': 'A'}, 'answer': [{'name': 'example.com.',
@@ -86,4 +73,4 @@ Module Example:
 """
 
 name = 'jc'
-__version__ = '1.15.6'
+__version__ = '1.17.0'

jc/cli.py

@@ -6,14 +6,14 @@ import sys
 import os
 import os.path
 import re
-import shlex
 import importlib
 import textwrap
 import signal
+import shlex
 import subprocess
 import json
 import jc
-import jc.appdirs as appdirs
+from jc import appdirs
 import jc.utils
 import jc.tracebackplus
 from jc.exceptions import LibraryNotInstalled, ParseError
@@ -26,9 +26,9 @@ try:
     from pygments.token import (Name, Number, String, Keyword)
     from pygments.lexers import JsonLexer
     from pygments.formatters import Terminal256Formatter
-    pygments_installed = True
+    PYGMENTS_INSTALLED = True
 except Exception:
-    pygments_installed = False
+    PYGMENTS_INSTALLED = False
 
 
 class info():
@@ -81,6 +81,7 @@ parsers = [
     'kv',
     'last',
     'ls',
+    'ls-s',
     'lsblk',
     'lsmod',
     'lsof',
@@ -89,11 +90,13 @@ parsers = [
     'ntpq',
     'passwd',
     'ping',
+    'ping-s',
     'pip-list',
     'pip-show',
     'ps',
     'route',
     'rpm-qi',
+    'sfdisk',
     'shadow',
     'ss',
     'stat',
@@ -112,6 +115,8 @@ parsers = [
     'uname',
     'upower',
     'uptime',
+    'vmstat',
+    'vmstat-s',
     'w',
     'wc',
     'who',
@@ -139,7 +144,7 @@ if os.path.isdir(local_parsers_dir):
 
 # We only support 2.3.0+, pygments changed color names in 2.4.0.
 # startswith is sufficient and avoids potential exceptions from split and int.
-if pygments_installed:
+if PYGMENTS_INSTALLED:
     if pygments.__version__.startswith('2.3.'):
         PYGMENT_COLOR = {
             'black': '#ansiblack',
@@ -214,7 +219,7 @@ def set_env_colors(env_colors=None):
 
     # if there is an issue with the env variable, just set all colors to default and move on
     if input_error:
-        jc.utils.warning_message('Could not parse JC_COLORS environment variable')
+        jc.utils.warning_message(['Could not parse JC_COLORS environment variable'])
         color_list = ['default', 'default', 'default', 'default']
 
     # Try the color set in the JC_COLORS env variable first. If it is set to default, then fall back to default colors
@@ -228,7 +233,7 @@ def set_env_colors(env_colors=None):
 
 def piped_output():
     """Return False if stdout is a TTY. True if output is being piped to another program"""
-    return False if sys.stdout.isatty() else True
+    return not sys.stdout.isatty()
 
 
 def ctrlc(signum, frame):
@@ -236,9 +241,9 @@ def ctrlc(signum, frame):
     sys.exit(JC_ERROR_EXIT)
 
 
-def parser_shortname(parser_argument):
+def parser_shortname(parser_arg):
     """Return short name of the parser with dashes and no -- prefix"""
-    return parser_argument[2:]
+    return parser_arg[2:]
 
 
 def parser_argument(parser):
@@ -325,14 +330,15 @@ def helptext():
     Parsers:
 {parsers_string}
     Options:
-            -a               about jc
-            -d               debug (-dd for verbose debug)
-            -h               help (-h --parser_name for parser documentation)
-            -m               monochrome output
-            -p               pretty print output
-            -q               quiet - suppress parser warnings
-            -r               raw JSON output
-            -v               version info
+            -a    about jc
+            -d    debug (-dd for verbose debug)
+            -h    help (-h --parser_name for parser documentation)
+            -m    monochrome output
+            -p    pretty print output
+            -q    quiet - suppress parser warnings (-qq to ignore streaming errors)
+            -r    raw JSON output
+            -u    unbuffer output
+            -v    version info
 
     Examples:
             Standard Syntax:
@@ -359,11 +365,10 @@ def help_doc(options):
             # load parser module just in time so we don't need to load all modules
             parser = parser_module(arg)
             compatible = ', '.join(parser.info.compatible)
-            doc_text = f'''{parser.__doc__}
-Compatibility:  {compatible}
-
-Version {parser.info.version} by {parser.info.author} ({parser.info.author_email})
-'''
+            doc_text = \
+                f'{parser.__doc__}\n'\
+                f'Compatibility:  {compatible}\n\n'\
+                f'Version {parser.info.version} by {parser.info.author} ({parser.info.author_email})\n'
 
             return doc_text
 
@@ -396,8 +401,7 @@ def json_out(data, pretty=False, env_colors=None, mono=False, piped_out=False):
         return str(highlight(json.dumps(data, indent=indent, separators=separators, ensure_ascii=False),
                              JsonLexer(), Terminal256Formatter(style=JcStyle))[0:-1])
 
-    else:
-        return json.dumps(data, indent=indent, separators=separators, ensure_ascii=False)
+    return json.dumps(data, indent=indent, separators=separators, ensure_ascii=False)
 
 
 def magic_parser(args):
@@ -414,8 +418,7 @@ def magic_parser(args):
     if len(args) <= 1 or args[1].startswith('--'):
         return False, None, None, []
 
-    # correctly parse escape characters and spaces with shlex
-    args_given = ' '.join(map(shlex.quote, args[1:])).split()
+    args_given = args[1:]
     options = []
 
     # find the options
@@ -425,7 +428,7 @@ def magic_parser(args):
             return False, None, None, []
 
         # option found - populate option list
-        elif arg.startswith('-'):
+        if arg.startswith('-'):
             options.extend(args_given.pop(0)[1:])
 
         # command found if iterator didn't already stop - stop iterating
@@ -456,7 +459,7 @@ def magic_parser(args):
     found_parser = magic_dict.get(two_word_command, magic_dict.get(one_word_command))
 
     return (
-        True if found_parser else False,    # was a suitable parser found?
+        bool(found_parser),                 # was a suitable parser found?
         args_given,                         # run_command
         found_parser,                       # the parser selected
         options                             # jc options to preserve
@@ -465,7 +468,11 @@ def magic_parser(args):
 
 def run_user_command(command):
     """Use subprocess to run the user's command. Returns the STDOUT, STDERR, and the Exit Code as a tuple."""
-    proc = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
+    proc = subprocess.Popen(command,
+                            stdout=subprocess.PIPE,
+                            stderr=subprocess.PIPE,
+                            close_fds=False,            # Allows inheriting file descriptors. Useful for process substitution
+                            universal_newlines=True)
     stdout, stderr = proc.communicate()
 
     return (
@@ -477,8 +484,7 @@ def run_user_command(command):
 
 def combined_exit_code(program_exit=0, jc_exit=0):
     exit_code = program_exit + jc_exit
-    if exit_code > 255:
-        exit_code = 255
+    exit_code = min(exit_code, 255)
     return exit_code
 
 
@@ -515,18 +521,20 @@ def main():
 
     about = 'a' in options
     debug = 'd' in options
-    verbose_debug = True if options.count('d') > 1 else False
+    verbose_debug = options.count('d') > 1
     mono = 'm' in options
     help_me = 'h' in options
     pretty = 'p' in options
     quiet = 'q' in options
+    ignore_exceptions = options.count('q') > 1
     raw = 'r' in options
+    unbuffer = 'u' in options
     version_info = 'v' in options
 
     if verbose_debug:
         jc.tracebackplus.enable(context=11)
 
-    if not pygments_installed:
+    if not PYGMENTS_INSTALLED:
         mono = True
 
     if about:
@@ -544,7 +552,10 @@ def main():
     # if magic syntax used, try to run the command and error if it's not found, etc.
     magic_stdout, magic_stderr, magic_exit_code = None, None, 0
     if run_command:
-        run_command_str = ' '.join(run_command)
+        try:
+            run_command_str = shlex.join(run_command)      # python 3.8+
+        except AttributeError:
+            run_command_str = ' '.join(run_command)        # older python versions
 
     if valid_command:
         try:
@@ -555,19 +566,26 @@ def main():
         except FileNotFoundError:
             if debug:
                 raise
-            else:
-                jc.utils.error_message(f'"{run_command_str}" command could not be found. For details use the -d or -dd option.')
-                sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
+
+            jc.utils.error_message([f'"{run_command_str}" command could not be found. For details use the -d or -dd option.'])
+            sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
+
+        except OSError:
+            if debug:
+                raise
+
+            jc.utils.error_message([f'"{run_command_str}" command could not be run due to too many open files. For details use the -d or -dd option.'])
+            sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
 
         except Exception:
             if debug:
                 raise
-            else:
-                jc.utils.error_message(f'"{run_command_str}" command could not be run. For details use the -d or -dd option.')
-                sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
+
+            jc.utils.error_message([f'"{run_command_str}" command could not be run. For details use the -d or -dd option.'])
+            sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
 
     elif run_command is not None:
-        jc.utils.error_message(f'"{run_command_str}" cannot be used with Magic syntax. Use "jc -h" for help.')
+        jc.utils.error_message([f'"{run_command_str}" cannot be used with Magic syntax. Use "jc -h" for help.'])
         sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
 
     # find the correct parser
@@ -586,56 +604,79 @@ def main():
                 break
 
         if not found:
-            jc.utils.error_message('Missing or incorrect arguments. Use "jc -h" for help.')
+            jc.utils.error_message(['Missing or incorrect arguments. Use "jc -h" for help.'])
             sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
 
     # check for input errors (pipe vs magic)
     if not sys.stdin.isatty() and magic_stdout:
-        jc.utils.error_message('Piped data and Magic syntax used simultaneously. Use "jc -h" for help.')
+        jc.utils.error_message(['Piped data and Magic syntax used simultaneously. Use "jc -h" for help.'])
         sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
 
     elif sys.stdin.isatty() and magic_stdout is None:
-        jc.utils.error_message('Missing piped data. Use "jc -h" for help.')
+        jc.utils.error_message(['Missing piped data. Use "jc -h" for help.'])
         sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
 
-    # parse the data
-    data = magic_stdout or sys.stdin.read()
-
+    # parse and print to stdout
     try:
-        result = parser.parse(data, raw=raw, quiet=quiet)
+        # differentiate between regular and streaming parsers
+
+        # streaming
+        if getattr(parser.info, 'streaming', None):
+            result = parser.parse(sys.stdin, raw=raw, quiet=quiet, ignore_exceptions=ignore_exceptions)
+            for line in result:
+                print(json_out(line,
+                               pretty=pretty,
+                               env_colors=jc_colors,
+                               mono=mono,
+                               piped_out=piped_output()),
+                      flush=unbuffer)
+
+            sys.exit(combined_exit_code(magic_exit_code, 0))
+
+        # regular
+        else:
+            data = magic_stdout or sys.stdin.read()
+            result = parser.parse(data, raw=raw, quiet=quiet)
+            print(json_out(result,
+                           pretty=pretty,
+                           env_colors=jc_colors,
+                           mono=mono,
+                           piped_out=piped_output()),
+                  flush=unbuffer)
+
+        sys.exit(combined_exit_code(magic_exit_code, 0))
 
     except (ParseError, LibraryNotInstalled) as e:
         if debug:
             raise
-        else:
-            jc.utils.error_message(
-                f'Parser issue with {parser_name}:\n'
-                f'             {e}\n'
-                '             For details use the -d or -dd option. Use "jc -h" for help.')
-            sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
+
+        jc.utils.error_message([f'Parser issue with {parser_name}:',
+                                f'{e.__class__.__name__}: {e}',
+                                'For details use the -d or -dd option. Use "jc -h" for help.'])
+        sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
+
+    except json.JSONDecodeError:
+        if debug:
+            raise
+
+        jc.utils.error_message(['There was an issue generating the JSON output.',
+                                'For details use the -d or -dd option.'])
+        sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
 
     except Exception:
         if debug:
             raise
-        else:
-            jc.utils.error_message(
-                f'{parser_name} parser could not parse the input data. Did you use the correct parser?\n'
-                '             For details use the -d or -dd option. Use "jc -h" for help.')
-            sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
 
-    # output the json
-    try:
-        print(json_out(result, pretty=pretty, env_colors=jc_colors, mono=mono, piped_out=piped_output()))
-        sys.exit(combined_exit_code(magic_exit_code, 0))
+        streaming_msg = ''
+        if getattr(parser.info, 'streaming', None):
+            streaming_msg = 'Use the -qq option to ignore streaming parser errors.'
 
-    except Exception:
-        if debug:
-            raise
-        else:
-            jc.utils.error_message(
-                'There was an issue generating the JSON output.\n'
-                '             For details use the -d or -dd option.')
-            sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
+        jc.utils.error_message([
+            f'{parser_name} parser could not parse the input data. Did you use the correct parser?',
+            f'{streaming_msg}',
+            'For details use the -d or -dd option. Use "jc -h" for help.'
+        ])
+        sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
 
 
 if __name__ == '__main__':

jc/parsers/file.py

@@ -130,7 +130,7 @@ def parse(data, raw=False, quiet=False):
                 )
             except IndexError:
                 if not warned:
-                    jc.utils.warning_message('Filenames with newline characters detected. Some filenames may be truncated.')
+                    jc.utils.warning_message(['Filenames with newline characters detected. Some filenames may be truncated.'])
                     warned = True
 
     if raw:

jc/parsers/foo.py

@@ -65,6 +65,7 @@ def _process(proc_data):
         List of Dictionaries. Structured to conform to the schema.
     """
 
+    # process the data here
     # rebuild output for added semantic information
     # use helper functions in jc.utils for int, float, bool conversions and timestamps
 
@@ -93,10 +94,9 @@ def parse(data, raw=False, quiet=False):
     if jc.utils.has_data(data):
 
         for line in filter(None, data.splitlines()):
-            # parse the content
+            #
+            # parse the content here
+            #
             pass
 
-    if raw:
-        return raw_output
-    else:
-        return _process(raw_output)
+    return raw_output if raw else _process(raw_output)

jc/parsers/foo_s.py

@@ -0,0 +1,115 @@
+"""jc - JSON CLI output utility `foo` command output streaming parser
+
+<<Short foo description and caveats>>
+
+Usage (cli):
+
+    $ foo | jc --foo-s
+
+Usage (module):
+
+    import jc.parsers.foo_s
+    result = jc.parsers.foo_s.parse(foo_command_output.splitlines())    # result is an iterable object
+    for item in result:
+        # do something
+
+Schema:
+
+    {
+      "foo":            string,
+      "_jc_meta":                    # This object only exists if using -qq or ignore_exceptions=True
+        {
+          "success":    booean,      # true if successfully parsed, false if error
+          "error":      string,      # exists if "success" is false
+          "line":       string       # exists if "success" is false
+        }
+    }
+
+Examples:
+
+    $ foo | jc --foo-s
+    {example output}
+    ...
+
+    $ foo | jc --foo-s -r
+    {example output}
+    ...
+"""
+import jc.utils
+from jc.utils import stream_success, stream_error
+from jc.exceptions import ParseError
+
+
+class info():
+    """Provides parser metadata (version, author, etc.)"""
+    version = '1.0'
+    description = '`foo` command streaming parser'
+    author = 'John Doe'
+    author_email = 'johndoe@gmail.com'
+
+    # compatible options: linux, darwin, cygwin, win32, aix, freebsd
+    compatible = ['linux', 'darwin', 'cygwin', 'aix', 'freebsd']
+    streaming = True
+
+
+__version__ = info.version
+
+
+def _process(proc_data):
+    """
+    Final processing to conform to the schema.
+
+    Parameters:
+
+        proc_data:   (Dictionary) raw structured data to process
+
+    Returns:
+
+        Dictionary. Structured data to conform to the schema.
+    """
+    #
+    # process the data here
+    # rebuild output for added semantic information
+    # use helper functions in jc.utils for int, float, bool conversions and timestamps
+    #
+
+    return proc_data
+
+
+def parse(data, raw=False, quiet=False, ignore_exceptions=False):
+    """
+    Main text parsing generator function. Returns an iterator object.
+
+    Parameters:
+
+        data:              (iterable)  line-based text data to parse (e.g. sys.stdin or str.splitlines())
+        raw:               (boolean)   output preprocessed JSON if True
+        quiet:             (boolean)   suppress warning messages if True
+        ignore_exceptions: (boolean)   ignore parsing exceptions if True
+
+    Yields:
+
+        Dictionary. Raw or processed structured data.
+
+    Returns:
+
+        Iterator object
+    """
+    if not quiet:
+        jc.utils.compatibility(__name__, info.compatible)
+
+    for line in data:
+        try:
+            output_line = {}
+
+            #
+            # parse the input here
+            #
+
+            if output_line:
+                yield stream_success(output_line, ignore_exceptions) if raw else stream_success(_process(output_line), ignore_exceptions)
+            else:
+                raise ParseError('Not foo data')
+
+        except Exception as e:
+            yield stream_error(e, ignore_exceptions, line)

jc/parsers/kv.py

@@ -35,7 +35,7 @@ Examples:
     # but can be preserved with the -r argument
     occupation:"Engineer"
 
-    $ cat keyvalue.txt | jc --ini -p
+    $ cat keyvalue.txt | jc --kv -p
     {
       "name": "John Doe",
       "address": "555 California Drive",

jc/parsers/ls.py

@@ -255,7 +255,7 @@ def parse(data, raw=False, quiet=False):
                     continue
 
                 if not quiet and next_is_parent and not entry.endswith(':') and not warned:
-                    jc.utils.warning_message('Newline characters detected. Filenames probably corrupted. Use ls -l or -b instead.')
+                    jc.utils.warning_message(['Newline characters detected. Filenames probably corrupted. Use ls -l or -b instead.'])
                     warned = True
 
                 output_line['filename'] = entry

jc/parsers/ls_s.py

@@ -0,0 +1,178 @@
+"""jc - JSON CLI output utility `ls` and `vdir` command output streaming parser
+
+This streaming parser requires the `-l` option to be used on `ls`. If there are newline characters in the filename, then make sure to use the `-b` option on `ls`.
+
+The `jc` `-q` option can be used to ignore parsing errors. (e.g. filenames with newline characters, but `-b` was not used)
+
+The `epoch` calculated timestamp field is naive (i.e. based on the local time of the system the parser is run on)
+
+The `epoch_utc` calculated timestamp field is timezone-aware and is only available if the timezone field is UTC.
+
+Usage (cli):
+
+    $ ls | jc --ls-s
+
+Usage (module):
+
+    import jc.parsers.ls_s
+    result = jc.parsers.ls_s.parse(ls_command_output.splitlines())    # result is an iterable object
+    for item in result:
+        # do something
+
+Schema:
+
+    {
+      "filename":       string,
+      "flags":          string,
+      "links":          integer,
+      "parent":         string,
+      "owner":          string,
+      "group":          string,
+      "size":           integer,
+      "date":           string,
+      "epoch":          integer,     # naive timestamp if date field exists and can be converted
+      "epoch_utc":      integer,     # timezone aware timestamp if date field is in UTC and can be converted
+      "_jc_meta":                    # This object only exists if using -qq or ignore_exceptions=True
+        {
+          "success":    booean,      # true if successfully parsed, false if error
+          "error":      string,      # exists if "success" is false
+          "line":       string       # exists if "success" is false
+        }
+    }
+
+Examples:
+
+    $ ls -l /usr/bin | jc --ls-s
+    {"filename":"2to3-","flags":"-rwxr-xr-x","links":4,"owner":"root","group":"wheel","size":925,"date":"Feb 22 2019"}
+    {"filename":"2to3-2.7","link_to":"../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/2to3-2.7","flags":"lrwxr-xr-x","links":1,"owner":"root","group":"wheel","size":74,"date":"May 4 2019"}
+    {"filename":"AssetCacheLocatorUtil","flags":"-rwxr-xr-x","links":1,"owner":"root","group":"wheel","size":55152,"date":"May 3 2019"}
+    ...
+
+    $ ls -l /usr/bin | jc --ls-s -r
+    {"filename":"2to3-","flags":"-rwxr-xr-x","links":"4","owner":"root","group":"wheel","size":"925","date":"Feb 22 2019"}
+    {"filename":"2to3-2.7","link_to":"../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/2to3-2.7","flags":"lrwxr-xr-x","links":"1","owner":"root","group":"wheel","size":"74","date":"May 4 2019"}
+    {"filename":"AssetCacheLocatorUtil","flags":"-rwxr-xr-x","links":"1","owner":"root","group":"wheel","size":"55152","date":"May 3 2019"}
+    ...
+"""
+import re
+import jc.utils
+from jc.utils import stream_success, stream_error
+from jc.exceptions import ParseError
+
+
+class info():
+    """Provides parser metadata (version, author, etc.)"""
+    version = '0.5'
+    description = '`ls` command streaming parser'
+    author = 'Kelly Brazil'
+    author_email = 'kellyjonbrazil@gmail.com'
+
+    # compatible options: linux, darwin, cygwin, win32, aix, freebsd
+    compatible = ['linux', 'darwin', 'cygwin', 'aix', 'freebsd']
+    streaming = True
+
+
+__version__ = info.version
+
+
+def _process(proc_data):
+    """
+    Final processing to conform to the schema.
+
+    Parameters:
+
+        proc_data:   (Dictionary) raw structured data to process
+
+    Returns:
+
+        Dictionary. Structured data to conform to the schema.
+    """
+    int_list = ['links', 'size']
+    for key in proc_data:
+        if key in int_list:
+            proc_data[key] = jc.utils.convert_to_int(proc_data[key])
+
+    if 'date' in proc_data:
+        # to speed up processing only try to convert the date if it's not the default format
+        if not re.match(r'[a-zA-Z]{3}\s{1,2}\d{1,2}\s{1,2}[0-9:]{4,5}', proc_data['date']):
+            ts = jc.utils.timestamp(proc_data['date'])
+            proc_data['epoch'] = ts.naive
+            proc_data['epoch_utc'] = ts.utc
+
+    return proc_data
+
+
+def parse(data, raw=False, quiet=False, ignore_exceptions=False):
+    """
+    Main text parsing generator function. Returns an iterator object.
+
+    Parameters:
+
+        data:              (iterable)  line-based text data to parse (e.g. sys.stdin or str.splitlines())
+        raw:               (boolean)   output preprocessed JSON if True
+        quiet:             (boolean)   suppress warning messages if True
+        ignore_exceptions: (boolean)   ignore parsing exceptions if True
+
+    Yields:
+
+        Dictionary. Raw or processed structured data.
+
+    Returns:
+
+        Iterator object
+    """
+    if not quiet:
+        jc.utils.compatibility(__name__, info.compatible)
+
+    parent = ''
+
+    for line in data:
+        try:
+
+            # skip line if it starts with 'total 1234'
+            if re.match(r'total [0-9]+', line):
+                continue
+
+            # skip blank lines
+            if line.strip() == '':
+                continue
+
+            # Look for parent line if glob or -R is used
+            if not re.match(r'[-dclpsbDCMnP?]([-r][-w][-xsS]){2}([-r][-w][-xtT])[+]?', line) \
+               and line.strip().endswith(':'):
+                parent = line.strip()[:-1]
+                continue
+
+            if not re.match(r'[-dclpsbDCMnP?]([-r][-w][-xsS]){2}([-r][-w][-xtT])[+]?', line):
+                raise ParseError('Not ls -l data')
+
+            parsed_line = line.strip().split(maxsplit=8)
+            output_line = {}
+
+            # split filenames and links
+            if len(parsed_line) == 9:
+                filename_field = parsed_line[8].split(' -> ')
+            else:
+                # in case of filenames starting with a newline character
+                filename_field = ['']
+
+            # create output object
+            output_line['filename'] = filename_field[0]
+
+            if len(filename_field) > 1:
+                output_line['link_to'] = filename_field[1]
+
+            if parent:
+                output_line['parent'] = parent
+
+            output_line['flags'] = parsed_line[0]
+            output_line['links'] = parsed_line[1]
+            output_line['owner'] = parsed_line[2]
+            output_line['group'] = parsed_line[3]
+            output_line['size'] = parsed_line[4]
+            output_line['date'] = ' '.join(parsed_line[5:8])
+
+            yield stream_success(output_line, ignore_exceptions) if raw else stream_success(_process(output_line), ignore_exceptions)
+
+        except Exception as e:
+            yield stream_error(e, ignore_exceptions, line)

jc/parsers/ping_s.py

@@ -0,0 +1,524 @@
+"""jc - JSON CLI output utility `ping` command output streaming parser
+
+Supports `ping` and `ping6` output.
+
+Usage (cli):
+
+    $ ping | jc --ping-s
+
+> Note: When piping `jc` converted ping output to other processes it may appear the output is hanging due to the OS pipe buffers. This is because `ping` output is too small to quickly fill up the buffer. Use the `-u` option to unbuffer the `jc` output if you would like immediate output. See the [readme](https://github.com/kellyjonbrazil/jc/tree/streaming#streaming-parsers) for more information.
+
+Usage (module):
+
+    import jc.parsers.ping_s
+    result = jc.parsers.ping_s.parse(ping_command_output.splitlines())    # result is an iterable object
+    for item in result:
+        # do something
+
+Schema:
+
+    {
+      "type":                        string,        # 'reply', 'timeout', 'summary', etc. See `_error_type.type_map` for all options.
+      "source_ip":                   string,
+      "destination_ip":              string,
+      "sent_bytes":                  integer,
+      "pattern":                     string,        # (null if not set)
+      "destination":                 string,
+      "timestamp":                   float,
+      "response_bytes":              integer,
+      "response_ip":                 string,
+      "icmp_seq":                    integer,
+      "ttl":                         integer,
+      "time_ms":                     float,
+      "duplicate":                   boolean,
+      "packets_transmitted":         integer,
+      "packets_received":            integer,
+      "packet_loss_percent":         float,
+      "duplicates":                  integer,
+      "round_trip_ms_min":           float,
+      "round_trip_ms_avg":           float,
+      "round_trip_ms_max":           float,
+      "round_trip_ms_stddev":        float,
+      "_jc_meta":                                  # This object only exists if using -qq or ignore_exceptions=True
+        {
+          "success":                 booean,       # true if successfully parsed, false if error
+          "error":                   string,       # exists if "success" is false
+          "line":                    string        # exists if "success" is false
+        }
+    }
+
+Examples:
+
+    $ ping 1.1.1.1 | jc --ping-s
+    {"type":"reply","destination_ip":"1.1.1.1","sent_bytes":56,"pattern":null,"response_bytes":64,"response_ip":"1.1.1.1","icmp_seq":0,"ttl":56,"time_ms":23.703}
+    {"type":"reply","destination_ip":"1.1.1.1","sent_bytes":56,"pattern":null,"response_bytes":64,"response_ip":"1.1.1.1","icmp_seq":1,"ttl":56,"time_ms":22.862}
+    {"type":"reply","destination_ip":"1.1.1.1","sent_bytes":56,"pattern":null,"response_bytes":64,"response_ip":"1.1.1.1","icmp_seq":2,"ttl":56,"time_ms":22.82}
+    ...
+
+    $ ping 1.1.1.1 | jc --ping-s -r
+    {"type":"reply","destination_ip":"1.1.1.1","sent_bytes":"56","pattern":null,"response_bytes":"64","response_ip":"1.1.1.1","icmp_seq":"0","ttl":"56","time_ms":"23.054"}
+    {"type":"reply","destination_ip":"1.1.1.1","sent_bytes":"56","pattern":null,"response_bytes":"64","response_ip":"1.1.1.1","icmp_seq":"1","ttl":"56","time_ms":"24.739"}
+    {"type":"reply","destination_ip":"1.1.1.1","sent_bytes":"56","pattern":null,"response_bytes":"64","response_ip":"1.1.1.1","icmp_seq":"2","ttl":"56","time_ms":"23.232"}
+    ...
+"""
+import string
+import ipaddress
+import jc.utils
+from jc.exceptions import ParseError
+from jc.utils import stream_success, stream_error
+
+
+class info():
+    """Provides parser metadata (version, author, etc.)"""
+    version = '0.5'
+    description = '`ping` and `ping6` command streaming parser'
+    author = 'Kelly Brazil'
+    author_email = 'kellyjonbrazil@gmail.com'
+
+    # compatible options: linux, darwin, cygwin, win32, aix, freebsd
+    compatible = ['linux', 'darwin', 'freebsd']
+    streaming = True
+
+
+__version__ = info.version
+
+
+def _process(proc_data):
+    """
+    Final processing to conform to the schema.
+
+    Parameters:
+
+        proc_data:   (Dictionary) raw structured data to process
+
+    Returns:
+
+        Dictionary. Structured data to conform to the schema.
+    """
+    int_list = ['sent_bytes', 'packets_transmitted', 'packets_received', 'response_bytes', 'icmp_seq', 'ttl',
+                'duplicates', 'vr', 'hl', 'tos', 'len', 'id', 'flg', 'off', 'pro', 'cks']
+    float_list = ['packet_loss_percent', 'round_trip_ms_min', 'round_trip_ms_avg', 'round_trip_ms_max',
+                  'round_trip_ms_stddev', 'timestamp', 'time_ms']
+
+    for key in proc_data:
+        if key in int_list:
+            proc_data[key] = jc.utils.convert_to_int(proc_data[key])
+
+        if key in float_list:
+            proc_data[key] = jc.utils.convert_to_float(proc_data[key])
+
+    return proc_data
+
+
+class _state:
+    os_detected = None
+    linux = None
+    bsd = None
+    ipv4 = None
+    hostname = None
+    destination_ip = None
+    sent_bytes = None
+    pattern = None
+    footer = False
+    packets_transmitted = None
+    packets_received = None
+    packet_loss_percent = None
+    time_ms = None
+    duplicates = None
+
+
+def _ipv6_in(line):
+    line_list = line.replace('(', ' ').replace(')', ' ').replace(',', ' ').replace('%', ' ').split()
+    ipv6 = False
+    for item in line_list:
+        try:
+            _ = ipaddress.IPv6Address(item)
+            ipv6 = True
+        except Exception:
+            pass
+    return ipv6
+
+
+def _error_type(line):
+    # from https://github.com/dgibson/iputils/blob/master/ping.c
+    # https://android.googlesource.com/platform/external/ping/+/8fc3c91cf9e7f87bc20b9e6d3ea2982d87b70d9a/ping.c
+    # https://opensource.apple.com/source/network_cmds/network_cmds-328/ping.tproj/ping.c
+    type_map = {
+        'Destination Net Unreachable': 'destination_net_unreachable',
+        'Destination Host Unreachable': 'destination_host_unreachable',
+        'Destination Protocol Unreachable': 'destination_protocol_unreachable',
+        'Destination Port Unreachable': 'destination_port_unreachable',
+        'Frag needed and DF set': 'frag_needed_and_df_set',
+        'Source Route Failed': 'source_route_failed',
+        'Destination Net Unknown': 'destination_net_unknown',
+        'Destination Host Unknown': 'destination_host_unknown',
+        'Source Host Isolated': 'source_host_isolated',
+        'Destination Net Prohibited': 'destination_net_prohibited',
+        'Destination Host Prohibited': 'destination_host_prohibited',
+        'Destination Net Unreachable for Type of Service': 'destination_net_unreachable_for_type_of_service',
+        'Destination Host Unreachable for Type of Service': 'destination_host_unreachable_for_type_of_service',
+        'Packet filtered': 'packet_filtered',
+        'Precedence Violation': 'precedence_violation',
+        'Precedence Cutoff': 'precedence_cutoff',
+        'Dest Unreachable, Bad Code': 'dest_unreachable_bad_code',
+        'Redirect Network': 'redirect_network',
+        'Redirect Host': 'redirect_host',
+        'Redirect Type of Service and Network': 'redirect_type_of_service_and_network',
+        'Redirect, Bad Code': 'redirect_bad_code',
+        'Time to live exceeded': 'time_to_live_exceeded',
+        'Frag reassembly time exceeded': 'frag_reassembly_time_exceeded',
+        'Time exceeded, Bad Code': 'time_exceeded_bad_code'
+    }
+
+    for err_type, code in type_map.items():
+        if err_type in line:
+            return code
+
+    return None
+
+
+def _bsd_parse(line, s):
+    output_line = {}
+
+    if line.startswith('PING '):
+        s.destination_ip = line.split()[2].lstrip('(').rstrip(':').rstrip(')')
+        s.sent_bytes = line.split()[3]
+        return None
+
+    if line.startswith('PING6('):
+        line = line.replace('(', ' ').replace(')', ' ').replace('=', ' ')
+        s.source_ip = line.split()[4]
+        s.destination_ip = line.split()[6]
+        s.sent_bytes = line.split()[1]
+        return None
+
+    if line.startswith('---'):
+        s.footer = True
+        return None
+
+    if s.footer:
+        if 'packets transmitted' in line:
+            if ' duplicates,' in line:
+                s.packets_transmitted = line.split()[0]
+                s.packets_received = line.split()[3]
+                s.packet_loss_percent = line.split()[8].rstrip('%')
+                s.duplicates = line.split()[6].lstrip('+')
+                return None
+
+            s.packets_transmitted = line.split()[0]
+            s.packets_received = line.split()[3]
+            s.packet_loss_percent = line.split()[6].rstrip('%')
+            s.duplicates = '0'
+            return None
+
+        split_line = line.split(' = ')[1]
+        split_line = split_line.split('/')
+
+        output_line = {
+            'type': 'summary',
+            'destination_ip': s.destination_ip or None,
+            'sent_bytes': s.sent_bytes or None,
+            'pattern': s.pattern or None,
+            'packets_transmitted': s.packets_transmitted or None,
+            'packets_received': s.packets_received or None,
+            'packet_loss_percent': s.packet_loss_percent or None,
+            'duplicates': s.duplicates or None,
+            'round_trip_ms_min': split_line[0],
+            'round_trip_ms_avg': split_line[1],
+            'round_trip_ms_max': split_line[2],
+            'round_trip_ms_stddev': split_line[3].replace(' ms', '')
+        }
+
+        return output_line
+
+    # ping response lines
+
+    # ipv4 lines
+    if not _ipv6_in(line):
+
+        # request timeout
+        if line.startswith('Request timeout for '):
+            output_line = {
+                'type': 'timeout',
+                'destination_ip': s.destination_ip or None,
+                'sent_bytes': s.sent_bytes or None,
+                'pattern': s.pattern or None,
+                'icmp_seq': line.split()[4]
+            }
+
+            return output_line
+
+        # catch error responses
+        err = _error_type(line)
+        if err:
+            output_line = {
+                'type': err
+            }
+
+            try:
+                output_line['bytes'] = line.split()[0]
+                output_line['destination_ip'] = s.destination_ip
+                output_line['response_ip'] = line.split()[4].strip(':').strip('(').strip(')')
+            except Exception:
+                pass
+
+            return output_line
+
+        # normal response
+        if ' bytes from ' in line:
+            line = line.replace(':', ' ').replace('=', ' ')
+
+            output_line = {
+                'type': 'reply',
+                'destination_ip': s.destination_ip or None,
+                'sent_bytes': s.sent_bytes or None,
+                'pattern': s.pattern or None,
+                'response_bytes': line.split()[0],
+                'response_ip': line.split()[3],
+                'icmp_seq': line.split()[5],
+                'ttl': line.split()[7],
+                'time_ms': line.split()[9]
+            }
+
+            return output_line
+
+    # ipv6 lines
+    elif ' bytes from ' in line:
+        line = line.replace(',', ' ').replace('=', ' ')
+
+        output_line = {
+            'type': 'reply',
+            'destination_ip': s.destination_ip or None,
+            'sent_bytes': s.sent_bytes or None,
+            'pattern': s.pattern or None,
+            'bytes': line.split()[0],
+            'response_ip': line.split()[3],
+            'icmp_seq': line.split()[5],
+            'ttl': line.split()[7],
+            'time_ms': line.split()[9]
+        }
+
+        return output_line
+
+
+def _linux_parse(line, s):
+    """
+    Linux ping line parsing function.
+
+    Parameters:
+
+        line:        (string)        line of text data to parse
+        s:           (state object)  global state
+
+    Returns:
+
+        Dictionary. Raw structured data.
+    """
+    output_line = {}
+
+    if line.startswith('PING '):
+        s.ipv4 = 'bytes of data' in line
+
+        if s.ipv4 and line[5] not in string.digits:
+            s.hostname = True
+        elif s.ipv4 and line[5] in string.digits:
+            s.hostname = False
+        elif not s.ipv4 and ' (' in line:
+            s.hostname = True
+        else:
+            s.hostname = False
+
+        if s.ipv4 and not s.hostname:
+            dst_ip, dta_byts = (2, 3)
+        elif s.ipv4 and s.hostname:
+            dst_ip, dta_byts = (2, 3)
+        elif not s.ipv4 and not s.hostname:
+            dst_ip, dta_byts = (2, 3)
+        else:
+            dst_ip, dta_byts = (3, 4)
+
+        line = line.replace('(', ' ').replace(')', ' ')
+        s.destination_ip = line.split()[dst_ip].lstrip('(').rstrip(')')
+        s.sent_bytes = line.split()[dta_byts]
+
+        return None
+
+    if line.startswith('---'):
+        s.footer = True
+        return None
+
+    if s.footer:
+        if 'packets transmitted' in line:
+            if ' duplicates,' in line:
+                s.packets_transmitted = line.split()[0]
+                s.packets_received = line.split()[3]
+                s.packet_loss_percent = line.split()[7].rstrip('%')
+                s.duplicates = line.split()[5].lstrip('+')
+                s.time_ms = line.split()[11].replace('ms', '')
+                return None
+
+            s.packets_transmitted = line.split()[0]
+            s.packets_received = line.split()[3]
+            s.packet_loss_percent = line.split()[5].rstrip('%')
+            s.duplicates = '0'
+            s.time_ms = line.split()[9].replace('ms', '')
+            return None
+
+        split_line = line.split(' = ')[1]
+        split_line = split_line.split('/')
+        output_line = {
+            'type': 'summary',
+            'destination_ip': s.destination_ip or None,
+            'sent_bytes': s.sent_bytes or None,
+            'pattern': s.pattern or None,
+            'packets_transmitted': s.packets_transmitted or None,
+            'packets_received': s.packets_received or None,
+            'packet_loss_percent': s.packet_loss_percent or None,
+            'duplicates': s.duplicates or None,
+            'time_ms': s.time_ms or None,
+            'round_trip_ms_min': split_line[0],
+            'round_trip_ms_avg': split_line[1],
+            'round_trip_ms_max': split_line[2],
+            'round_trip_ms_stddev': split_line[3].split()[0]
+        }
+
+        return output_line
+
+    # ping response lines
+
+    # request timeout
+    if 'no answer yet for icmp_seq=' in line:
+        timestamp = False
+        isequence = 5
+
+        # if timestamp option is specified, then shift icmp sequence field right by one
+        if line[0] == '[':
+            timestamp = True
+            isequence = 6
+
+        output_line = {
+            'type': 'timeout',
+            'destination_ip': s.destination_ip or None,
+            'sent_bytes': s.sent_bytes or None,
+            'pattern': s.pattern or None,
+            'timestamp': line.split()[0].lstrip('[').rstrip(']') if timestamp else None,
+            'icmp_seq': line.replace('=', ' ').split()[isequence]
+        }
+
+        return output_line
+
+    # normal responses
+    if ' bytes from ' in line:
+
+        line = line.replace('(', ' ').replace(')', ' ').replace('=', ' ')
+
+        # positions of items depend on whether ipv4/ipv6 and/or ip/hostname is used
+        if s.ipv4 and not s.hostname:
+            bts, rip, iseq, t2l, tms = (0, 3, 5, 7, 9)
+        elif s.ipv4 and s.hostname:
+            bts, rip, iseq, t2l, tms = (0, 4, 7, 9, 11)
+        elif not s.ipv4 and not s.hostname:
+            bts, rip, iseq, t2l, tms = (0, 3, 5, 7, 9)
+        elif not s.ipv4 and s.hostname:
+            bts, rip, iseq, t2l, tms = (0, 4, 7, 9, 11)
+
+        # if timestamp option is specified, then shift everything right by one
+        timestamp = False
+        if line[0] == '[':
+            timestamp = True
+            bts, rip, iseq, t2l, tms = (bts + 1, rip + 1, iseq + 1, t2l + 1, tms + 1)
+
+        output_line = {
+            'type': 'reply',
+            'destination_ip': s.destination_ip or None,
+            'sent_bytes': s.sent_bytes or None,
+            'pattern': s.pattern or None,
+            'timestamp': line.split()[0].lstrip('[').rstrip(']') if timestamp else None,
+            'response_bytes': line.split()[bts],
+            'response_ip': line.split()[rip].rstrip(':'),
+            'icmp_seq': line.split()[iseq],
+            'ttl': line.split()[t2l],
+            'time_ms': line.split()[tms],
+            'duplicate': 'DUP!' in line
+        }
+
+        return output_line
+
+
+def parse(data, raw=False, quiet=False, ignore_exceptions=False):
+    """
+    Main text parsing generator function. Returns an iterator object.
+
+    Parameters:
+
+        data:              (iterable)  line-based text data to parse (e.g. sys.stdin or str.splitlines())
+        raw:               (boolean)   output preprocessed JSON if True
+        quiet:             (boolean)   suppress warning messages if True
+        ignore_exceptions: (boolean)   ignore parsing exceptions if True
+
+    Yields:
+
+        Dictionary. Raw or processed structured data.
+
+    Returns:
+
+        Iterator object
+    """
+    s = _state()
+
+    if not quiet:
+        jc.utils.compatibility(__name__, info.compatible)
+
+    for line in data:
+
+        output_line = {}
+
+        try:
+            # skip blank lines
+            if line.strip() == '':
+                continue
+
+            # skip warning lines
+            if line.startswith('WARNING: '):
+                continue
+
+            # check for PATTERN
+            if line.startswith('PATTERN: '):
+                s.pattern = line.strip().split(': ')[1]
+                continue
+
+            # detect Linux vs. BSD ping
+            if not s.os_detected and line.strip().endswith('bytes of data.'):
+                s.os_detected = True
+                s.linux = True
+
+            elif not s.os_detected and '-->' in line:
+                s.os_detected = True
+                s.bsd = True
+
+            elif not s.os_detected and _ipv6_in(line) and line.strip().endswith('data bytes'):
+                s.os_detected = True
+                s.linux = True
+
+            elif not s.os_detected and not _ipv6_in(line) and line.strip().endswith('data bytes'):
+                s.os_detected = True
+                s.bsd = True
+
+            # parse the data
+            if s.os_detected and s.linux:
+                output_line = _linux_parse(line, s)
+
+            elif s.os_detected and s.bsd:
+                output_line = _bsd_parse(line, s)
+
+            else:
+                raise ParseError('Could not detect ping OS')
+
+            # yield the output line if it has data
+            if output_line:
+                yield stream_success(output_line, ignore_exceptions) if raw else stream_success(_process(output_line), ignore_exceptions)
+            else:
+                continue
+
+        except Exception as e:
+            yield stream_error(e, ignore_exceptions, line)

jc/parsers/route.py

@@ -22,7 +22,7 @@ Schema:
         "genmask":         string,
         "flags":           string,
         "flags_pretty": [
-                           string,
+                           string
         ]
         "metric":          integer,
         "ref":             integer,
@@ -73,7 +73,6 @@ Examples:
       }
     ]
 
-
     $ route -ee | jc --route -p -r
     [
       {
@@ -103,7 +102,6 @@ Examples:
         "irtt": "0"
       }
     ]
-
 """
 import jc.utils
 import jc.parsers.universal

jc/parsers/sfdisk.py

@@ -0,0 +1,407 @@
+"""jc - JSON CLI output utility `sfdisk` command output parser
+
+Supports the following `sfdisk` options:
+- `-l`
+- `-F`
+- `-d`   (deprecated - only for older versions of util-linux)
+- `-uM`  (deprecated - only for older versions of util-linux)
+- `-uC`  (deprecated - only for older versions of util-linux)
+- `-uS`  (deprecated - only for older versions of util-linux)
+- `-uB`  (deprecated - only for older versions of util-linux)
+
+Usage (cli):
+
+    # sfdisk -l | jc --sfdisk
+
+    or
+
+    # jc sfdisk -l
+
+Usage (module):
+
+    import jc.parsers.sfdisk
+    result = jc.parsers.sfdisk.parse(sfdisk_command_output)
+
+Schema:
+
+    [
+      {
+        "disk":                     string,
+        "disk_size":                string,
+        "free_disk_size":           string,
+        "bytes":                    integer,
+        "free_bytes":               integer,
+        "sectors":                  integer,
+        "free_sectors":             integer,
+        "cylinders":                integer,
+        "heads":                    integer,
+        "sectors_per_track":        integer,
+        "units":                    string,
+        "logical_sector_size":      integer,
+        "physical_sector_size":     integer,
+        "min_io_size":              integer,
+        "optimal_io_size":          integer,
+        "disk_label_type":          string,
+        "disk_identifier":          string,
+        "disk_model":               string,
+        "partitions": [
+          {
+            "device":               string,
+            "boot":                 boolean,
+            "start":                integer,
+            "end":                  integer,
+            "size":                 string,    # Note: will be integer when using deprecated -d sfdisk option
+            "cyls":                 integer,
+            "mib":                  integer,
+            "blocks":               integer,
+            "sectors":              integer,
+            "id":                   string,
+            "system":               string,
+            "type":                 string
+          }
+        ]
+      }
+    ]
+
+Examples:
+
+    # sfdisk -l | jc --sfdisk -p
+    [
+      {
+        "disk": "/dev/sda",
+        "cylinders": 2610,
+        "heads": 255,
+        "sectors_per_track": 63,
+        "units": "cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0",
+        "partitions": [
+          {
+            "device": "/dev/sda1",
+            "boot": true,
+            "start": 0,
+            "end": 130,
+            "cyls": 131,
+            "blocks": 1048576,
+            "id": "83",
+            "system": "Linux"
+          },
+          {
+            "device": "/dev/sda2",
+            "boot": false,
+            "start": 130,
+            "end": 2610,
+            "cyls": 2481,
+            "blocks": 19921920,
+            "id": "8e",
+            "system": "Linux LVM"
+          },
+          {
+            "device": "/dev/sda3",
+            "boot": false,
+            "start": 0,
+            "end": null,
+            "cyls": 0,
+            "blocks": 0,
+            "id": "0",
+            "system": "Empty"
+          },
+          {
+            "device": "/dev/sda4",
+            "boot": false,
+            "start": 0,
+            "end": null,
+            "cyls": 0,
+            "blocks": 0,
+            "id": "0",
+            "system": "Empty"
+          }
+        ]
+      },
+      {
+        "disk": "/dev/mapper/centos-root",
+        "cylinders": 2218,
+        "heads": 255,
+        "sectors_per_track": 63
+      },
+      {
+        "disk": "/dev/mapper/centos-swap",
+        "cylinders": 261,
+        "heads": 255,
+        "sectors_per_track": 63
+      }
+    ]
+
+    # sfdisk -l | jc --sfdisk -p -r
+    [
+      {
+        "disk": "/dev/sda",
+        "cylinders": "2610",
+        "heads": "255",
+        "sectors_per_track": "63",
+        "units": "cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0",
+        "partitions": [
+          {
+            "device": "/dev/sda1",
+            "boot": "*",
+            "start": "0+",
+            "end": "130-",
+            "cyls": "131-",
+            "blocks": "1048576",
+            "id": "83",
+            "system": "Linux"
+          },
+          {
+            "device": "/dev/sda2",
+            "boot": null,
+            "start": "130+",
+            "end": "2610-",
+            "cyls": "2481-",
+            "blocks": "19921920",
+            "id": "8e",
+            "system": "Linux LVM"
+          },
+          {
+            "device": "/dev/sda3",
+            "boot": null,
+            "start": "0",
+            "end": "-",
+            "cyls": "0",
+            "blocks": "0",
+            "id": "0",
+            "system": "Empty"
+          },
+          {
+            "device": "/dev/sda4",
+            "boot": null,
+            "start": "0",
+            "end": "-",
+            "cyls": "0",
+            "blocks": "0",
+            "id": "0",
+            "system": "Empty"
+          }
+        ]
+      },
+      {
+        "disk": "/dev/mapper/centos-root",
+        "cylinders": "2218",
+        "heads": "255",
+        "sectors_per_track": "63"
+      },
+      {
+        "disk": "/dev/mapper/centos-swap",
+        "cylinders": "261",
+        "heads": "255",
+        "sectors_per_track": "63"
+      }
+    ]
+"""
+import jc.utils
+import jc.parsers.universal
+
+
+class info():
+    """Provides parser metadata (version, author, etc.)"""
+    version = '1.1'
+    description = '`sfdisk` command parser'
+    author = 'Kelly Brazil'
+    author_email = 'kellyjonbrazil@gmail.com'
+    compatible = ['linux']
+    magic_commands = ['sfdisk']
+
+
+__version__ = info.version
+
+
+def _process(proc_data):
+    """
+    Final processing to conform to the schema.
+
+    Parameters:
+
+        proc_data:   (List of Dictionaries) raw structured data to process
+
+    Returns:
+
+        List of Dictionaries. Structured to conform to the schema.
+    """
+    int_list = ['cylinders', 'heads', 'sectors_per_track', 'start', 'end', 'cyls', 'mib',
+                'blocks', 'sectors', 'bytes', 'logical_sector_size', 'physical_sector_size',
+                'min_io_size', 'optimal_io_size', 'free_bytes', 'free_sectors']
+    bool_list = ['boot']
+
+    for entry in proc_data:
+        for key in entry:
+            if key in int_list:
+                entry[key] = jc.utils.convert_to_int(entry[key].replace('-', ''))
+
+        if 'partitions' in entry:
+            for p in entry['partitions']:
+                for key in p:
+                    # legacy conversion for -d option
+                    if key == 'size':
+                        if p[key].isnumeric():
+                            p[key] = jc.utils.convert_to_int(p[key])
+
+                    # normal conversions
+                    if key in int_list:
+                        p[key] = jc.utils.convert_to_int(p[key].replace('-', ''))
+                    if key in bool_list:
+                        p[key] = jc.utils.convert_to_bool(p[key])
+
+    return proc_data
+
+
+def parse(data, raw=False, quiet=False):
+    """
+    Main text parsing function
+
+    Parameters:
+
+        data:        (string)  text data to parse
+        raw:         (boolean) output preprocessed JSON if True
+        quiet:       (boolean) suppress warning messages if True
+
+    Returns:
+
+        List of Dictionaries. Raw or processed structured data.
+    """
+    if not quiet:
+        jc.utils.compatibility(__name__, info.compatible)
+
+    raw_output = []
+    item = {}
+    partitions = []
+    option = ''
+    section = ''
+
+    if jc.utils.has_data(data):
+
+        for line in data.splitlines():
+            # deprecated - only for older versions of util-linux
+            if line.startswith('# partition table of'):
+                if item:
+                    raw_output.append(item)
+
+                item = {}
+                partitions = []
+                option = 'd'
+                item['disk'] = line.split()[4]
+                continue
+
+            # deprecated - only for older versions of util-linux
+            if option == 'd':
+                if line.startswith('unit: '):
+                    item['units'] = line.split()[1]
+                    section = 'partitions'
+                    continue
+
+                if section == 'partitions' and line:
+                    part = {}
+                    part['device'] = line.split()[0]
+                    line = line.replace(',', ' ').replace('=', ' ')
+                    part['start'] = line.split()[3]
+                    part['size'] = line.split()[5]
+                    part['id'] = line.split()[7]
+                    part['boot'] = '*' if 'bootable' in line else None
+                    partitions.append(part)
+                    item['partitions'] = partitions
+                    continue
+
+            else:
+                # older versions of util-linux
+                # Disk /dev/sda: 2610 cylinders, 255 heads, 63 sectors/track
+                if line.startswith('Disk ') and 'sectors/track' in line:
+                    if item:
+                        raw_output.append(item)
+
+                    item = {}
+                    partitions = []
+                    line = line.replace(':', '').replace(',', '')
+                    fields = line.split()
+                    item['disk'] = fields[1]
+                    item['cylinders'] = fields[2]
+                    item['heads'] = fields[4]
+                    item['sectors_per_track'] = fields[6]
+                    continue
+
+                # util-linux v2.32.0+ (?)
+                # Disk /dev/sda: 20 GiB, 21474836480 bytes, 41943040 sectors
+                if line.startswith('Disk ') and line.endswith('sectors'):
+                    if item:
+                        raw_output.append(item)
+
+                    item = {}
+                    partitions = []
+                    line = line.replace(':', '').replace(',', '')
+                    fields = line.split()
+                    item['disk'] = fields[1]
+                    item['disk_size'] = ' '.join(fields[2:4])
+                    item['bytes'] = fields[4]
+                    item['sectors'] = fields[6]
+                    continue
+
+                if line.startswith('Disk model: '):
+                    item['disk_model'] = line.split(':', maxsplit=1)[1].strip()
+                    continue
+
+                if line.startswith('Sector size (logical/physical)'):
+                    fields = line.split()
+                    item['logical_sector_size'] = fields[3]
+                    item['physical_sector_size'] = fields[6]
+                    continue
+
+                if line.startswith('I/O size (minimum/optimal)'):
+                    fields = line.split()
+                    item['min_io_size'] = fields[3]
+                    item['optimal_io_size'] = fields[6]
+                    continue
+
+                if line.startswith('Disklabel type'):
+                    item['disk_label_type'] = line.split(':', maxsplit=1)[1].strip()
+                    continue
+
+                if line.startswith('Disk identifier'):
+                    item['disk_identifier'] = line.split(':', maxsplit=1)[1].strip()
+                    continue
+
+                if line.startswith('Units: '):
+                    item['units'] = line.split(':')[1].strip()
+                    continue
+
+                # sfdisk -F
+                if line.startswith('Unpartitioned space'):
+                    line = line.replace(':', '').replace(',', '')
+                    fields = line.split()
+                    item['disk'] = fields[2]
+                    item['free_disk_size'] = ' '.join(fields[3:5])
+                    item['free_bytes'] = fields[5]
+                    item['free_sectors'] = fields[7]
+                    continue
+
+                # partition lines
+                if 'Start' in line and 'End' in line and ('Sectors' in line or 'Device' in line):
+                    section = 'partitions'
+                    partitions.append(line.lower().replace('#', ' '))
+                    continue
+
+                if section == 'partitions' and line:
+                    partitions.append(line)
+                    continue
+
+                if section == 'partitions' and line == '':
+                    item['partitions'] = jc.parsers.universal.sparse_table_parse(partitions)
+                    section = ''
+                    partitions = []
+                    continue
+
+        # get final partitions if there are any left over
+        if section == 'partitions' and option != 'd' and partitions:
+            item['partitions'] = jc.parsers.universal.sparse_table_parse(partitions)
+
+        if item:
+            raw_output.append(item)
+
+    if raw:
+        return raw_output
+    else:
+        return _process(raw_output)

jc/parsers/stat.py

@@ -169,7 +169,7 @@ import jc.utils
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.8'
+    version = '1.9'
     description = '`stat` command parser'
     author = 'Kelly Brazil'
     author_email = 'kellyjonbrazil@gmail.com'
@@ -322,7 +322,7 @@ def parse(data, raw=False, quiet=False):
             for line in cleandata:
                 value = shlex.split(line)
                 output_line = {
-                    'file': value[15],
+                    'file': ' '.join(value[15:]),
                     'unix_device': value[0],
                     'inode': value[1],
                     'flags': value[2],

jc/parsers/traceroute.py

@@ -389,7 +389,7 @@ def parse(data, raw=False, quiet=False):
 
             # print warning to STDERR
             if not quiet:
-                jc.utils.warning_message('No header row found. For destination info redirect STDERR to STDOUT')
+                jc.utils.warning_message(['No header row found. For destination info redirect STDERR to STDOUT'])
 
         data = '\n'.join(new_data)
 

jc/parsers/vmstat.py

@@ -0,0 +1,279 @@
+"""jc - JSON CLI output utility `vmstat` command output parser
+
+Options supported: `-a`, `-w`, `-d`, `-t`
+
+The `epoch` calculated timestamp field is naive (i.e. based on the local time of the system the parser is run on)
+
+The `epoch_utc` calculated timestamp field is timezone-aware and is only available if the timezone field is UTC.
+
+Usage (cli):
+
+    $ vmstat | jc --vmstat
+
+    or
+
+    $ jc vmstat
+
+Usage (module):
+
+    import jc.parsers.vmstat
+    result = jc.parsers.vmstat.parse(vmstat_command_output)
+
+Schema:
+
+    [
+      {
+        "runnable_procs":                    integer,
+        "uninterruptible_sleeping_procs":    integer,
+        "virtual_mem_used":                  integer,
+        "free_mem":                          integer,
+        "buffer_mem":                        integer,
+        "cache_mem":                         integer,
+        "inactive_mem":                      integer,
+        "active_mem":                        integer,
+        "swap_in":                           integer,
+        "swap_out":                          integer,
+        "blocks_in":                         integer,
+        "blocks_out":                        integer,
+        "interrupts":                        integer,
+        "context_switches":                  integer,
+        "user_time":                         integer,
+        "system_time":                       integer,
+        "idle_time":                         integer,
+        "io_wait_time":                      integer,
+        "stolen_time":                       integer,
+        "disk":                              string,
+        "total_reads":                       integer,
+        "merged_reads":                      integer,
+        "sectors_read":                      integer,
+        "reading_ms":                        integer,
+        "total_writes":                      integer,
+        "merged_writes":                     integer,
+        "sectors_written":                   integer,
+        "writing_ms":                        integer,
+        "current_io":                        integer,
+        "io_seconds":                        integer,
+        "timestamp":                         string,
+        "timezone":                          string,
+        "epoch":                             integer,     # naive timestamp if -t flag is used
+        "epoch_utc":                         integer      # aware timestamp if -t flag is used and UTC TZ
+      }
+    ]
+
+Examples:
+
+    $ vmstat | jc --vmstat -p
+    [
+      {
+        "runnable_procs": 2,
+        "uninterruptible_sleeping_procs": 0,
+        "virtual_mem_used": 0,
+        "free_mem": 2794468,
+        "buffer_mem": 2108,
+        "cache_mem": 741208,
+        "inactive_mem": null,
+        "active_mem": null,
+        "swap_in": 0,
+        "swap_out": 0,
+        "blocks_in": 1,
+        "blocks_out": 3,
+        "interrupts": 29,
+        "context_switches": 57,
+        "user_time": 0,
+        "system_time": 0,
+        "idle_time": 99,
+        "io_wait_time": 0,
+        "stolen_time": 0,
+        "timestamp": null,
+        "timezone": null
+      }
+    ]
+
+    $ vmstat | jc --vmstat -p -r
+    [
+      {
+        "runnable_procs": "2",
+        "uninterruptible_sleeping_procs": "0",
+        "virtual_mem_used": "0",
+        "free_mem": "2794468",
+        "buffer_mem": "2108",
+        "cache_mem": "741208",
+        "inactive_mem": null,
+        "active_mem": null,
+        "swap_in": "0",
+        "swap_out": "0",
+        "blocks_in": "1",
+        "blocks_out": "3",
+        "interrupts": "29",
+        "context_switches": "57",
+        "user_time": "0",
+        "system_time": "0",
+        "idle_time": "99",
+        "io_wait_time": "0",
+        "stolen_time": "0",
+        "timestamp": null,
+        "timezone": null
+      }
+    ]
+"""
+import jc.utils
+
+
+class info():
+    """Provides parser metadata (version, author, etc.)"""
+    version = '1.0'
+    description = '`vmstat` command parser'
+    author = 'Kelly Brazil'
+    author_email = 'kellyjonbrazil@gmail.com'
+    # details = 'enter any other details here'
+
+    # compatible options: linux, darwin, cygwin, win32, aix, freebsd
+    compatible = ['linux']
+    magic_commands = ['vmstat']
+
+
+__version__ = info.version
+
+
+def _process(proc_data):
+    """
+    Final processing to conform to the schema.
+
+    Parameters:
+
+        proc_data:   (List of Dictionaries) raw structured data to process
+
+    Returns:
+
+        List of Dictionaries. Structured to conform to the schema.
+    """
+
+    int_list = ['runnable_procs', 'uninterruptible_sleeping_procs', 'virtual_mem_used', 'free_mem', 'buffer_mem',
+                'cache_mem', 'inactive_mem', 'active_mem', 'swap_in', 'swap_out', 'blocks_in', 'blocks_out',
+                'interrupts', 'context_switches', 'user_time', 'system_time', 'idle_time', 'io_wait_time',
+                'stolen_time', 'total_reads', 'merged_reads', 'sectors_read', 'reading_ms', 'total_writes',
+                'merged_writes', 'sectors_written', 'writing_ms', 'current_io', 'io_seconds']
+
+    for entry in proc_data:
+        for key in entry:
+            if key in int_list:
+                entry[key] = jc.utils.convert_to_int(entry[key])
+
+        if entry['timestamp']:
+            ts = jc.utils.timestamp(f'{entry["timestamp"]} {entry["timezone"]}')
+            entry['epoch'] = ts.naive
+            entry['epoch_utc'] = ts.utc
+
+    return proc_data
+
+
+def parse(data, raw=False, quiet=False):
+    """
+    Main text parsing function
+
+    Parameters:
+
+        data:        (string)  text data to parse
+        raw:         (boolean) output preprocessed JSON if True
+        quiet:       (boolean) suppress warning messages if True
+
+    Returns:
+
+        List of Dictionaries. Raw or processed structured data.
+    """
+    if not quiet:
+        jc.utils.compatibility(__name__, info.compatible)
+
+    raw_output = []
+    output_line = {}
+    procs = None
+    buff_cache = None
+    disk = None
+    tstamp = None
+    tz = None
+
+    if jc.utils.has_data(data):
+
+        for line in filter(None, data.splitlines()):
+
+            # detect output type
+            if not procs and not disk and line.startswith('procs'):
+                procs = True
+                tstamp = '-timestamp-' in line
+                continue
+
+            if not procs and not disk and line.startswith('disk'):
+                disk = True
+                tstamp = '-timestamp-' in line
+                continue
+
+            # skip header rows
+            if (procs or disk) and (line.startswith('procs') or line.startswith('disk')):
+                continue
+
+            if 'swpd' in line and 'free' in line and 'buff' in line and 'cache' in line:
+                buff_cache = True
+                tz = line.strip().split()[-1] if tstamp else None
+                continue
+
+            if 'swpd' in line and 'free' in line and 'inact' in line and 'active' in line:
+                buff_cache = False
+                tz = line.strip().split()[-1] if tstamp else None
+                continue
+
+            if 'total' in line and 'merged' in line and 'sectors' in line:
+                tz = line.strip().split()[-1] if tstamp else None
+                continue
+
+            # line parsing
+            if procs:
+                line_list = line.strip().split(maxsplit=17)
+
+                output_line = {
+                    'runnable_procs': line_list[0],
+                    'uninterruptible_sleeping_procs': line_list[1],
+                    'virtual_mem_used': line_list[2],
+                    'free_mem': line_list[3],
+                    'buffer_mem': line_list[4] if buff_cache else None,
+                    'cache_mem': line_list[5] if buff_cache else None,
+                    'inactive_mem': line_list[4] if not buff_cache else None,
+                    'active_mem': line_list[5] if not buff_cache else None,
+                    'swap_in': line_list[6],
+                    'swap_out': line_list[7],
+                    'blocks_in': line_list[8],
+                    'blocks_out': line_list[9],
+                    'interrupts': line_list[10],
+                    'context_switches': line_list[11],
+                    'user_time': line_list[12],
+                    'system_time': line_list[13],
+                    'idle_time': line_list[14],
+                    'io_wait_time': line_list[15],
+                    'stolen_time': line_list[16],
+                    'timestamp': line_list[17] if tstamp else None,
+                    'timezone': tz or None
+                }
+
+                raw_output.append(output_line)
+
+            if disk:
+                line_list = line.strip().split(maxsplit=11)
+
+                output_line = {
+                    'disk': line_list[0],
+                    'total_reads': line_list[1],
+                    'merged_reads': line_list[2],
+                    'sectors_read': line_list[3],
+                    'reading_ms': line_list[4],
+                    'total_writes': line_list[5],
+                    'merged_writes': line_list[6],
+                    'sectors_written': line_list[7],
+                    'writing_ms': line_list[8],
+                    'current_io': line_list[9],
+                    'io_seconds': line_list[10],
+                    'timestamp': line_list[11] if tstamp else None,
+                    'timezone': tz or None
+                }
+
+                raw_output.append(output_line)
+
+    return raw_output if raw else _process(raw_output)

jc/parsers/vmstat_s.py

@@ -0,0 +1,242 @@
+"""jc - JSON CLI output utility `vmstat` command output streaming parser
+
+Options supported: `-a`, `-w`, `-d`, `-t`
+
+The `epoch` calculated timestamp field is naive (i.e. based on the local time of the system the parser is run on)
+
+The `epoch_utc` calculated timestamp field is timezone-aware and is only available if the timezone field is UTC.
+
+Usage (cli):
+
+    $ vmstat | jc --vmstat-s
+
+Usage (module):
+
+    import jc.parsers.vmstat_s
+    result = jc.parsers.vmstat_s.parse(vmstat_command_output.splitlines())    # result is an iterable object
+    for item in result:
+        # do something
+
+Schema:
+
+    {
+      "runnable_procs":                    integer,
+      "uninterruptible_sleeping_procs":    integer,
+      "virtual_mem_used":                  integer,
+      "free_mem":                          integer,
+      "buffer_mem":                        integer,
+      "cache_mem":                         integer,
+      "inactive_mem":                      integer,
+      "active_mem":                        integer,
+      "swap_in":                           integer,
+      "swap_out":                          integer,
+      "blocks_in":                         integer,
+      "blocks_out":                        integer,
+      "interrupts":                        integer,
+      "context_switches":                  integer,
+      "user_time":                         integer,
+      "system_time":                       integer,
+      "idle_time":                         integer,
+      "io_wait_time":                      integer,
+      "stolen_time":                       integer,
+      "disk":                              string,
+      "total_reads":                       integer,
+      "merged_reads":                      integer,
+      "sectors_read":                      integer,
+      "reading_ms":                        integer,
+      "total_writes":                      integer,
+      "merged_writes":                     integer,
+      "sectors_written":                   integer,
+      "writing_ms":                        integer,
+      "current_io":                        integer,
+      "io_seconds":                        integer,
+      "timestamp":                         string,
+      "timezone":                          string,
+      "epoch":                             integer,     # naive timestamp if -t flag is used
+      "epoch_utc":                         integer      # aware timestamp if -t flag is used and UTC TZ
+      "_jc_meta":                                       # This object only exists if using -qq or ignore_exceptions=True
+        {
+          "success":                       booean,      # true if successfully parsed, false if error
+          "error":                         string,      # exists if "success" is false
+          "line":                          string       # exists if "success" is false
+        }
+    }
+
+Examples:
+
+    $ vmstat | jc --vmstat-s
+    {"runnable_procs":2,"uninterruptible_sleeping_procs":0,"virtual_mem_used":0,"free_mem":2794468,"buffer_mem":2108,"cache_mem":741208,"inactive_mem":null,"active_mem":null,"swap_in":0,"swap_out":0,"blocks_in":1,"blocks_out":3,"interrupts":29,"context_switches":57,"user_time":0,"system_time":0,"idle_time":99,"io_wait_time":0,"stolen_time":0,"timestamp":null,"timezone":null}
+    ...
+
+    $ vmstat | jc --vmstat-s -r
+    {"runnable_procs":"2","uninterruptible_sleeping_procs":"0","virtual_mem_used":"0","free_mem":"2794468","buffer_mem":"2108","cache_mem":"741208","inactive_mem":null,"active_mem":null,"swap_in":"0","swap_out":"0","blocks_in":"1","blocks_out":"3","interrupts":"29","context_switches":"57","user_time":"0","system_time":"0","idle_time":"99","io_wait_time":"0","stolen_time":"0","timestamp":null,"timezone":null}
+    ...
+"""
+import jc.utils
+from jc.utils import stream_success, stream_error
+from jc.exceptions import ParseError
+
+
+class info():
+    """Provides parser metadata (version, author, etc.)"""
+    version = '0.5'
+    description = '`vmstat` command streaming parser'
+    author = 'Kelly Brazil'
+    author_email = 'kellyjonbrazil@gmail.com'
+
+    # compatible options: linux, darwin, cygwin, win32, aix, freebsd
+    compatible = ['linux']
+    streaming = True
+
+
+__version__ = info.version
+
+
+def _process(proc_data):
+    """
+    Final processing to conform to the schema.
+
+    Parameters:
+
+        proc_data:   (Dictionary) raw structured data to process
+
+    Returns:
+
+        Dictionary. Structured data to conform to the schema.
+    """
+    int_list = ['runnable_procs', 'uninterruptible_sleeping_procs', 'virtual_mem_used', 'free_mem', 'buffer_mem',
+                'cache_mem', 'inactive_mem', 'active_mem', 'swap_in', 'swap_out', 'blocks_in', 'blocks_out',
+                'interrupts', 'context_switches', 'user_time', 'system_time', 'idle_time', 'io_wait_time',
+                'stolen_time', 'total_reads', 'merged_reads', 'sectors_read', 'reading_ms', 'total_writes',
+                'merged_writes', 'sectors_written', 'writing_ms', 'current_io', 'io_seconds']
+
+    for key in proc_data:
+        if key in int_list:
+            proc_data[key] = jc.utils.convert_to_int(proc_data[key])
+
+    if proc_data['timestamp']:
+        ts = jc.utils.timestamp(f'{proc_data["timestamp"]} {proc_data["timezone"]}')
+        proc_data['epoch'] = ts.naive
+        proc_data['epoch_utc'] = ts.utc
+
+    return proc_data
+
+
+def parse(data, raw=False, quiet=False, ignore_exceptions=False):
+    """
+    Main text parsing generator function. Returns an iterator object.
+
+    Parameters:
+
+        data:              (iterable)  line-based text data to parse (e.g. sys.stdin or str.splitlines())
+        raw:               (boolean)   output preprocessed JSON if True
+        quiet:             (boolean)   suppress warning messages if True
+        ignore_exceptions: (boolean)   ignore parsing exceptions if True
+
+    Yields:
+
+        Dictionary. Raw or processed structured data.
+
+    Returns:
+
+        Iterator object
+    """
+    if not quiet:
+        jc.utils.compatibility(__name__, info.compatible)
+
+    output_line = {}
+    procs = None
+    buff_cache = None
+    disk = None
+    tstamp = None
+    tz = None
+
+    for line in data:
+        try:
+            # skip blank lines
+            if line.strip() == '':
+                continue
+
+            # detect output type
+            if not procs and not disk and line.startswith('procs'):
+                procs = True
+                tstamp = '-timestamp-' in line
+                continue
+
+            if not procs and not disk and line.startswith('disk'):
+                disk = True
+                tstamp = '-timestamp-' in line
+                continue
+
+            # skip header rows
+            if (procs or disk) and (line.startswith('procs') or line.startswith('disk')):
+                continue
+
+            if 'swpd' in line and 'free' in line and 'buff' in line and 'cache' in line:
+                buff_cache = True
+                tz = line.strip().split()[-1] if tstamp else None
+                continue
+
+            if 'swpd' in line and 'free' in line and 'inact' in line and 'active' in line:
+                buff_cache = False
+                tz = line.strip().split()[-1] if tstamp else None
+                continue
+
+            if 'total' in line and 'merged' in line and 'sectors' in line:
+                tz = line.strip().split()[-1] if tstamp else None
+                continue
+
+            # line parsing
+            if procs:
+                line_list = line.strip().split(maxsplit=17)
+
+                output_line = {
+                    'runnable_procs': line_list[0],
+                    'uninterruptible_sleeping_procs': line_list[1],
+                    'virtual_mem_used': line_list[2],
+                    'free_mem': line_list[3],
+                    'buffer_mem': line_list[4] if buff_cache else None,
+                    'cache_mem': line_list[5] if buff_cache else None,
+                    'inactive_mem': line_list[4] if not buff_cache else None,
+                    'active_mem': line_list[5] if not buff_cache else None,
+                    'swap_in': line_list[6],
+                    'swap_out': line_list[7],
+                    'blocks_in': line_list[8],
+                    'blocks_out': line_list[9],
+                    'interrupts': line_list[10],
+                    'context_switches': line_list[11],
+                    'user_time': line_list[12],
+                    'system_time': line_list[13],
+                    'idle_time': line_list[14],
+                    'io_wait_time': line_list[15],
+                    'stolen_time': line_list[16],
+                    'timestamp': line_list[17] if tstamp else None,
+                    'timezone': tz or None
+                }
+
+            if disk:
+                line_list = line.strip().split(maxsplit=11)
+
+                output_line = {
+                    'disk': line_list[0],
+                    'total_reads': line_list[1],
+                    'merged_reads': line_list[2],
+                    'sectors_read': line_list[3],
+                    'reading_ms': line_list[4],
+                    'total_writes': line_list[5],
+                    'merged_writes': line_list[6],
+                    'sectors_written': line_list[7],
+                    'writing_ms': line_list[8],
+                    'current_io': line_list[9],
+                    'io_seconds': line_list[10],
+                    'timestamp': line_list[11] if tstamp else None,
+                    'timezone': tz or None
+                }
+
+            if output_line:
+                yield stream_success(output_line, ignore_exceptions) if raw else stream_success(_process(output_line), ignore_exceptions)
+            else:
+                raise ParseError('Not vmstat data')
+
+        except Exception as e:
+            yield stream_error(e, ignore_exceptions, line)

jc/utils.py

@@ -2,41 +2,77 @@
 import sys
 import re
 import locale
+import shutil
 from datetime import datetime, timezone
+from textwrap import TextWrapper
 
 
-def warning_message(message):
+def warning_message(message_lines):
     """
-    Prints a warning message for non-fatal issues
+    Prints warning message for non-fatal issues. The first line is prepended with
+    'jc:  Warning - ' and subsequent lines are indented. Wraps text as needed based
+    on the terminal width.
 
     Parameters:
 
-        message:        (string) text of message
+        message:   (list) list of string lines
 
     Returns:
 
         None - just prints output to STDERR
     """
+    # this is for backwards compatibility with existing custom parsers
+    if isinstance(message_lines, str):
+        message_lines = [message_lines]
 
-    error_string = f'jc:  Warning - {message}'
-    print(error_string, file=sys.stderr)
+    columns = shutil.get_terminal_size().columns
+
+    first_wrapper = TextWrapper(width=columns, subsequent_indent=' ' * 15)
+    next_wrapper = TextWrapper(width=columns, initial_indent=' ' * 15,
+                               subsequent_indent=' ' * 19)
+
+    first_line = message_lines.pop(0)
+    first_str = f'jc:  Warning - {first_line}'
+    first_str = first_wrapper.fill(first_str)
+    print(first_str, file=sys.stderr)
+
+    for line in message_lines:
+        if line == '':
+            continue
+        message = next_wrapper.fill(line)
+        print(message, file=sys.stderr)
 
 
-def error_message(message):
+def error_message(message_lines):
     """
-    Prints an error message for fatal issues
+    Prints an error message for fatal issues. The first line is prepended with
+    'jc:  Error - ' and subsequent lines are indented. Wraps text as needed based
+    on the terminal width.
 
     Parameters:
 
-        message:        (string) text of message
+        message:   (list) list of string lines
 
     Returns:
 
         None - just prints output to STDERR
     """
+    columns = shutil.get_terminal_size().columns
 
-    error_string = f'jc:  Error - {message}'
-    print(error_string, file=sys.stderr)
+    first_wrapper = TextWrapper(width=columns, subsequent_indent=' ' * 13)
+    next_wrapper = TextWrapper(width=columns, initial_indent=' ' * 13,
+                               subsequent_indent=' ' * 17)
+
+    first_line = message_lines.pop(0)
+    first_str = f'jc:  Error - {first_line}'
+    first_str = first_wrapper.fill(first_str)
+    print(first_str, file=sys.stderr)
+
+    for line in message_lines:
+        if line == '':
+            continue
+        message = next_wrapper.fill(line)
+        print(message, file=sys.stderr)
 
 
 def compatibility(mod_name, compatible):
@@ -64,8 +100,8 @@ def compatibility(mod_name, compatible):
     if not platform_found:
         mod = mod_name.split('.')[-1]
         compat_list = ', '.join(compatible)
-        warning_message(f'{mod} parser not compatible with your OS ({sys.platform}).\n'
-                        f'               Compatible platforms: {compat_list}')
+        warning_message([f'{mod} parser not compatible with your OS ({sys.platform}).',
+                         f'Compatible platforms: {compat_list}'])
 
 
 def has_data(data):
@@ -80,7 +116,7 @@ def has_data(data):
 
         Boolean      True if input string (data) contains non-whitespace characters, otherwise False
     """
-    return True if data and not data.isspace() else False
+    return bool(data and not data.isspace())
 
 
 def convert_to_int(value):
@@ -154,7 +190,7 @@ def convert_to_bool(value):
     #   if float converts, then bool the result
     #   if float does not convert then look for truthy string and bool True
     #   else False
-    truthy = ['y', 'yes', 'true']
+    truthy = ['y', 'yes', 'true', '*']
 
     if isinstance(value, (int, float)):
         return bool(value)
@@ -168,11 +204,37 @@ def convert_to_bool(value):
             pass
 
         if value:
-            return True if value.lower() in truthy else False
+            return value.lower() in truthy
 
     return False
 
 
+def stream_success(output_line, ignore_exceptions):
+    """Add `_jc_meta` object to output line if `ignore_exceptions=True`"""
+    if ignore_exceptions:
+        output_line.update({'_jc_meta': {'success': True}})
+
+    return output_line
+
+
+def stream_error(e, ignore_exceptions, line):
+    """Reraise the stream exception with annotation or print an error `_jc_meta`
+       field if `ignore_exceptions=True`
+    """
+    if not ignore_exceptions:
+        e.args = (str(e) + '... Use the ignore_exceptions option (-qq) to ignore streaming parser errors.',)
+        raise e
+
+    return {
+        '_jc_meta':
+            {
+                'success': False,
+                'error': f'{e.__class__.__name__}: {e}',
+                'line': line.strip()
+            }
+    }
+
+
 class timestamp:
     """
     Input a date-time text string of several formats and convert to a naive or timezone-aware epoch timestamp in UTC
@@ -184,7 +246,7 @@ class timestamp:
     Attributes:
 
         string              (str)   the input datetime string
-        format              (int)   the format rule that was used to decode the datetime string
+        format              (int)   the format rule that was used to decode the datetime string. None if conversion fails
         naive               (int)   timestamp based on locally configured timezone. None if conversion fails
         utc                 (int)   aware timestamp only if UTC timezone detected in datetime string. None if conversion fails
     """
@@ -245,10 +307,8 @@ class timestamp:
         if 'UTC' in data:
             utc_tz = True
             if 'UTC+' in data or 'UTC-' in data:
-                if 'UTC+0000' in data or 'UTC-0000' in data:
-                    utc_tz = True
-                else:
-                    utc_tz = False
+                utc_tz = bool('UTC+0000' in data or 'UTC-0000' in data)
+
         elif '+0000' in data or '-0000' in data:
             utc_tz = True
 
@@ -267,6 +327,8 @@ class timestamp:
             {'id': 7000, 'format': '%a %b %d %H:%M:%S %Z %Y', 'locale': None},  # C locale format (found in date cli): Wed Mar 24 11:11:30 UTC 2021
             {'id': 7100, 'format': '%b %d %H:%M:%S %Y', 'locale': None},  # C locale format (found in stat cli output - osx): # Mar 29 11:49:05 2021
             {'id': 7200, 'format': '%Y-%m-%d %H:%M:%S.%f %z', 'locale': None},  # C locale format (found in stat cli output - linux): 2019-08-13 18:13:43.555604315 -0400
+            {'id': 7250, 'format': '%Y-%m-%d %H:%M:%S', 'locale': None},  # C locale format with non-UTC tz (found in modified vmstat cli output): # 2021-09-16 20:32:28 PDT
+            {'id': 7255, 'format': '%Y-%m-%d %H:%M:%S %Z', 'locale': None},  # C locale format (found in modified vmstat cli output): # 2021-09-16 20:32:28 UTC
             {'id': 7300, 'format': '%a %Y-%m-%d %H:%M:%S %Z', 'locale': None},  # C locale format (found in timedatectl cli output): # Wed 2020-03-11 00:53:21 UTC
             # attempt locale changes last
             {'id': 8000, 'format': '%a %d %b %Y %H:%M:%S %Z', 'locale': ''},  # current locale format (found in upower cli output): # mar. 23 mars 2021 23:12:11 UTC

man/jc.1

@@ -0,0 +1,609 @@
+.TH jc 1 2021-09-26 1.17.0 "JSON CLI output utility"
+.SH NAME
+jc \- JSONifies the output of many CLI tools and file-types
+.SH SYNOPSIS
+COMMAND | jc PARSER [OPTIONS]
+
+or "Magic" syntax:
+
+jc [OPTIONS] COMMAND
+
+.SH DESCRIPTION
+jc JSONifies the output of many CLI tools and file-types for easier parsing in scripts. jc accepts piped input from \fBSTDIN\fP and outputs a JSON representation of the previous command's output to \fBSTDOUT\fP. Alternatively, the "Magic" syntax can be used by prepending jc to the command to be converted. Options can be passed to jc immediately before the command is given. (Note: "Magic" syntax does not support shell builtins or command aliases)
+
+.SH OPTIONS
+.B
+Parsers:
+.RS
+
+
+.TP
+.B
+\fB--acpi\fP
+`acpi` command parser
+
+.TP
+.B
+\fB--airport\fP
+`airport -I` command parser
+
+.TP
+.B
+\fB--airport-s\fP
+`airport -s` command parser
+
+.TP
+.B
+\fB--arp\fP
+`arp` command parser
+
+.TP
+.B
+\fB--blkid\fP
+`blkid` command parser
+
+.TP
+.B
+\fB--cksum\fP
+`cksum` and `sum` command parser
+
+.TP
+.B
+\fB--crontab\fP
+`crontab` command and file parser
+
+.TP
+.B
+\fB--crontab-u\fP
+`crontab` file parser with user support
+
+.TP
+.B
+\fB--csv\fP
+CSV file parser
+
+.TP
+.B
+\fB--date\fP
+`date` command parser
+
+.TP
+.B
+\fB--df\fP
+`df` command parser
+
+.TP
+.B
+\fB--dig\fP
+`dig` command parser
+
+.TP
+.B
+\fB--dir\fP
+`dir` command parser
+
+.TP
+.B
+\fB--dmidecode\fP
+`dmidecode` command parser
+
+.TP
+.B
+\fB--dpkg-l\fP
+`dpkg -l` command parser
+
+.TP
+.B
+\fB--du\fP
+`du` command parser
+
+.TP
+.B
+\fB--env\fP
+`env` command parser
+
+.TP
+.B
+\fB--file\fP
+`file` command parser
+
+.TP
+.B
+\fB--finger\fP
+`finger` command parser
+
+.TP
+.B
+\fB--free\fP
+`free` command parser
+
+.TP
+.B
+\fB--fstab\fP
+`/etc/fstab` file parser
+
+.TP
+.B
+\fB--group\fP
+`/etc/group` file parser
+
+.TP
+.B
+\fB--gshadow\fP
+`/etc/gshadow` file parser
+
+.TP
+.B
+\fB--hash\fP
+`hash` command parser
+
+.TP
+.B
+\fB--hashsum\fP
+hashsum command parser (`md5sum`, `shasum`, etc.)
+
+.TP
+.B
+\fB--hciconfig\fP
+`hciconfig` command parser
+
+.TP
+.B
+\fB--history\fP
+`history` command parser
+
+.TP
+.B
+\fB--hosts\fP
+`/etc/hosts` file parser
+
+.TP
+.B
+\fB--id\fP
+`id` command parser
+
+.TP
+.B
+\fB--ifconfig\fP
+`ifconfig` command parser
+
+.TP
+.B
+\fB--ini\fP
+INI file parser
+
+.TP
+.B
+\fB--iptables\fP
+`iptables` command parser
+
+.TP
+.B
+\fB--iw-scan\fP
+`iw dev [device] scan` command parser
+
+.TP
+.B
+\fB--jobs\fP
+`jobs` command parser
+
+.TP
+.B
+\fB--kv\fP
+Key/Value file parser
+
+.TP
+.B
+\fB--last\fP
+`last` and `lastb` command parser
+
+.TP
+.B
+\fB--ls\fP
+`ls` command parser
+
+.TP
+.B
+\fB--ls-s\fP
+`ls` command streaming parser
+
+.TP
+.B
+\fB--lsblk\fP
+`lsblk` command parser
+
+.TP
+.B
+\fB--lsmod\fP
+`lsmod` command parser
+
+.TP
+.B
+\fB--lsof\fP
+`lsof` command parser
+
+.TP
+.B
+\fB--mount\fP
+`mount` command parser
+
+.TP
+.B
+\fB--netstat\fP
+`netstat` command parser
+
+.TP
+.B
+\fB--ntpq\fP
+`ntpq -p` command parser
+
+.TP
+.B
+\fB--passwd\fP
+`/etc/passwd` file parser
+
+.TP
+.B
+\fB--ping\fP
+`ping` and `ping6` command parser
+
+.TP
+.B
+\fB--ping-s\fP
+`ping` and `ping6` command streaming parser
+
+.TP
+.B
+\fB--pip-list\fP
+`pip list` command parser
+
+.TP
+.B
+\fB--pip-show\fP
+`pip show` command parser
+
+.TP
+.B
+\fB--ps\fP
+`ps` command parser
+
+.TP
+.B
+\fB--route\fP
+`route` command parser
+
+.TP
+.B
+\fB--rpm-qi\fP
+`rpm -qi` command parser
+
+.TP
+.B
+\fB--sfdisk\fP
+`sfdisk` command parser
+
+.TP
+.B
+\fB--shadow\fP
+`/etc/shadow` file parser
+
+.TP
+.B
+\fB--ss\fP
+`ss` command parser
+
+.TP
+.B
+\fB--stat\fP
+`stat` command parser
+
+.TP
+.B
+\fB--sysctl\fP
+`sysctl` command parser
+
+.TP
+.B
+\fB--systemctl\fP
+`systemctl` command parser
+
+.TP
+.B
+\fB--systemctl-lj\fP
+`systemctl list-jobs` command parser
+
+.TP
+.B
+\fB--systemctl-ls\fP
+`systemctl list-sockets` command parser
+
+.TP
+.B
+\fB--systemctl-luf\fP
+`systemctl list-unit-files` command parser
+
+.TP
+.B
+\fB--systeminfo\fP
+`systeminfo` command parser
+
+.TP
+.B
+\fB--time\fP
+`/usr/bin/time` command parser
+
+.TP
+.B
+\fB--timedatectl\fP
+`timedatectl status` command parser
+
+.TP
+.B
+\fB--tracepath\fP
+`tracepath` and `tracepath6` command parser
+
+.TP
+.B
+\fB--traceroute\fP
+`traceroute` and `traceroute6` command parser
+
+.TP
+.B
+\fB--ufw\fP
+`ufw status` command parser
+
+.TP
+.B
+\fB--ufw-appinfo\fP
+`ufw app info [application]` command parser
+
+.TP
+.B
+\fB--uname\fP
+`uname -a` command parser
+
+.TP
+.B
+\fB--upower\fP
+`upower` command parser
+
+.TP
+.B
+\fB--uptime\fP
+`uptime` command parser
+
+.TP
+.B
+\fB--vmstat\fP
+`vmstat` command parser
+
+.TP
+.B
+\fB--vmstat-s\fP
+`vmstat` command streaming parser
+
+.TP
+.B
+\fB--w\fP
+`w` command parser
+
+.TP
+.B
+\fB--wc\fP
+`wc` command parser
+
+.TP
+.B
+\fB--who\fP
+`who` command parser
+
+.TP
+.B
+\fB--xml\fP
+XML file parser
+
+.TP
+.B
+\fB--yaml\fP
+YAML file parser
+
+
+.RE
+.PP
+.B
+Options:
+.RS
+
+.TP
+.B
+\fB-a\fP
+about jc (JSON output)
+.TP
+.B
+\fB-d\fP
+debug - show traceback (use \fB-dd\fP for verbose traceback)
+.TP
+.B
+\fB-h\fP
+help (\fB-h --parser_name\fP for parser documentation)
+.TP
+.B
+\fB-m\fP
+monochrome output
+.TP
+.B
+\fB-p\fP
+pretty print output
+.TP
+.B
+\fB-q\fP
+quiet - suppress warnings (use \fB-qq\fP to ignore streaming parser errors)
+.TP
+.B
+\fB-r\fP
+raw JSON output
+.TP
+.B
+\fB-u\fP
+unbuffer output (useful for slow streaming data with streaming parsers)
+.TP
+.B
+\fB-v\fP
+version information
+
+.SH EXIT CODES
+Any fatal errors within jc will generate an exit code of \fB100\fP, otherwise the exit code will be \fB0\fP. When using the "Magic" syntax (e.g. \fBjc ifconfig eth0\fP), jc will store the exit code of the program being parsed and add it to the  jc exit code. This way it is easier to determine if an error was from the parsed program or jc.
+
+Consider the following examples using `ifconfig`:
+
+.RS
+ifconfig exit code = \fB0\fP, jc exit code = \fB0\fP, combined exit code = \fB0\fP (no errors)
+
+ifconfig exit code = \fB1\fP, jc exit code = \fB0\fP, combined exit code = \fB1\fP (error in ifconfig)
+
+ifconfig exit code = \fB0\fP, jc exit code = \fB100\fP, combined exit code = \fB100\fP (error in jc)
+
+ifconfig exit code = \fB1\fP, jc exit code = \fB100\fP, combined exit code = \fB101\fP (error in both ifconfig and jc)
+.RE
+
+.SH ENVIRONMENT
+You can specify custom colors via the \fBJC_COLORS\fP environment variable. The \fBJC_COLORS\fP environment variable takes four comma separated string values in the following format:
+
+JC_COLORS=<keyname_color>,<keyword_color>,<number_color>,<string_color>
+
+Where colors are: \fBblack\fP, \fBred\fP, \fBgreen\fP, \fByellow\fP, \fBblue\fP, \fBmagenta\fP, \fBcyan\fP, \fBgray\fP, \fBbrightblack\fP, \fBbrightred\fP, \fBbrightgreen\fP, \fBbrightyellow\fP, \fBbrightblue\fP, \fBbrightmagenta\fP, \fBbrightcyan\fP, \fBwhite\fP, or \fBdefault\fP
+
+For example, to set to the default colors:
+
+.RS
+JC_COLORS=blue,brightblack,magenta,green
+
+or
+
+JC_COLORS=default,default,default,default
+.RE
+
+
+.SH STREAMING PARSERS
+Most parsers load all of the data from \fBSTDIN\fP, parse it, then output the entire JSON document serially. There are some streaming parsers (e.g. \fBls-s\fP and \fBping-s\fP) that immediately start processing and outputing the data line-by-line as JSON Lines (aka NDJSON) while it is being received from \fBSTDIN\fP. This can significantly reduce the amount of memory required to parse large amounts of command output (e.g. \fBls -lR /\fP) and can sometimes process the data more quickly. Streaming parsers have slightly different behavior than standard parsers as outlined below.
+
+.RS
+Note: Streaming parsers cannot be used with the "magic" syntax
+.RE
+
+\fBIgnoring Errors\fP
+
+You may want to ignore parsing errors when using streaming parsers since these may be used in long-lived processing pipelines and errors can break the pipe. To ignore parsing errors, use the \fB-qq\fP cli option. This will add a \fB_jc_meta\fP object to the JSON output with a \fBsuccess\fP attribute. If \fBsuccess\fP is \fBtrue\fP, then there were no issues parsing the line. If \fBsuccess\fP is \fBfalse\fP, then a parsing issue was found and \fBerror\fP and \fBline\fP fields will be added to include a short error description and the contents of the unparsable line, respectively:
+
+.RS
+Successfully parsed line with \fB-qq\fP option:
+.RS
+{
+
+  "command_data": "data",
+  
+  "_jc_meta": {
+  
+    "success": true
+    
+  }
+  
+}
+.RE
+
+Unsuccessfully parsed line with \fB-qq\fP option:
+.RS
+{
+
+  "_jc_meta": {
+  
+    "success": false,
+    
+    "error": "error message",
+    
+    "line": "original line data"
+    
+  }
+  
+}
+.RE
+
+.RE
+\fBUnbuffering Output\fP
+
+Most operating systems will buffer output that is being piped from process to process. The buffer is usually around 4KB. When viewing the output in the terminal the OS buffer is not engaged so output is immediately displayed on the screen. When piping multiple processes together, though, it may seem as if the output is hanging when the input data is very slow (e.g. \fBping\fP):
+
+.RS
+$ ping 1.1.1.1 | jc --ping-s | jq
+
+<slow output>
+.RE
+
+This is because the OS engages the 4KB buffer between \fBjc\fP and \fBjq\fP in this example. To display the data on the terminal in realtime, you can disable the buffer with the \fB-u\fP (unbuffer) cli option:
+
+.RS
+$ ping 1.1.1.1 | jc --ping-s -u | jq
+
+{"type":"reply","pattern":null,"timestamp":null,"bytes":"64","response_ip":"1.1.1.1","icmp_seq":"1","ttl":"128","time_ms":"24.6","duplicate":false}
+
+{"type":"reply","pattern":null,"timestamp":null,"bytes":"64","response_ip":"1.1.1.1","icmp_seq":"2","ttl":"128","time_ms":"26.8","duplicate":false}
+
+etc...
+
+Note: Unbuffered output can be slower for large data streams.
+.RE
+
+.SH CUSTOM PARSERS
+Custom local parser plugins may be placed in a \fBjc/jcparsers\fP folder in your local "App data directory":
+
+.RS
+- Linux/unix: \fB$HOME/.local/share/jc/jcparsers\fP
+
+- macOS: \fB$HOME/Library/Application Support/jc/jcparsers\fP
+
+- Windows: \fB$LOCALAPPDATA\\jc\\jc\\jcparsers\fP
+.RE
+
+Local parser plugins are standard python module files. Use the \fBjc/parsers/foo.py\fP parser as a template and simply place a \fB.py\fP file in the \fBjcparsers\fP subfolder.
+
+Local plugin filenames must be valid python module names, therefore must consist entirely of alphanumerics and start with a letter. Local plugins may override default plugins.
+
+Note: The application data directory follows the XDG Base Directory Specification
+
+.SH CAVEATS
+\fBLocale:\fP For best results set the \fBLANG\fP locale environment variable to \fBC\fP or \fBen_US.UTF-8\fP. For example, either by setting directly on the command-line:
+
+\fB$ LANG=C date | jc --date\fP
+
+or by exporting to the environment before running commands:
+
+\fB$ export LANG=C\fP
+
+\fBTimezones:\fP Some parsers have calculated epoch timestamp fields added to the output. Unless a timestamp field name has a \fB_utc\fP suffix it is considered naive. (i.e. based on the local timezone of the system the \fBjc\fP parser was run on).
+
+If a UTC timezone can be detected in the text of the command output, the timestamp will be timezone aware and have a \fB_utc\fP suffix on the key name. (e.g. \fBepoch_utc\fP) No other timezones are supported for aware timestamps.
+
+.SH EXAMPLES
+Standard Syntax:
+.RS
+$ dig www.google.com | jc \fB--dig\fP \fB-p\fP
+.RE
+
+Magic Syntax:
+.RS
+$ jc \fB-p\fP dig www.google.com
+.RE
+
+For parser documentation:
+.RS
+$ jc \fB-h\fP \fB--dig\fP
+.RE
+.SH AUTHOR
+Kelly Brazil (kellyjonbrazil@gmail.com)
+
+https://github.com/kellyjonbrazil/jc
+
+.SH COPYRIGHT
+Copyright (c) 2019-2021 Kelly Brazil
+
+License:  MIT License

mangen.py

@@ -15,11 +15,3 @@ output = template.render(today=date.today(),
 
 with open('man/jc.1', 'w') as f:
     f.write(output)
-
-with open('man/jc.1', 'rb') as f_in:
-    with gzip.open('man/jc.1.gz', 'wb') as f_out:
-        shutil.copyfileobj(f_in, f_out)
-
-shutil.copyfile('man/jc.1.gz', 'jc/man/jc.1.gz')
-
-os.remove('man/jc.1')

setup.py

@@ -5,7 +5,7 @@ with open('README.md', 'r') as f:
 
 setuptools.setup(
     name='jc',
-    version='1.15.6',
+    version='1.17.0',
     author='Kelly Brazil',
     author_email='kellyjonbrazil@gmail.com',
     description='Converts the output of popular command-line tools and file-types to JSON.',

templates/manpage_template

@@ -36,7 +36,7 @@ about jc (JSON output)
 .TP
 .B
 \fB-d\fP
-debug - show traceback (\fB-dd\fP for verbose traceback)
+debug - show traceback (use \fB-dd\fP for verbose traceback)
 .TP
 .B
 \fB-h\fP
@@ -52,13 +52,17 @@ pretty print output
 .TP
 .B
 \fB-q\fP
-quiet - suppress warnings
+quiet - suppress warnings (use \fB-qq\fP to ignore streaming parser errors)
 .TP
 .B
 \fB-r\fP
 raw JSON output
 .TP
 .B
+\fB-u\fP
+unbuffer output (useful for slow streaming data with streaming parsers)
+.TP
+.B
 \fB-v\fP
 version information
 
@@ -94,6 +98,76 @@ or
 JC_COLORS=default,default,default,default
 .RE
 
+
+.SH STREAMING PARSERS
+Most parsers load all of the data from \fBSTDIN\fP, parse it, then output the entire JSON document serially. There are some streaming parsers (e.g. \fBls-s\fP and \fBping-s\fP) that immediately start processing and outputing the data line-by-line as JSON Lines (aka NDJSON) while it is being received from \fBSTDIN\fP. This can significantly reduce the amount of memory required to parse large amounts of command output (e.g. \fBls -lR /\fP) and can sometimes process the data more quickly. Streaming parsers have slightly different behavior than standard parsers as outlined below.
+
+.RS
+Note: Streaming parsers cannot be used with the "magic" syntax
+.RE
+
+\fBIgnoring Errors\fP
+
+You may want to ignore parsing errors when using streaming parsers since these may be used in long-lived processing pipelines and errors can break the pipe. To ignore parsing errors, use the \fB-qq\fP cli option. This will add a \fB_jc_meta\fP object to the JSON output with a \fBsuccess\fP attribute. If \fBsuccess\fP is \fBtrue\fP, then there were no issues parsing the line. If \fBsuccess\fP is \fBfalse\fP, then a parsing issue was found and \fBerror\fP and \fBline\fP fields will be added to include a short error description and the contents of the unparsable line, respectively:
+
+.RS
+Successfully parsed line with \fB-qq\fP option:
+.RS
+{
+
+  "command_data": "data",
+  
+  "_jc_meta": {
+  
+    "success": true
+    
+  }
+  
+}
+.RE
+
+Unsuccessfully parsed line with \fB-qq\fP option:
+.RS
+{
+
+  "_jc_meta": {
+  
+    "success": false,
+    
+    "error": "error message",
+    
+    "line": "original line data"
+    
+  }
+  
+}
+.RE
+
+.RE
+\fBUnbuffering Output\fP
+
+Most operating systems will buffer output that is being piped from process to process. The buffer is usually around 4KB. When viewing the output in the terminal the OS buffer is not engaged so output is immediately displayed on the screen. When piping multiple processes together, though, it may seem as if the output is hanging when the input data is very slow (e.g. \fBping\fP):
+
+.RS
+$ ping 1.1.1.1 | jc --ping-s | jq
+
+<slow output>
+.RE
+
+This is because the OS engages the 4KB buffer between \fBjc\fP and \fBjq\fP in this example. To display the data on the terminal in realtime, you can disable the buffer with the \fB-u\fP (unbuffer) cli option:
+
+.RS
+$ ping 1.1.1.1 | jc --ping-s -u | jq
+
+{"type":"reply","pattern":null,"timestamp":null,"bytes":"64","response_ip":"1.1.1.1","icmp_seq":"1","ttl":"128","time_ms":"24.6","duplicate":false}
+
+{"type":"reply","pattern":null,"timestamp":null,"bytes":"64","response_ip":"1.1.1.1","icmp_seq":"2","ttl":"128","time_ms":"26.8","duplicate":false}
+
+etc...
+
+Note: Unbuffered output can be slower for large data streams.
+.RE
+
 .SH CUSTOM PARSERS
 Custom local parser plugins may be placed in a \fBjc/jcparsers\fP folder in your local "App data directory":
 

templates/readme_template

@@ -1,7 +1,7 @@
 ![Tests](https://github.com/kellyjonbrazil/jc/workflows/Tests/badge.svg?branch=master)
 ![Pypi](https://img.shields.io/pypi/v/jc.svg)
 
-> Try the new `jc` [web demo](https://jc-web-demo.herokuapp.com/)!
+> Try the `jc` [web demo](https://jc-web-demo.herokuapp.com/)
 
 > JC is [now available](https://galaxy.ansible.com/community/general) as an Ansible filter plugin in the `community.general` collection! See this [blog post](https://blog.kellybrazil.com/2020/08/30/parsing-command-output-in-ansible-with-jc/) for an example.
 
@@ -31,28 +31,13 @@ $ jc dig example.com | jq -r '.[].answer[].data'
 ```
 The `jc` parsers can also be used as python modules. In this case the output will be a python dictionary, or list of dictionaries, instead of JSON:
 ```python
+>>> import subprocess
 >>> import jc.parsers.dig
 >>> 
->>> data = '''; <<>> DiG 9.10.6 <<>> example.com
-... ;; global options: +cmd
-... ;; Got answer:
-... ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64612
-... ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
-... 
-... ;; OPT PSEUDOSECTION:
-... ; EDNS: version: 0, flags:; udp: 4096
-... ;; QUESTION SECTION:
-... ;example.com.           IN  A
-... 
-... ;; ANSWER SECTION:
-... example.com.        29658   IN  A   93.184.216.34
-... 
-... ;; Query time: 52 msec
-... ;; SERVER: 2600:1700:bab0:d40::1#53(2600:1700:bab0:d40::1)
-... ;; WHEN: Fri Apr 16 16:13:00 PDT 2021
-... ;; MSG SIZE  rcvd: 56'''
+>>> cmd_output = subprocess.check_output(['dig', 'example.com'], text=True)
+>>> data = jc.parsers.dig.parse(cmd_output)
 >>>
->>> jc.parsers.dig.parse(data)
+>>> data
 [{'id': 64612, 'opcode': 'QUERY', 'status': 'NOERROR', 'flags': ['qr', 'rd', 'ra'], 'query_num': 1, 'answer_num':
 1, 'authority_num': 0, 'additional_num': 1, 'opt_pseudosection': {'edns': {'version': 0, 'flags': [], 'udp':
 4096}}, 'question': {'name': 'example.com.', 'class': 'IN', 'type': 'A'}, 'answer': [{'name': 'example.com.',
@@ -75,6 +60,8 @@ See also:
 - [libxo on FreeBSD](http://juniper.github.io/libxo/libxo-manual.html)
 - [powershell](https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/convertto-json?view=powershell-7)
 - [blog: linux apps should have a json flag](https://thomashunter.name/posts/2012-06-06-linux-cli-apps-should-have-a-json-flag)
+- [Hacker News discussion](https://news.ycombinator.com/item?id=28266193)
+- [Reddit discussion](https://www.reddit.com/r/programming/comments/pa4cbb/bringing_the_unix_philosophy_to_the_21st_century/)
 
 Use Cases:
 - [Bash scripting](https://blog.kellybrazil.com/2021/04/12/practical-json-at-the-command-line/)
@@ -83,7 +70,7 @@ Use Cases:
 - [Nornir command output parsing](https://blog.kellybrazil.com/2020/12/09/parsing-command-output-in-nornir-with-jc/)
 
 ## Installation
-There are several ways to get `jc`. You can install via `pip`; other OS package repositories like `apt-get`, `dnf`, `zypper`, `pacman`, `nix-env`, `guix`, `brew`, or `portsnap`; via DEB, RPM, and MSI packaged binaries for linux and Windows; or by downloading the correct binary for your architecture and running it anywhere on your filesystem.
+There are several ways to get `jc`. You can install via `pip`, OS package repositories, via DEB/RPM/MSI packaged binaries for linux and Windows, or by downloading the correct binary for your architecture and running it anywhere on your filesystem.
 
 ### Pip (macOS, linux, unix, Windows)
 ```bash
@@ -100,7 +87,7 @@ pip3 install jc
 | Arch linux            | `pacman -S jc`                                                                |
 | NixOS linux           | `nix-env -iA nixpkgs.jc` or `nix-env -iA nixos.jc`                            |
 | Guix System linux     | `guix install jc`                                                             |
-| MacOS                 | `brew install jc`                                                             |
+| macOS                 | `brew install jc`                                                             |
 | FreeBSD               | `portsnap fetch update && cd /usr/ports/textproc/py-jc && make install clean` |
 | Ansible filter plugin | `ansible-galaxy collection install community.general`                         |
 
@@ -124,11 +111,12 @@ The JSON output can be compact (default) or pretty formatted with the `-p` optio
 ### Options
 - `-a` about `jc`. Prints information about `jc` and the parsers (in JSON, of course!)
 - `-d` debug mode. Prints trace messages if parsing issues are encountered (use `-dd` for verbose debugging)
-- `-h` `jc` help. Use `jc -h --parser_name` for parser documentation
+- `-h` help. Use `jc -h --parser_name` for parser documentation
 - `-m` monochrome JSON output
 - `-p` pretty format the JSON output
-- `-q` quiet mode. Suppresses parser warning messages
+- `-q` quiet mode. Suppresses parser warning messages (use `-qq` to ignore streaming parser errors)
 - `-r` raw output. Provides a more literal JSON output, typically with string values and no additional semantic processing
+- `-u` unbuffer output
 - `-v` version information
 
 ### Exit Codes
@@ -160,6 +148,64 @@ or
 JC_COLORS=default,default,default,default
 ```
 
+### Streaming Parsers
+Most parsers load all of the data from STDIN, parse it, then output the entire JSON document serially. There are some streaming parsers (e.g. `ls-s` and `ping-s`) that immediately start processing and outputing the data line-by-line as [JSON Lines](https://jsonlines.org/) (aka [NDJSON](http://ndjson.org/)) while it is being received from STDIN. This can significantly reduce the amount of memory required to parse large amounts of command output (e.g. `ls -lR /`) and can sometimes process the data more quickly. Streaming parsers have slightly different behavior than standard parsers as outlined below.
+
+> Note: Streaming parsers cannot be used with the "magic" syntax
+
+#### Ignoring Errors
+
+You may want to ignore parsing errors when using streaming parsers since these may be used in long-lived processing pipelines and errors can break the pipe. To ignore parsing errors, use the `-qq` cli option or the `ignore_exceptions=True` argument with the `parse()` function. This will add a `_jc_meta` object to the JSON output with a `success` attribute. If `success` is `true`, then there were no issues parsing the line. If `success` is `false`, then a parsing issue was found and `error` and `line` fields will be added to include a short error description and the contents of the unparsable line, respectively:
+
+Successfully parsed line with `-qq` option:
+```json
+{
+  "command_data": "data",
+  "_jc_meta": {
+    "success": true
+  }
+}
+```
+Unsuccessfully parsed line with `-qq` option:
+```json
+{
+  "_jc_meta": {
+    "success": false,
+    "error": "error message",
+    "line": "original line data"
+  }
+}
+```
+
+#### Unbuffering Output
+
+Most operating systems will buffer output that is being piped from process to process. The buffer is usually around 4KB. When viewing the output in the terminal the OS buffer is not engaged so output is immediately displayed on the screen. When piping multiple processes together, though, it may seem as if the output is hanging when the input data is very slow (e.g. `ping`):
+```
+$ ping 1.1.1.1 | jc --ping-s | jq
+<slow output>
+```
+This is because the OS engages the 4KB buffer between `jc` and `jq` in this example. To display the data on the terminal in realtime, you can disable the buffer with the `-u` (unbuffer) cli option:
+```
+$ ping 1.1.1.1 | jc --ping-s -u | jq
+{"type":"reply","pattern":null,"timestamp":null,"bytes":"64","response_ip":"1.1.1.1","icmp_seq":"1","ttl":"128","time_ms":"24.6","duplicate":false}
+{"type":"reply","pattern":null,"timestamp":null,"bytes":"64","response_ip":"1.1.1.1","icmp_seq":"2","ttl":"128","time_ms":"26.8","duplicate":false}
+...
+```
+> Note: Unbuffered output can be slower for large data streams.
+
+#### Using Streaming Parsers as Python Modules
+
+Streaming parsers accept any iterable object and return a generator iterator object allowing lazy processing of the data. The input data should iterate on lines of string data. Examples of good input data are `sys.stdin` or `str.splitlines()`.
+
+To use the generator object in your code, simply loop through it or use the [next()](https://docs.python.org/3/library/functions.html#next) builtin function:
+```python
+import jc.parsers.ls_s
+
+result = jc.parsers.ls_s.parse(ls_command_output.splitlines())
+for item in result:
+    print(item["filename"])
+```
+
 ### Custom Parsers
 Custom local parser plugins may be placed in a `jc/jcparsers` folder in your local **"App data directory"**:
 
@@ -174,7 +220,8 @@ Local plugin filenames must be valid python module names, therefore must consist
 > Note: The application data directory follows the [XDG Base Directory Specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html)
 
 ### Caveats
-**Locale:**
+
+#### Locale
 
 For best results set the `LANG` locale environment variable to `C` or `en_US.UTF-8`. For example, either by setting directly on the command-line:
 ```
@@ -185,7 +232,7 @@ or by exporting to the environment before running commands:
 $ export LANG=C
 ```
 
-**Timezones:**
+#### Timezones
 
 Some parsers have calculated epoch timestamp fields added to the output. Unless a timestamp field name has a `_utc` suffix it is considered naive. (i.e. based on the local timezone of the system the `jc` parser was run on).
 
@@ -219,7 +266,7 @@ Tested on:
 - Windows 2019 Server
 
 ## Contributions
-Feel free to add/improve code or parsers! You can use the [`jc/parsers/foo.py`](https://github.com/kellyjonbrazil/jc/blob/master/jc/parsers/foo.py) parser as a template and submit your parser with a pull request.
+Feel free to add/improve code or parsers! You can use the [`jc/parsers/foo.py`](https://github.com/kellyjonbrazil/jc/blob/master/jc/parsers/foo.py) or [`jc/parsers/foo_s.py (streaming)`](https://github.com/kellyjonbrazil/jc/blob/master/jc/parsers/foo_s.py) parsers as a template and submit your parser with a pull request.
 
 Please see the [Contributing Guidelines](https://github.com/kellyjonbrazil/jc/blob/master/CONTRIBUTING.md) for more information.
 

tests/fixtures/centos-7.7/ls-al-streaming.json

@@ -0,0 +1 @@
+[{"filename":".","flags":"dr-xr-xr-x.","links":17,"owner":"root","group":"root","size":224,"date":"Aug 15 10:56"},{"filename":"..","flags":"dr-xr-xr-x.","links":17,"owner":"root","group":"root","size":224,"date":"Aug 15 10:56"},{"filename":"bin","link_to":"usr/bin","flags":"lrwxrwxrwx.","links":1,"owner":"root","group":"root","size":7,"date":"Aug 15 10:53"},{"filename":"boot","flags":"dr-xr-xr-x.","links":5,"owner":"root","group":"root","size":4096,"date":"Oct 21 13:18"},{"filename":"dev","flags":"drwxr-xr-x.","links":20,"owner":"root","group":"root","size":3180,"date":"Oct 25 18:21"},{"filename":"etc","flags":"drwxr-xr-x.","links":78,"owner":"root","group":"root","size":8192,"date":"Oct 25 18:47"},{"filename":"home","flags":"drwxr-xr-x.","links":3,"owner":"root","group":"root","size":21,"date":"Aug 15 10:56"},{"filename":"lib","link_to":"usr/lib","flags":"lrwxrwxrwx.","links":1,"owner":"root","group":"root","size":7,"date":"Aug 15 10:53"},{"filename":"lib64","link_to":"usr/lib64","flags":"lrwxrwxrwx.","links":1,"owner":"root","group":"root","size":9,"date":"Aug 15 10:53"},{"filename":"media","flags":"drwxr-xr-x.","links":2,"owner":"root","group":"root","size":6,"date":"Apr 10 2018"},{"filename":"mnt","flags":"drwxr-xr-x.","links":2,"owner":"root","group":"root","size":6,"date":"Apr 10 2018"},{"filename":"opt","flags":"drwxr-xr-x.","links":2,"owner":"root","group":"root","size":6,"date":"Apr 10 2018"},{"filename":"proc","flags":"dr-xr-xr-x.","links":121,"owner":"root","group":"root","size":0,"date":"Oct 25 18:21"},{"filename":"root","flags":"dr-xr-x---.","links":3,"owner":"root","group":"root","size":170,"date":"Oct 15 11:11"},{"filename":"run","flags":"drwxr-xr-x.","links":26,"owner":"root","group":"root","size":800,"date":"Oct 25 18:47"},{"filename":"sbin","link_to":"usr/sbin","flags":"lrwxrwxrwx.","links":1,"owner":"root","group":"root","size":8,"date":"Aug 15 10:53"},{"filename":"srv","flags":"drwxr-xr-x.","links":2,"owner":"root","group":"root","size":6,"date":"Apr 10 2018"},{"filename":"sys","flags":"dr-xr-xr-x.","links":13,"owner":"root","group":"root","size":0,"date":"Oct 25 18:21"},{"filename":"tmp","flags":"drwxrwxrwt.","links":19,"owner":"root","group":"root","size":4096,"date":"Oct 26 10:14"},{"filename":"usr","flags":"drwxr-xr-x.","links":13,"owner":"root","group":"root","size":155,"date":"Aug 15 10:53"},{"filename":"var","flags":"drwxr-xr-x.","links":19,"owner":"root","group":"root","size":267,"date":"Aug 15 10:57"}]

tests/fixtures/centos-7.7/ls-alh-streaming.json

@@ -0,0 +1 @@
+[{"filename":".","flags":"dr-xr-xr-x.","links":17,"owner":"root","group":"root","size":224,"date":"Aug 15 10:56"},{"filename":"..","flags":"dr-xr-xr-x.","links":17,"owner":"root","group":"root","size":224,"date":"Aug 15 10:56"},{"filename":"bin","link_to":"usr/bin","flags":"lrwxrwxrwx.","links":1,"owner":"root","group":"root","size":7,"date":"Aug 15 10:53"},{"filename":"boot","flags":"dr-xr-xr-x.","links":5,"owner":"root","group":"root","size":4,"date":"Oct 21 13:18"},{"filename":"dev","flags":"drwxr-xr-x.","links":20,"owner":"root","group":"root","size":3,"date":"Oct 25 18:21"},{"filename":"etc","flags":"drwxr-xr-x.","links":78,"owner":"root","group":"root","size":8,"date":"Oct 25 18:47"},{"filename":"home","flags":"drwxr-xr-x.","links":3,"owner":"root","group":"root","size":21,"date":"Aug 15 10:56"},{"filename":"lib","link_to":"usr/lib","flags":"lrwxrwxrwx.","links":1,"owner":"root","group":"root","size":7,"date":"Aug 15 10:53"},{"filename":"lib64","link_to":"usr/lib64","flags":"lrwxrwxrwx.","links":1,"owner":"root","group":"root","size":9,"date":"Aug 15 10:53"},{"filename":"media","flags":"drwxr-xr-x.","links":2,"owner":"root","group":"root","size":6,"date":"Apr 10 2018"},{"filename":"mnt","flags":"drwxr-xr-x.","links":2,"owner":"root","group":"root","size":6,"date":"Apr 10 2018"},{"filename":"opt","flags":"drwxr-xr-x.","links":2,"owner":"root","group":"root","size":6,"date":"Apr 10 2018"},{"filename":"proc","flags":"dr-xr-xr-x.","links":121,"owner":"root","group":"root","size":0,"date":"Oct 25 18:21"},{"filename":"root","flags":"dr-xr-x---.","links":3,"owner":"root","group":"root","size":170,"date":"Oct 15 11:11"},{"filename":"run","flags":"drwxr-xr-x.","links":26,"owner":"root","group":"root","size":800,"date":"Oct 25 18:47"},{"filename":"sbin","link_to":"usr/sbin","flags":"lrwxrwxrwx.","links":1,"owner":"root","group":"root","size":8,"date":"Aug 15 10:53"},{"filename":"srv","flags":"drwxr-xr-x.","links":2,"owner":"root","group":"root","size":6,"date":"Apr 10 2018"},{"filename":"sys","flags":"dr-xr-xr-x.","links":13,"owner":"root","group":"root","size":0,"date":"Oct 25 18:21"},{"filename":"tmp","flags":"drwxrwxrwt.","links":19,"owner":"root","group":"root","size":4,"date":"Oct 26 10:14"},{"filename":"usr","flags":"drwxr-xr-x.","links":13,"owner":"root","group":"root","size":155,"date":"Aug 15 10:53"},{"filename":"var","flags":"drwxr-xr-x.","links":19,"owner":"root","group":"root","size":267,"date":"Aug 15 10:57"}]

tests/fixtures/centos-7.7/ping-hostname-O-D-p-s-streaming.json

@@ -0,0 +1 @@
+[{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978465.914536,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":1,"ttl":59,"time_ms":31.4,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978465.993009,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":2,"ttl":59,"time_ms":30.3,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978467.010196,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":3,"ttl":59,"time_ms":32.0,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978468.033743,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":4,"ttl":59,"time_ms":38.8,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978469.051227,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":5,"ttl":59,"time_ms":38.0,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978470.048764,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":6,"ttl":59,"time_ms":29.9,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978471.051945,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":7,"ttl":59,"time_ms":28.9,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978472.064206,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":8,"ttl":59,"time_ms":37.4,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978473.062587,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":9,"ttl":59,"time_ms":31.5,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978474.074343,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":10,"ttl":59,"time_ms":38.3,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978475.079703,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":11,"ttl":59,"time_ms":38.8,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978476.076383,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":12,"ttl":59,"time_ms":30.7,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978477.084119,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":13,"ttl":59,"time_ms":30.7,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978478.092207,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":14,"ttl":59,"time_ms":31.6,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978479.104358,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":15,"ttl":59,"time_ms":37.7,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978480.106907,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":16,"ttl":59,"time_ms":37.5,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978481.11558,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":17,"ttl":59,"time_ms":37.3,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978482.119872,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":18,"ttl":59,"time_ms":33.8,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978483.131901,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":19,"ttl":59,"time_ms":37.0,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978484.141117,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":20,"ttl":59,"time_ms":36.9,"duplicate":false},{"type":"summary","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","packets_transmitted":20,"packets_received":20,"packet_loss_percent":0.0,"duplicates":0,"time_ms":19146.0,"round_trip_ms_min":28.96,"round_trip_ms_avg":34.468,"round_trip_ms_max":38.892,"round_trip_ms_stddev":3.497}]

tests/fixtures/centos-7.7/ping-hostname-O-p-streaming.json

@@ -0,0 +1 @@
+[{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":1,"ttl":59,"time_ms":24.4,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":2,"ttl":59,"time_ms":23.3,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":3,"ttl":59,"time_ms":32.6,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":4,"ttl":59,"time_ms":32.6,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":5,"ttl":59,"time_ms":26.9,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":6,"ttl":59,"time_ms":24.1,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":7,"ttl":59,"time_ms":24.6,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":8,"ttl":59,"time_ms":33.9,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":9,"ttl":59,"time_ms":32.7,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":10,"ttl":59,"time_ms":31.2,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":11,"ttl":59,"time_ms":25.7,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":12,"ttl":59,"time_ms":33.8,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":13,"ttl":59,"time_ms":23.7,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":14,"ttl":59,"time_ms":23.9,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":15,"ttl":59,"time_ms":33.6,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":16,"ttl":59,"time_ms":24.5,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":17,"ttl":59,"time_ms":30.1,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":18,"ttl":59,"time_ms":24.1,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":19,"ttl":59,"time_ms":32.2,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":20,"ttl":59,"time_ms":31.0,"duplicate":false},{"type":"summary","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","packets_transmitted":20,"packets_received":20,"packet_loss_percent":0.0,"duplicates":0,"time_ms":19233.0,"round_trip_ms_min":23.359,"round_trip_ms_avg":28.495,"round_trip_ms_max":33.979,"round_trip_ms_stddev":4.081}]

tests/fixtures/centos-7.7/ping-hostname-O-streaming.json

@@ -0,0 +1 @@
+[{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":1,"ttl":59,"time_ms":29.6,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":2,"ttl":59,"time_ms":30.1,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":3,"ttl":59,"time_ms":35.5,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":4,"ttl":59,"time_ms":35.5,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":5,"ttl":59,"time_ms":34.9,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":6,"ttl":59,"time_ms":29.9,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":7,"ttl":59,"time_ms":27.6,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":8,"ttl":59,"time_ms":28.6,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":9,"ttl":59,"time_ms":35.2,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":10,"ttl":59,"time_ms":34.4,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":11,"ttl":59,"time_ms":35.9,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":12,"ttl":59,"time_ms":35.8,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":13,"ttl":59,"time_ms":34.4,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":14,"ttl":59,"time_ms":35.5,"duplicate":false},{"type":"timeout","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"icmp_seq":15},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":16,"ttl":59,"time_ms":36.6,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":17,"ttl":59,"time_ms":34.6,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":18,"ttl":59,"time_ms":34.6,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":19,"ttl":59,"time_ms":36.7,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":20,"ttl":59,"time_ms":34.3,"duplicate":false},{"type":"summary","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"packets_transmitted":20,"packets_received":19,"packet_loss_percent":5.0,"duplicates":0,"time_ms":19125.0,"round_trip_ms_min":27.656,"round_trip_ms_avg":33.717,"round_trip_ms_max":36.758,"round_trip_ms_stddev":2.814}]

tests/fixtures/centos-7.7/ping-ip-O-D-streaming.json

@@ -0,0 +1 @@
+[{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037214.261953,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":1,"ttl":64,"time_ms":0.041,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037215.264798,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":2,"ttl":64,"time_ms":0.048,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037216.272296,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":3,"ttl":64,"time_ms":0.047,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037217.275851,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":4,"ttl":64,"time_ms":0.062,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037218.284242,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":5,"ttl":64,"time_ms":0.045,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037219.283712,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":6,"ttl":64,"time_ms":0.043,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037220.290949,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":7,"ttl":64,"time_ms":0.046,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037221.295962,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":8,"ttl":64,"time_ms":0.044,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037222.30702,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":9,"ttl":64,"time_ms":0.048,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037223.313919,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":10,"ttl":64,"time_ms":0.081,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037224.313679,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":11,"ttl":64,"time_ms":0.043,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037225.320748,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":12,"ttl":64,"time_ms":0.044,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037226.324322,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":13,"ttl":64,"time_ms":0.045,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037227.325835,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":14,"ttl":64,"time_ms":0.046,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037228.327028,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":15,"ttl":64,"time_ms":0.046,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037229.329891,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":16,"ttl":64,"time_ms":0.052,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037230.333891,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":17,"ttl":64,"time_ms":0.044,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037231.338137,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":18,"ttl":64,"time_ms":0.046,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037232.340475,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":19,"ttl":64,"time_ms":0.048,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037233.343058,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":20,"ttl":64,"time_ms":0.045,"duplicate":false},{"type":"summary","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"packets_transmitted":20,"packets_received":20,"packet_loss_percent":0.0,"duplicates":0,"time_ms":19081.0,"round_trip_ms_min":0.041,"round_trip_ms_avg":0.048,"round_trip_ms_max":0.081,"round_trip_ms_stddev":0.009}]

tests/fixtures/centos-7.7/ping-ip-O-streaming-ignore-exceptions.json

@@ -0,0 +1 @@
+[{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":1,"ttl":64,"time_ms":0.038,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":2,"ttl":64,"time_ms":0.043,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":3,"ttl":64,"time_ms":0.044,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":4,"ttl":64,"time_ms":0.052,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":5,"ttl":64,"time_ms":0.08,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":6,"ttl":64,"time_ms":0.043,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":7,"ttl":64,"time_ms":0.047,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":8,"ttl":64,"time_ms":0.04,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":9,"ttl":64,"time_ms":0.052,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":10,"ttl":64,"time_ms":0.044,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":11,"ttl":64,"time_ms":0.043,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":12,"ttl":64,"time_ms":0.043,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":13,"ttl":64,"time_ms":0.05,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":14,"ttl":64,"time_ms":0.045,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":15,"ttl":64,"time_ms":0.062,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":16,"ttl":64,"time_ms":0.046,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":17,"ttl":64,"time_ms":0.046,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":18,"ttl":64,"time_ms":0.045,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":19,"ttl":64,"time_ms":0.044,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":20,"ttl":64,"time_ms":0.044,"duplicate":false,"_jc_meta":{"success":true}},{"type":"summary","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"packets_transmitted":20,"packets_received":20,"packet_loss_percent":0.0,"duplicates":0,"time_ms":19070.0,"round_trip_ms_min":0.038,"round_trip_ms_avg":0.047,"round_trip_ms_max":0.08,"round_trip_ms_stddev":0.011,"_jc_meta":{"success":true}}]

tests/fixtures/centos-7.7/ping-ip-O-streaming.json

@@ -0,0 +1 @@
+[{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":1,"ttl":64,"time_ms":0.038,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":2,"ttl":64,"time_ms":0.043,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":3,"ttl":64,"time_ms":0.044,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":4,"ttl":64,"time_ms":0.052,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":5,"ttl":64,"time_ms":0.08,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":6,"ttl":64,"time_ms":0.043,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":7,"ttl":64,"time_ms":0.047,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":8,"ttl":64,"time_ms":0.04,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":9,"ttl":64,"time_ms":0.052,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":10,"ttl":64,"time_ms":0.044,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":11,"ttl":64,"time_ms":0.043,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":12,"ttl":64,"time_ms":0.043,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":13,"ttl":64,"time_ms":0.05,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":14,"ttl":64,"time_ms":0.045,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":15,"ttl":64,"time_ms":0.062,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":16,"ttl":64,"time_ms":0.046,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":17,"ttl":64,"time_ms":0.046,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":18,"ttl":64,"time_ms":0.045,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":19,"ttl":64,"time_ms":0.044,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":20,"ttl":64,"time_ms":0.044,"duplicate":false},{"type":"summary","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"packets_transmitted":20,"packets_received":20,"packet_loss_percent":0.0,"duplicates":0,"time_ms":19070.0,"round_trip_ms_min":0.038,"round_trip_ms_avg":0.047,"round_trip_ms_max":0.08,"round_trip_ms_stddev":0.011}]

tests/fixtures/centos-7.7/ping-ip-dup-streaming.json

@@ -0,0 +1 @@
+[{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.221","icmp_seq":1,"ttl":64,"time_ms":0.586,"duplicate":false},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.88","icmp_seq":1,"ttl":64,"time_ms":382.0,"duplicate":true},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.78","icmp_seq":1,"ttl":128,"time_ms":382.0,"duplicate":true},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.217","icmp_seq":1,"ttl":255,"time_ms":387.0,"duplicate":true},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.186","icmp_seq":1,"ttl":64,"time_ms":389.0,"duplicate":true},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.89","icmp_seq":1,"ttl":64,"time_ms":389.0,"duplicate":true},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.75","icmp_seq":1,"ttl":64,"time_ms":584.0,"duplicate":true},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.221","icmp_seq":2,"ttl":64,"time_ms":0.861,"duplicate":false},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.78","icmp_seq":2,"ttl":128,"time_ms":4.17,"duplicate":true},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.88","icmp_seq":2,"ttl":64,"time_ms":4.19,"duplicate":true},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.89","icmp_seq":2,"ttl":64,"time_ms":12.7,"duplicate":true},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.81","icmp_seq":1,"ttl":64,"time_ms":1029.0,"duplicate":true},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.72","icmp_seq":1,"ttl":64,"time_ms":1276.0,"duplicate":true},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.251","icmp_seq":1,"ttl":64,"time_ms":1276.0,"duplicate":true},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.251","icmp_seq":2,"ttl":64,"time_ms":262.0,"duplicate":true},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.72","icmp_seq":2,"ttl":64,"time_ms":263.0,"duplicate":true},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.246","icmp_seq":2,"ttl":255,"time_ms":263.0,"duplicate":true},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.217","icmp_seq":2,"ttl":255,"time_ms":919.0,"duplicate":true},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.186","icmp_seq":2,"ttl":64,"time_ms":919.0,"duplicate":true},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.75","icmp_seq":2,"ttl":64,"time_ms":919.0,"duplicate":true},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.81","icmp_seq":2,"ttl":64,"time_ms":919.0,"duplicate":true},{"type":"summary","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"packets_transmitted":2,"packets_received":2,"packet_loss_percent":0.0,"duplicates":19,"time_ms":1013.0,"round_trip_ms_min":0.586,"round_trip_ms_avg":504.26,"round_trip_ms_max":1276.448,"round_trip_ms_stddev":417.208}]

tests/fixtures/centos-7.7/ping6-hostname-O-D-p-s-streaming.json

@@ -0,0 +1 @@
+[{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978345.609669,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":1,"ttl":59,"time_ms":32.4,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978346.58542,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":2,"ttl":59,"time_ms":39.9,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978347.594128,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":3,"ttl":59,"time_ms":42.3,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978348.595221,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":4,"ttl":59,"time_ms":40.2,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978349.600372,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":5,"ttl":59,"time_ms":43.2,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978350.590676,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":6,"ttl":59,"time_ms":31.8,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978351.601527,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":7,"ttl":59,"time_ms":41.8,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978352.604195,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":8,"ttl":59,"time_ms":41.7,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978353.607212,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":9,"ttl":59,"time_ms":42.0,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978354.610771,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":10,"ttl":59,"time_ms":40.7,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978355.613729,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":11,"ttl":59,"time_ms":40.4,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978356.611887,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":12,"ttl":59,"time_ms":32.6,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978357.62481,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":13,"ttl":59,"time_ms":40.1,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978358.629185,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":14,"ttl":59,"time_ms":42.0,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978359.634854,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":15,"ttl":59,"time_ms":41.2,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978360.638344,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":16,"ttl":59,"time_ms":40.6,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978361.640968,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":17,"ttl":59,"time_ms":40.7,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978362.645739,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":18,"ttl":59,"time_ms":39.9,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978363.6467,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":19,"ttl":59,"time_ms":37.5,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978364.650853,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":20,"ttl":59,"time_ms":33.6,"duplicate":false},{"type":"summary","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","packets_transmitted":20,"packets_received":20,"packet_loss_percent":0.0,"duplicates":0,"time_ms":19077.0,"round_trip_ms_min":31.845,"round_trip_ms_avg":39.274,"round_trip_ms_max":43.243,"round_trip_ms_stddev":3.522}]

tests/fixtures/centos-7.7/ping6-hostname-O-p-streaming.json

@@ -0,0 +1 @@
+[{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":1,"ttl":59,"time_ms":30.9,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":2,"ttl":59,"time_ms":39.0,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":3,"ttl":59,"time_ms":32.6,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":4,"ttl":59,"time_ms":38.4,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":5,"ttl":59,"time_ms":38.8,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":6,"ttl":59,"time_ms":42.6,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":7,"ttl":59,"time_ms":30.7,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":8,"ttl":59,"time_ms":39.4,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":9,"ttl":59,"time_ms":39.3,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":10,"ttl":59,"time_ms":38.9,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":11,"ttl":59,"time_ms":38.6,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":12,"ttl":59,"time_ms":38.2,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":13,"ttl":59,"time_ms":39.6,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":14,"ttl":59,"time_ms":37.4,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":15,"ttl":59,"time_ms":33.7,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":16,"ttl":59,"time_ms":39.4,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":17,"ttl":59,"time_ms":38.9,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":18,"ttl":59,"time_ms":41.3,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":19,"ttl":59,"time_ms":32.2,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":20,"ttl":59,"time_ms":38.4,"duplicate":false},{"type":"summary","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","packets_transmitted":20,"packets_received":20,"packet_loss_percent":0.0,"duplicates":0,"time_ms":19164.0,"round_trip_ms_min":30.757,"round_trip_ms_avg":37.455,"round_trip_ms_max":42.652,"round_trip_ms_stddev":3.338}]

tests/fixtures/centos-7.7/ping6-ip-O-D-p-streaming.json

@@ -0,0 +1 @@
+[{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976827.240914,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":1,"ttl":59,"time_ms":28.7,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976828.25493,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":2,"ttl":59,"time_ms":37.2,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976829.252877,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":3,"ttl":59,"time_ms":29.7,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976830.262654,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":4,"ttl":59,"time_ms":37.7,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976831.265626,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":5,"ttl":59,"time_ms":34.8,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976832.269834,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":6,"ttl":59,"time_ms":35.6,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976833.268059,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":7,"ttl":59,"time_ms":28.4,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976834.274292,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":8,"ttl":59,"time_ms":28.1,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976835.287123,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":9,"ttl":59,"time_ms":34.9,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976836.287707,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":10,"ttl":59,"time_ms":34.4,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976837.290589,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":11,"ttl":59,"time_ms":35.2,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976838.293514,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":12,"ttl":59,"time_ms":35.4,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976839.290914,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":13,"ttl":59,"time_ms":29.8,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976840.292897,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":14,"ttl":59,"time_ms":28.5,"duplicate":false},{"type":"timeout","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976842.269238,"icmp_seq":15},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976842.30145,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":16,"ttl":59,"time_ms":31.8,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976843.312998,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":17,"ttl":59,"time_ms":39.8,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976844.314228,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":18,"ttl":59,"time_ms":35.7,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976845.315518,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":19,"ttl":59,"time_ms":35.1,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976846.321706,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":20,"ttl":59,"time_ms":35.4,"duplicate":false},{"type":"summary","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","packets_transmitted":20,"packets_received":19,"packet_loss_percent":5.0,"duplicates":0,"time_ms":19074.0,"round_trip_ms_min":28.15,"round_trip_ms_avg":33.534,"round_trip_ms_max":39.843,"round_trip_ms_stddev":3.489}]

tests/fixtures/centos-7.7/ping6-ip-O-p-streaming.json

@@ -0,0 +1 @@
+[{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":1,"ttl":59,"time_ms":27.9,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":2,"ttl":59,"time_ms":28.4,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":3,"ttl":59,"time_ms":36.0,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":4,"ttl":59,"time_ms":28.5,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":5,"ttl":59,"time_ms":35.8,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":6,"ttl":59,"time_ms":34.4,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":7,"ttl":59,"time_ms":30.7,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":8,"ttl":59,"time_ms":28.5,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":9,"ttl":59,"time_ms":36.5,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":10,"ttl":59,"time_ms":36.3,"duplicate":false},{"type":"timeout","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"icmp_seq":11},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":12,"ttl":59,"time_ms":37.4,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":13,"ttl":59,"time_ms":30.7,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":14,"ttl":59,"time_ms":36.5,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":15,"ttl":59,"time_ms":35.4,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":16,"ttl":59,"time_ms":36.3,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":17,"ttl":59,"time_ms":37.5,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":18,"ttl":59,"time_ms":36.2,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":19,"ttl":59,"time_ms":27.0,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":20,"ttl":59,"time_ms":38.1,"duplicate":false},{"type":"summary","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","packets_transmitted":20,"packets_received":19,"packet_loss_percent":5.0,"duplicates":0,"time_ms":19067.0,"round_trip_ms_min":27.064,"round_trip_ms_avg":33.626,"round_trip_ms_max":38.146,"round_trip_ms_stddev":3.803}]

tests/fixtures/centos-7.7/ping6-ip-dup-streaming.json

@@ -0,0 +1 @@
+[{"type":"reply","destination_ip":"ff02::1%ens33","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"fe80::c48:5896:526d:81ba%ens33","icmp_seq":1,"ttl":64,"time_ms":0.245,"duplicate":false},{"type":"reply","destination_ip":"ff02::1%ens33","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"fe80::feae:34ff:fea1:3a80%ens33","icmp_seq":1,"ttl":64,"time_ms":3.65,"duplicate":true},{"type":"reply","destination_ip":"ff02::1%ens33","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"fe80::c48:5896:526d:81ba%ens33","icmp_seq":2,"ttl":64,"time_ms":0.329,"duplicate":false},{"type":"reply","destination_ip":"ff02::1%ens33","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"fe80::feae:34ff:fea1:3a80%ens33","icmp_seq":2,"ttl":64,"time_ms":11.7,"duplicate":true},{"type":"reply","destination_ip":"ff02::1%ens33","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"fe80::c48:5896:526d:81ba%ens33","icmp_seq":3,"ttl":64,"time_ms":0.592,"duplicate":false},{"type":"reply","destination_ip":"ff02::1%ens33","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"fe80::feae:34ff:fea1:3a80%ens33","icmp_seq":3,"ttl":64,"time_ms":12.3,"duplicate":true},{"type":"reply","destination_ip":"ff02::1%ens33","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"fe80::c48:5896:526d:81ba%ens33","icmp_seq":4,"ttl":64,"time_ms":0.51,"duplicate":false},{"type":"reply","destination_ip":"ff02::1%ens33","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"fe80::feae:34ff:fea1:3a80%ens33","icmp_seq":4,"ttl":64,"time_ms":12.5,"duplicate":true},{"type":"reply","destination_ip":"ff02::1%ens33","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"fe80::c48:5896:526d:81ba%ens33","icmp_seq":5,"ttl":64,"time_ms":0.538,"duplicate":false},{"type":"summary","destination_ip":"ff02::1%ens33","sent_bytes":56,"pattern":null,"packets_transmitted":5,"packets_received":5,"packet_loss_percent":0.0,"duplicates":4,"time_ms":4017.0,"round_trip_ms_min":0.245,"round_trip_ms_avg":4.726,"round_trip_ms_max":12.568,"round_trip_ms_stddev":5.395}]

tests/fixtures/centos-7.7/sfdisk-d-multi.json

@@ -0,0 +1 @@
+[{"disk":"/dev/sda","units":"sectors","partitions":[{"device":"/dev/sda1","start":2048,"size":2097152,"id":"83","boot":true},{"device":"/dev/sda2","start":2099200,"size":39843840,"id":"8e","boot":false},{"device":"/dev/sda3","start":0,"size":0,"id":"0","boot":false},{"device":"/dev/sda4","start":0,"size":0,"id":"0","boot":false}]},{"disk":"/dev/sdb","units":"sectors","partitions":[{"device":"/dev/sdb1","start":2048,"size":2097152,"id":"83","boot":true},{"device":"/dev/sdb2","start":2099200,"size":39843840,"id":"8e","boot":false},{"device":"/dev/sdb3","start":0,"size":0,"id":"0","boot":false},{"device":"/dev/sdb4","start":0,"size":0,"id":"0","boot":false}]}]

tests/fixtures/centos-7.7/sfdisk-d-multi.out

@@ -0,0 +1,16 @@
+# partition table of /dev/sda
+unit: sectors
+
+/dev/sda1 : start=     2048, size=  2097152, Id=83, bootable
+/dev/sda2 : start=  2099200, size= 39843840, Id=8e
+/dev/sda3 : start=        0, size=        0, Id= 0
+/dev/sda4 : start=        0, size=        0, Id= 0
+
+# partition table of /dev/sdb
+unit: sectors
+
+/dev/sdb1 : start=     2048, size=  2097152, Id=83, bootable
+/dev/sdb2 : start=  2099200, size= 39843840, Id=8e
+/dev/sdb3 : start=        0, size=        0, Id= 0
+/dev/sdb4 : start=        0, size=        0, Id= 0
+

tests/fixtures/centos-7.7/sfdisk-d.json

@@ -0,0 +1 @@
+[{"disk":"/dev/sda","units":"sectors","partitions":[{"device":"/dev/sda1","start":2048,"size":2097152,"id":"83","boot":true},{"device":"/dev/sda2","start":2099200,"size":39843840,"id":"8e","boot":false},{"device":"/dev/sda3","start":0,"size":0,"id":"0","boot":false},{"device":"/dev/sda4","start":0,"size":0,"id":"0","boot":false}]}]

tests/fixtures/centos-7.7/sfdisk-d.out

@@ -0,0 +1,7 @@
+# partition table of /dev/sda
+unit: sectors
+
+/dev/sda1 : start=     2048, size=  2097152, Id=83, bootable
+/dev/sda2 : start=  2099200, size= 39843840, Id=8e
+/dev/sda3 : start=        0, size=        0, Id= 0
+/dev/sda4 : start=        0, size=        0, Id= 0

tests/fixtures/centos-7.7/sfdisk-l-multi.json

@@ -0,0 +1 @@
+[{"disk":"/dev/sda","cylinders":2610,"heads":255,"sectors_per_track":63,"units":"cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0","partitions":[{"device":"/dev/sda1","boot":true,"start":0,"end":130,"cyls":131,"blocks":1048576,"id":"83","system":"Linux"},{"device":"/dev/sda2","boot":false,"start":130,"end":2610,"cyls":2481,"blocks":19921920,"id":"8e","system":"Linux LVM"},{"device":"/dev/sda3","boot":false,"start":0,"end":null,"cyls":0,"blocks":0,"id":"0","system":"Empty"},{"device":"/dev/sda4","boot":false,"start":0,"end":null,"cyls":0,"blocks":0,"id":"0","system":"Empty"}]},{"disk":"/dev/mapper/centos-root","cylinders":2218,"heads":255,"sectors_per_track":63},{"disk":"/dev/mapper/centos-swap","cylinders":261,"heads":255,"sectors_per_track":63,"units":"cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0","partitions":[{"device":"/dev/sda1","boot":true,"start":0,"end":130,"cyls":131,"blocks":1048576,"id":"83","system":"Linux"},{"device":"/dev/sda2","boot":false,"start":130,"end":2610,"cyls":2481,"blocks":19921920,"id":"8e","system":"Linux LVM"},{"device":"/dev/sda3","boot":false,"start":0,"end":null,"cyls":0,"blocks":0,"id":"0","system":"Empty"},{"device":"/dev/sda4","boot":false,"start":0,"end":null,"cyls":0,"blocks":0,"id":"0","system":"Empty"}]}]

tests/fixtures/centos-7.7/sfdisk-l-multi.out

@@ -0,0 +1,21 @@
+
+Disk /dev/sda: 2610 cylinders, 255 heads, 63 sectors/track
+Units: cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0
+
+   Device Boot Start     End   #cyls    #blocks   Id  System
+/dev/sda1   *      0+    130-    131-   1048576   83  Linux
+/dev/sda2        130+   2610-   2481-  19921920   8e  Linux LVM
+/dev/sda3          0       -       0          0    0  Empty
+/dev/sda4          0       -       0          0    0  Empty
+
+Disk /dev/mapper/centos-root: 2218 cylinders, 255 heads, 63 sectors/track
+
+Disk /dev/mapper/centos-swap: 261 cylinders, 255 heads, 63 sectors/track
+Units: cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0
+
+   Device Boot Start     End   #cyls    #blocks   Id  System
+/dev/sda1   *      0+    130-    131-   1048576   83  Linux
+/dev/sda2        130+   2610-   2481-  19921920   8e  Linux LVM
+/dev/sda3          0       -       0          0    0  Empty
+/dev/sda4          0       -       0          0    0  Empty
+

tests/fixtures/centos-7.7/sfdisk-l.json

@@ -0,0 +1 @@
+[{"disk":"/dev/sda","cylinders":2610,"heads":255,"sectors_per_track":63,"units":"cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0","partitions":[{"device":"/dev/sda1","boot":true,"start":0,"end":130,"cyls":131,"blocks":1048576,"id":"83","system":"Linux"},{"device":"/dev/sda2","boot":false,"start":130,"end":2610,"cyls":2481,"blocks":19921920,"id":"8e","system":"Linux LVM"},{"device":"/dev/sda3","boot":false,"start":0,"end":null,"cyls":0,"blocks":0,"id":"0","system":"Empty"},{"device":"/dev/sda4","boot":false,"start":0,"end":null,"cyls":0,"blocks":0,"id":"0","system":"Empty"}]},{"disk":"/dev/mapper/centos-root","cylinders":2218,"heads":255,"sectors_per_track":63},{"disk":"/dev/mapper/centos-swap","cylinders":261,"heads":255,"sectors_per_track":63}]

tests/fixtures/centos-7.7/sfdisk-l.out

@@ -0,0 +1,14 @@
+
+Disk /dev/sda: 2610 cylinders, 255 heads, 63 sectors/track
+Units: cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0
+
+   Device Boot Start     End   #cyls    #blocks   Id  System
+/dev/sda1   *      0+    130-    131-   1048576   83  Linux
+/dev/sda2        130+   2610-   2481-  19921920   8e  Linux LVM
+/dev/sda3          0       -       0          0    0  Empty
+/dev/sda4          0       -       0          0    0  Empty
+
+Disk /dev/mapper/centos-root: 2218 cylinders, 255 heads, 63 sectors/track
+
+Disk /dev/mapper/centos-swap: 261 cylinders, 255 heads, 63 sectors/track
+

tests/fixtures/centos-7.7/sfdisk-luB.json

@@ -0,0 +1 @@
+[{"disk":"/dev/sda","cylinders":2610,"heads":255,"sectors_per_track":63,"units":"blocks of 1024 bytes, counting from 0","partitions":[{"device":"/dev/sda1","boot":true,"start":1024,"end":1049599,"blocks":1048576,"id":"83","system":"Linux"},{"device":"/dev/sda2","boot":false,"start":1049600,"end":20971519,"blocks":19921920,"id":"8e","system":"Linux LVM"},{"device":"/dev/sda3","boot":false,"start":0,"end":null,"blocks":0,"id":"0","system":"Empty"},{"device":"/dev/sda4","boot":false,"start":0,"end":null,"blocks":0,"id":"0","system":"Empty"}]},{"disk":"/dev/mapper/centos-root","cylinders":2218,"heads":255,"sectors_per_track":63},{"disk":"/dev/mapper/centos-swap","cylinders":261,"heads":255,"sectors_per_track":63}]

tests/fixtures/centos-7.7/sfdisk-luB.out

@@ -0,0 +1,14 @@
+
+Disk /dev/sda: 2610 cylinders, 255 heads, 63 sectors/track
+Units: blocks of 1024 bytes, counting from 0
+
+   Device Boot   Start       End    #blocks   Id  System
+/dev/sda1   *     1024   1049599    1048576   83  Linux
+/dev/sda2      1049600  20971519   19921920   8e  Linux LVM
+/dev/sda3            0         -          0    0  Empty
+/dev/sda4            0         -          0    0  Empty
+
+Disk /dev/mapper/centos-root: 2218 cylinders, 255 heads, 63 sectors/track
+
+Disk /dev/mapper/centos-swap: 261 cylinders, 255 heads, 63 sectors/track
+

tests/fixtures/centos-7.7/sfdisk-luM.json

@@ -0,0 +1 @@
+[{"disk":"/dev/sda","cylinders":2610,"heads":255,"sectors_per_track":63,"units":"1MiB = 1024*1024 bytes, blocks of 1024 bytes, counting from 0","partitions":[{"device":"/dev/sda1","boot":true,"start":1,"end":1024,"mib":1024,"blocks":1048576,"id":"83","system":"Linux"},{"device":"/dev/sda2","boot":false,"start":1025,"end":20479,"mib":19455,"blocks":19921920,"id":"8e","system":"Linux LVM"},{"device":"/dev/sda3","boot":false,"start":0,"end":null,"mib":0,"blocks":0,"id":"0","system":"Empty"},{"device":"/dev/sda4","boot":false,"start":0,"end":null,"mib":0,"blocks":0,"id":"0","system":"Empty"}]},{"disk":"/dev/mapper/centos-root","cylinders":2218,"heads":255,"sectors_per_track":63},{"disk":"/dev/mapper/centos-swap","cylinders":261,"heads":255,"sectors_per_track":63}]

tests/fixtures/centos-7.7/sfdisk-luM.out

@@ -0,0 +1,14 @@
+
+Disk /dev/sda: 2610 cylinders, 255 heads, 63 sectors/track
+Units: 1MiB = 1024*1024 bytes, blocks of 1024 bytes, counting from 0
+
+   Device Boot Start   End    MiB    #blocks   Id  System
+/dev/sda1   *     1   1024   1024    1048576   83  Linux
+/dev/sda2      1025  20479  19455   19921920   8e  Linux LVM
+/dev/sda3         0      -      0          0    0  Empty
+/dev/sda4         0      -      0          0    0  Empty
+
+Disk /dev/mapper/centos-root: 2218 cylinders, 255 heads, 63 sectors/track
+
+Disk /dev/mapper/centos-swap: 261 cylinders, 255 heads, 63 sectors/track
+

tests/fixtures/centos-7.7/sfdisk-luS.json

@@ -0,0 +1 @@
+[{"disk":"/dev/sda","cylinders":2610,"heads":255,"sectors_per_track":63,"units":"sectors of 512 bytes, counting from 0","partitions":[{"device":"/dev/sda1","boot":true,"start":2048,"end":2099199,"sectors":2097152,"id":"83","system":"Linux"},{"device":"/dev/sda2","boot":false,"start":2099200,"end":41943039,"sectors":39843840,"id":"8e","system":"Linux LVM"},{"device":"/dev/sda3","boot":false,"start":0,"end":null,"sectors":0,"id":"0","system":"Empty"},{"device":"/dev/sda4","boot":false,"start":0,"end":null,"sectors":0,"id":"0","system":"Empty"}]},{"disk":"/dev/mapper/centos-root","cylinders":2218,"heads":255,"sectors_per_track":63},{"disk":"/dev/mapper/centos-swap","cylinders":261,"heads":255,"sectors_per_track":63}]

tests/fixtures/centos-7.7/sfdisk-luS.out

@@ -0,0 +1,14 @@
+
+Disk /dev/sda: 2610 cylinders, 255 heads, 63 sectors/track
+Units: sectors of 512 bytes, counting from 0
+
+   Device Boot    Start       End   #sectors  Id  System
+/dev/sda1   *      2048   2099199    2097152  83  Linux
+/dev/sda2       2099200  41943039   39843840  8e  Linux LVM
+/dev/sda3             0         -          0   0  Empty
+/dev/sda4             0         -          0   0  Empty
+
+Disk /dev/mapper/centos-root: 2218 cylinders, 255 heads, 63 sectors/track
+
+Disk /dev/mapper/centos-swap: 261 cylinders, 255 heads, 63 sectors/track
+

tests/fixtures/centos-7.7/vmstat-a-streaming.json

@@ -0,0 +1 @@
+[{"runnable_procs":2,"uninterruptible_sleeping_procs":0,"virtual_mem_used":0,"free_mem":2794696,"buffer_mem":null,"cache_mem":null,"inactive_mem":295968,"active_mem":312736,"swap_in":0,"swap_out":0,"blocks_in":1,"blocks_out":3,"interrupts":29,"context_switches":57,"user_time":0,"system_time":0,"idle_time":99,"io_wait_time":0,"stolen_time":0,"timestamp":null,"timezone":null}]

tests/fixtures/centos-7.7/vmstat-a.json

@@ -0,0 +1 @@
+[{"runnable_procs":2,"uninterruptible_sleeping_procs":0,"virtual_mem_used":0,"free_mem":2794696,"buffer_mem":null,"cache_mem":null,"inactive_mem":295968,"active_mem":312736,"swap_in":0,"swap_out":0,"blocks_in":1,"blocks_out":3,"interrupts":29,"context_switches":57,"user_time":0,"system_time":0,"idle_time":99,"io_wait_time":0,"stolen_time":0,"timestamp":null,"timezone":null}]

tests/fixtures/centos-7.7/vmstat-a.out

@@ -0,0 +1,4 @@
+procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
+ r  b   swpd   free  inact active   si   so    bi    bo   in   cs us sy id wa st
+ 2  0      0 2794696 295968 312736    0    0     1     3   29   57  0  0 99  0  0
+

tests/fixtures/centos-7.7/vmstat-at-5-10-streaming.json

@@ -0,0 +1 @@
+[{"runnable_procs":2,"uninterruptible_sleeping_procs":0,"virtual_mem_used":0,"free_mem":2794716,"buffer_mem":null,"cache_mem":null,"inactive_mem":295968,"active_mem":312724,"swap_in":0,"swap_out":0,"blocks_in":1,"blocks_out":3,"interrupts":29,"context_switches":57,"user_time":0,"system_time":0,"idle_time":99,"io_wait_time":0,"stolen_time":0,"timestamp":"2021-09-16 20:33:13","timezone":"PDT","epoch":1631849593,"epoch_utc":null},{"runnable_procs":0,"uninterruptible_sleeping_procs":0,"virtual_mem_used":0,"free_mem":2794716,"buffer_mem":null,"cache_mem":null,"inactive_mem":295968,"active_mem":312744,"swap_in":0,"swap_out":0,"blocks_in":0,"blocks_out":0,"interrupts":41,"context_switches":84,"user_time":0,"system_time":0,"idle_time":100,"io_wait_time":0,"stolen_time":0,"timestamp":"2021-09-16 20:33:18","timezone":"PDT","epoch":1631849598,"epoch_utc":null},{"runnable_procs":0,"uninterruptible_sleeping_procs":0,"virtual_mem_used":0,"free_mem":2794716,"buffer_mem":null,"cache_mem":null,"inactive_mem":295968,"active_mem":312752,"swap_in":0,"swap_out":0,"blocks_in":0,"blocks_out":0,"interrupts":41,"context_switches":84,"user_time":0,"system_time":0,"idle_time":100,"io_wait_time":0,"stolen_time":0,"timestamp":"2021-09-16 20:33:23","timezone":"PDT","epoch":1631849603,"epoch_utc":null},{"runnable_procs":0,"uninterruptible_sleeping_procs":0,"virtual_mem_used":0,"free_mem":2794716,"buffer_mem":null,"cache_mem":null,"inactive_mem":295968,"active_mem":312752,"swap_in":0,"swap_out":0,"blocks_in":0,"blocks_out":0,"interrupts":44,"context_switches":87,"user_time":0,"system_time":0,"idle_time":100,"io_wait_time":0,"stolen_time":0,"timestamp":"2021-09-16 20:33:28","timezone":"PDT","epoch":1631849608,"epoch_utc":null}]

tests/fixtures/centos-7.7/vmstat-at-5-10.json

@@ -0,0 +1 @@
+[{"runnable_procs":2,"uninterruptible_sleeping_procs":0,"virtual_mem_used":0,"free_mem":2794716,"buffer_mem":null,"cache_mem":null,"inactive_mem":295968,"active_mem":312724,"swap_in":0,"swap_out":0,"blocks_in":1,"blocks_out":3,"interrupts":29,"context_switches":57,"user_time":0,"system_time":0,"idle_time":99,"io_wait_time":0,"stolen_time":0,"timestamp":"2021-09-16 20:33:13","timezone":"PDT","epoch":1631849593,"epoch_utc":null},{"runnable_procs":0,"uninterruptible_sleeping_procs":0,"virtual_mem_used":0,"free_mem":2794716,"buffer_mem":null,"cache_mem":null,"inactive_mem":295968,"active_mem":312744,"swap_in":0,"swap_out":0,"blocks_in":0,"blocks_out":0,"interrupts":41,"context_switches":84,"user_time":0,"system_time":0,"idle_time":100,"io_wait_time":0,"stolen_time":0,"timestamp":"2021-09-16 20:33:18","timezone":"PDT","epoch":1631849598,"epoch_utc":null},{"runnable_procs":0,"uninterruptible_sleeping_procs":0,"virtual_mem_used":0,"free_mem":2794716,"buffer_mem":null,"cache_mem":null,"inactive_mem":295968,"active_mem":312752,"swap_in":0,"swap_out":0,"blocks_in":0,"blocks_out":0,"interrupts":41,"context_switches":84,"user_time":0,"system_time":0,"idle_time":100,"io_wait_time":0,"stolen_time":0,"timestamp":"2021-09-16 20:33:23","timezone":"PDT","epoch":1631849603,"epoch_utc":null},{"runnable_procs":0,"uninterruptible_sleeping_procs":0,"virtual_mem_used":0,"free_mem":2794716,"buffer_mem":null,"cache_mem":null,"inactive_mem":295968,"active_mem":312752,"swap_in":0,"swap_out":0,"blocks_in":0,"blocks_out":0,"interrupts":44,"context_switches":87,"user_time":0,"system_time":0,"idle_time":100,"io_wait_time":0,"stolen_time":0,"timestamp":"2021-09-16 20:33:28","timezone":"PDT","epoch":1631849608,"epoch_utc":null}]

tests/fixtures/centos-7.7/vmstat-at-5-10.out

@@ -0,0 +1,7 @@
+procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- -----timestamp-----
+ r  b   swpd   free  inact active   si   so    bi    bo   in   cs us sy id wa st                 PDT
+ 2  0      0 2794716 295968 312724    0    0     1     3   29   57  0  0 99  0  0 2021-09-16 20:33:13
+ 0  0      0 2794716 295968 312744    0    0     0     0   41   84  0  0 100  0  0 2021-09-16 20:33:18
+ 0  0      0 2794716 295968 312752    0    0     0     0   41   84  0  0 100  0  0 2021-09-16 20:33:23
+ 0  0      0 2794716 295968 312752    0    0     0     0   44   87  0  0 100  0  0 2021-09-16 20:33:28
+

tests/fixtures/centos-7.7/vmstat-awt-streaming.json

@@ -0,0 +1 @@
+[{"runnable_procs":2,"uninterruptible_sleeping_procs":0,"virtual_mem_used":0,"free_mem":2794864,"buffer_mem":null,"cache_mem":null,"inactive_mem":295968,"active_mem":312724,"swap_in":0,"swap_out":0,"blocks_in":1,"blocks_out":3,"interrupts":29,"context_switches":57,"user_time":0,"system_time":0,"idle_time":99,"io_wait_time":0,"stolen_time":0,"timestamp":"2021-09-16 20:31:30","timezone":"PDT","epoch":1631849490,"epoch_utc":null}]

tests/fixtures/centos-7.7/vmstat-awt.json

@@ -0,0 +1 @@
+[{"runnable_procs":2,"uninterruptible_sleeping_procs":0,"virtual_mem_used":0,"free_mem":2794864,"buffer_mem":null,"cache_mem":null,"inactive_mem":295968,"active_mem":312724,"swap_in":0,"swap_out":0,"blocks_in":1,"blocks_out":3,"interrupts":29,"context_switches":57,"user_time":0,"system_time":0,"idle_time":99,"io_wait_time":0,"stolen_time":0,"timestamp":"2021-09-16 20:31:30","timezone":"PDT","epoch":1631849490,"epoch_utc":null}]

tests/fixtures/centos-7.7/vmstat-awt.out

@@ -0,0 +1,4 @@
+procs -----------------------memory---------------------- ---swap-- -----io---- -system-- --------cpu-------- -----timestamp-----
+ r  b         swpd         free        inact       active   si   so    bi    bo   in   cs  us  sy  id  wa  st                 PDT
+ 2  0            0      2794864       295968       312724    0    0     1     3   29   57   0   0  99   0   0 2021-09-16 20:31:30
+

tests/fixtures/centos-7.7/vmstat-d-streaming.json

@@ -0,0 +1 @@
+[{"disk":"sda","total_reads":15257,"merged_reads":100,"sectors_read":841035,"reading_ms":8395,"total_writes":50851,"merged_writes":5502,"sectors_written":1648657,"writing_ms":146540,"current_io":0,"io_seconds":44,"timestamp":null,"timezone":null},{"disk":"sr0","total_reads":0,"merged_reads":0,"sectors_read":0,"reading_ms":0,"total_writes":0,"merged_writes":0,"sectors_written":0,"writing_ms":0,"current_io":0,"io_seconds":0,"timestamp":null,"timezone":null},{"disk":"dm-0","total_reads":14910,"merged_reads":0,"sectors_read":775075,"reading_ms":8350,"total_writes":56343,"merged_writes":0,"sectors_written":1644520,"writing_ms":180880,"current_io":0,"io_seconds":44,"timestamp":null,"timezone":null},{"disk":"dm-1","total_reads":88,"merged_reads":0,"sectors_read":4408,"reading_ms":13,"total_writes":0,"merged_writes":0,"sectors_written":0,"writing_ms":0,"current_io":0,"io_seconds":0,"timestamp":null,"timezone":null}]

tests/fixtures/centos-7.7/vmstat-d.json

@@ -0,0 +1 @@
+[{"disk":"sda","total_reads":15257,"merged_reads":100,"sectors_read":841035,"reading_ms":8395,"total_writes":50851,"merged_writes":5502,"sectors_written":1648657,"writing_ms":146540,"current_io":0,"io_seconds":44,"timestamp":null,"timezone":null},{"disk":"sr0","total_reads":0,"merged_reads":0,"sectors_read":0,"reading_ms":0,"total_writes":0,"merged_writes":0,"sectors_written":0,"writing_ms":0,"current_io":0,"io_seconds":0,"timestamp":null,"timezone":null},{"disk":"dm-0","total_reads":14910,"merged_reads":0,"sectors_read":775075,"reading_ms":8350,"total_writes":56343,"merged_writes":0,"sectors_written":1644520,"writing_ms":180880,"current_io":0,"io_seconds":44,"timestamp":null,"timezone":null},{"disk":"dm-1","total_reads":88,"merged_reads":0,"sectors_read":4408,"reading_ms":13,"total_writes":0,"merged_writes":0,"sectors_written":0,"writing_ms":0,"current_io":0,"io_seconds":0,"timestamp":null,"timezone":null}]

tests/fixtures/centos-7.7/vmstat-d.out

@@ -0,0 +1,7 @@
+disk- ------------reads------------ ------------writes----------- -----IO------
+       total merged sectors      ms  total merged sectors      ms    cur    sec
+sda    15257    100  841035    8395  50851   5502 1648657  146540      0     44
+sr0        0      0       0       0      0      0       0       0      0      0
+dm-0   14910      0  775075    8350  56343      0 1644520  180880      0     44
+dm-1      88      0    4408      13      0      0       0       0      0      0
+

tests/fixtures/centos-7.7/vmstat-dt-streaming.json

@@ -0,0 +1 @@
+[{"disk":"sda","total_reads":15257,"merged_reads":100,"sectors_read":841035,"reading_ms":8395,"total_writes":50851,"merged_writes":5502,"sectors_written":1648657,"writing_ms":146540,"current_io":0,"io_seconds":44,"timestamp":"2021-09-16 20:32:28","timezone":"UTC","epoch":1631849548,"epoch_utc":1631824348},{"disk":"sr0","total_reads":0,"merged_reads":0,"sectors_read":0,"reading_ms":0,"total_writes":0,"merged_writes":0,"sectors_written":0,"writing_ms":0,"current_io":0,"io_seconds":0,"timestamp":"2021-09-16 20:32:28","timezone":"UTC","epoch":1631849548,"epoch_utc":1631824348},{"disk":"dm-0","total_reads":14910,"merged_reads":0,"sectors_read":775075,"reading_ms":8350,"total_writes":56343,"merged_writes":0,"sectors_written":1644520,"writing_ms":180880,"current_io":0,"io_seconds":44,"timestamp":"2021-09-16 20:32:28","timezone":"UTC","epoch":1631849548,"epoch_utc":1631824348},{"disk":"dm-1","total_reads":88,"merged_reads":0,"sectors_read":4408,"reading_ms":13,"total_writes":0,"merged_writes":0,"sectors_written":0,"writing_ms":0,"current_io":0,"io_seconds":0,"timestamp":"2021-09-16 20:32:28","timezone":"UTC","epoch":1631849548,"epoch_utc":1631824348}]

tests/fixtures/centos-7.7/vmstat-dt.json

@@ -0,0 +1 @@
+[{"disk":"sda","total_reads":15257,"merged_reads":100,"sectors_read":841035,"reading_ms":8395,"total_writes":50851,"merged_writes":5502,"sectors_written":1648657,"writing_ms":146540,"current_io":0,"io_seconds":44,"timestamp":"2021-09-16 20:32:28","timezone":"UTC","epoch":1631849548,"epoch_utc":1631824348},{"disk":"sr0","total_reads":0,"merged_reads":0,"sectors_read":0,"reading_ms":0,"total_writes":0,"merged_writes":0,"sectors_written":0,"writing_ms":0,"current_io":0,"io_seconds":0,"timestamp":"2021-09-16 20:32:28","timezone":"UTC","epoch":1631849548,"epoch_utc":1631824348},{"disk":"dm-0","total_reads":14910,"merged_reads":0,"sectors_read":775075,"reading_ms":8350,"total_writes":56343,"merged_writes":0,"sectors_written":1644520,"writing_ms":180880,"current_io":0,"io_seconds":44,"timestamp":"2021-09-16 20:32:28","timezone":"UTC","epoch":1631849548,"epoch_utc":1631824348},{"disk":"dm-1","total_reads":88,"merged_reads":0,"sectors_read":4408,"reading_ms":13,"total_writes":0,"merged_writes":0,"sectors_written":0,"writing_ms":0,"current_io":0,"io_seconds":0,"timestamp":"2021-09-16 20:32:28","timezone":"UTC","epoch":1631849548,"epoch_utc":1631824348}]

tests/fixtures/centos-7.7/vmstat-dt.out

@@ -0,0 +1,7 @@
+disk- ------------reads------------ ------------writes----------- -----IO------ -----timestamp-----
+       total merged sectors      ms  total merged sectors      ms    cur    sec                 UTC
+sda    15257    100  841035    8395  50851   5502 1648657  146540      0     44 2021-09-16 20:32:28
+sr0        0      0       0       0      0      0       0       0      0      0 2021-09-16 20:32:28
+dm-0   14910      0  775075    8350  56343      0 1644520  180880      0     44 2021-09-16 20:32:28
+dm-1      88      0    4408      13      0      0       0       0      0      0 2021-09-16 20:32:28
+

tests/fixtures/centos-7.7/vmstat-streaming.json

@@ -0,0 +1 @@
+[{"runnable_procs":2,"uninterruptible_sleeping_procs":0,"virtual_mem_used":0,"free_mem":2794468,"buffer_mem":2108,"cache_mem":741208,"inactive_mem":null,"active_mem":null,"swap_in":0,"swap_out":0,"blocks_in":1,"blocks_out":3,"interrupts":29,"context_switches":57,"user_time":0,"system_time":0,"idle_time":99,"io_wait_time":0,"stolen_time":0,"timestamp":null,"timezone":null}]

tests/fixtures/centos-7.7/vmstat-w-streaming.json

@@ -0,0 +1 @@
+[{"runnable_procs":2,"uninterruptible_sleeping_procs":0,"virtual_mem_used":0,"free_mem":2794484,"buffer_mem":2108,"cache_mem":741208,"inactive_mem":null,"active_mem":null,"swap_in":0,"swap_out":0,"blocks_in":1,"blocks_out":3,"interrupts":29,"context_switches":57,"user_time":0,"system_time":0,"idle_time":99,"io_wait_time":0,"stolen_time":0,"timestamp":null,"timezone":null}]

tests/fixtures/centos-7.7/vmstat-w.json

@@ -0,0 +1 @@
+[{"runnable_procs":2,"uninterruptible_sleeping_procs":0,"virtual_mem_used":0,"free_mem":2794484,"buffer_mem":2108,"cache_mem":741208,"inactive_mem":null,"active_mem":null,"swap_in":0,"swap_out":0,"blocks_in":1,"blocks_out":3,"interrupts":29,"context_switches":57,"user_time":0,"system_time":0,"idle_time":99,"io_wait_time":0,"stolen_time":0,"timestamp":null,"timezone":null}]

tests/fixtures/centos-7.7/vmstat-w.out

@@ -0,0 +1,4 @@
+procs -----------------------memory---------------------- ---swap-- -----io---- -system-- --------cpu--------
+ r  b         swpd         free         buff        cache   si   so    bi    bo   in   cs  us  sy  id  wa  st
+ 2  0            0      2794484         2108       741208    0    0     1     3   29   57   0   0  99   0   0
+

tests/fixtures/centos-7.7/vmstat.json

@@ -0,0 +1 @@
+[{"runnable_procs":2,"uninterruptible_sleeping_procs":0,"virtual_mem_used":0,"free_mem":2794468,"buffer_mem":2108,"cache_mem":741208,"inactive_mem":null,"active_mem":null,"swap_in":0,"swap_out":0,"blocks_in":1,"blocks_out":3,"interrupts":29,"context_switches":57,"user_time":0,"system_time":0,"idle_time":99,"io_wait_time":0,"stolen_time":0,"timestamp":null,"timezone":null}]

tests/fixtures/centos-7.7/vmstat.out

@@ -0,0 +1,4 @@
+procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
+ r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
+ 2  0      0 2794468   2108 741208    0    0     1     3   29   57  0  0 99  0  0
+

tests/fixtures/centos-8/sfdisk-F.json

@@ -0,0 +1 @@
+[{"disk":"/dev/sda","free_disk_size":"0 B","free_bytes":0,"free_sectors":0,"units":"sectors of 1 * 512 = 512 bytes","logical_sector_size":512,"physical_sector_size":512}]

tests/fixtures/centos-8/sfdisk-F.out

@@ -0,0 +1,9 @@
+Unpartitioned space /dev/sda: 0 B, 0 bytes, 0 sectors
+Units: sectors of 1 * 512 = 512 bytes
+Sector size (logical/physical): 512 bytes / 512 bytes
+
+
+
+
+
+

tests/fixtures/centos-8/sfdisk-l.json

@@ -0,0 +1 @@
+[{"disk":"/dev/sda","disk_size":"20 GiB","bytes":21474836480,"sectors":41943040,"units":"sectors of 1 * 512 = 512 bytes","logical_sector_size":512,"physical_sector_size":512,"min_io_size":512,"optimal_io_size":512,"disk_label_type":"dos","disk_identifier":"0x94988ac4","partitions":[{"device":"/dev/sda1","boot":true,"start":2048,"end":2099199,"sectors":2097152,"size":"1G","id":"83","type":"Linux"},{"device":"/dev/sda2","boot":false,"start":2099200,"end":41943039,"sectors":39843840,"size":"19G","id":"8e","type":"Linux LVM"}]},{"disk":"/dev/mapper/cl-root","disk_size":"17 GiB","bytes":18249416704,"sectors":35643392,"units":"sectors of 1 * 512 = 512 bytes","logical_sector_size":512,"physical_sector_size":512,"min_io_size":512,"optimal_io_size":512},{"disk":"/dev/mapper/cl-swap","disk_size":"2 GiB","bytes":2147483648,"sectors":4194304,"units":"sectors of 1 * 512 = 512 bytes","logical_sector_size":512,"physical_sector_size":512,"min_io_size":512,"optimal_io_size":512}]

tests/fixtures/centos-8/sfdisk-l.out

@@ -0,0 +1,24 @@
+Disk /dev/sda: 20 GiB, 21474836480 bytes, 41943040 sectors
+Units: sectors of 1 * 512 = 512 bytes
+Sector size (logical/physical): 512 bytes / 512 bytes
+I/O size (minimum/optimal): 512 bytes / 512 bytes
+Disklabel type: dos
+Disk identifier: 0x94988ac4
+
+Device     Boot   Start      End  Sectors Size Id Type
+/dev/sda1  *       2048  2099199  2097152   1G 83 Linux
+/dev/sda2       2099200 41943039 39843840  19G 8e Linux LVM
+
+
+
+
+Disk /dev/mapper/cl-root: 17 GiB, 18249416704 bytes, 35643392 sectors
+Units: sectors of 1 * 512 = 512 bytes
+Sector size (logical/physical): 512 bytes / 512 bytes
+I/O size (minimum/optimal): 512 bytes / 512 bytes
+
+
+Disk /dev/mapper/cl-swap: 2 GiB, 2147483648 bytes, 4194304 sectors
+Units: sectors of 1 * 512 = 512 bytes
+Sector size (logical/physical): 512 bytes / 512 bytes
+I/O size (minimum/optimal): 512 bytes / 512 bytes

tests/fixtures/debian10/sfdisk-F.json

@@ -0,0 +1 @@
+[{"disk":"/dev/sda","free_disk_size":"0 B","free_bytes":0,"free_sectors":0,"units":"sectors of 1 * 512 = 512 bytes","logical_sector_size":512,"physical_sector_size":512}]

tests/fixtures/debian10/sfdisk-F.out

@@ -0,0 +1,3 @@
+Unpartitioned space /dev/sda: 0 B, 0 bytes, 0 sectors
+Units: sectors of 1 * 512 = 512 bytes
+Sector size (logical/physical): 512 bytes / 512 bytes

tests/fixtures/debian10/sfdisk-F2.json

@@ -0,0 +1 @@
+[{"disk":"/dev/sdc","free_disk_size":"1 GiB","free_bytes":1073741824,"free_sectors":2097152,"units":"sectors of 1 * 512 = 512 bytes","logical_sector_size":512,"physical_sector_size":512,"partitions":[{"start":4194304,"end":6291455,"sectors":2097152,"size":"1G"}]}]

tests/fixtures/debian10/sfdisk-F2.out

@@ -0,0 +1,6 @@
+Unpartitioned space /dev/sdc: 1 GiB, 1073741824 bytes, 2097152 sectors
+Units: sectors of 1 * 512 = 512 bytes
+Sector size (logical/physical): 512 bytes / 512 bytes
+
+  Start     End Sectors Size
+4194304 6291455 2097152   1G

tests/fixtures/debian10/sfdisk-l.json

@@ -0,0 +1 @@
+[{"disk":"/dev/sda","disk_size":"20 GiB","bytes":21474836480,"sectors":41943040,"disk_model":"VMware Virtual S","units":"sectors of 1 * 512 = 512 bytes","logical_sector_size":512,"physical_sector_size":512,"min_io_size":512,"optimal_io_size":512,"disk_label_type":"dos","disk_identifier":"0x3719ef0f","partitions":[{"device":"/dev/sda1","boot":true,"start":2048,"end":39942143,"sectors":39940096,"size":"19G","id":"83","type":"Linux"},{"device":"/dev/sda2","boot":false,"start":39944190,"end":41940991,"sectors":1996802,"size":"975M","id":"5","type":"Extended"},{"device":"/dev/sda5","boot":false,"start":39944192,"end":41940991,"sectors":1996800,"size":"975M","id":"82","type":"Linux swap / Solaris"}]}]

tests/fixtures/debian10/sfdisk-l.out

@@ -0,0 +1,12 @@
+Disk /dev/sda: 20 GiB, 21474836480 bytes, 41943040 sectors
+Disk model: VMware Virtual S
+Units: sectors of 1 * 512 = 512 bytes
+Sector size (logical/physical): 512 bytes / 512 bytes
+I/O size (minimum/optimal): 512 bytes / 512 bytes
+Disklabel type: dos
+Disk identifier: 0x3719ef0f
+
+Device     Boot    Start      End  Sectors  Size Id Type
+/dev/sda1  *        2048 39942143 39940096   19G 83 Linux
+/dev/sda2       39944190 41940991  1996802  975M  5 Extended
+/dev/sda5       39944192 41940991  1996800  975M 82 Linux swap / Solaris

tests/fixtures/debian10/sfdisk-l2.json

@@ -0,0 +1 @@
+[{"disk":"/dev/sda","disk_size":"32 GiB","bytes":34359738368,"sectors":67108864,"disk_model":"QEMU HARDDISK","units":"sectors of 1 * 512 = 512 bytes","logical_sector_size":512,"physical_sector_size":512,"min_io_size":512,"optimal_io_size":512,"disk_label_type":"dos","disk_identifier":"0x4f1ce2bf","partitions":[{"device":"/dev/sda1","boot":true,"start":2048,"end":13260799,"sectors":13258752,"size":"6.3G","id":"83","type":"Linux"},{"device":"/dev/sda2","boot":false,"start":13262846,"end":67106815,"sectors":53843970,"size":"25.7G","id":"5","type":"Extended"},{"device":"/dev/sda5","boot":false,"start":13262848,"end":18331647,"sectors":5068800,"size":"2.4G","id":"83","type":"Linux"},{"device":"/dev/sda6","boot":false,"start":18333696,"end":20332543,"sectors":1998848,"size":"976M","id":"82","type":"Linux swap / Solaris"},{"device":"/dev/sda7","boot":false,"start":20334592,"end":21352447,"sectors":1017856,"size":"497M","id":"83","type":"Linux"},{"device":"/dev/sda8","boot":false,"start":21354496,"end":67106815,"sectors":45752320,"size":"21.8G","id":"83","type":"Linux"}]}]

tests/fixtures/debian10/sfdisk-l2.out

@@ -0,0 +1,15 @@
+Disk /dev/sda: 32 GiB, 34359738368 bytes, 67108864 sectors
+Disk model: QEMU HARDDISK   
+Units: sectors of 1 * 512 = 512 bytes
+Sector size (logical/physical): 512 bytes / 512 bytes
+I/O size (minimum/optimal): 512 bytes / 512 bytes
+Disklabel type: dos
+Disk identifier: 0x4f1ce2bf
+
+Device     Boot    Start      End  Sectors  Size Id Type
+/dev/sda1  *        2048 13260799 13258752  6.3G 83 Linux
+/dev/sda2       13262846 67106815 53843970 25.7G  5 Extended
+/dev/sda5       13262848 18331647  5068800  2.4G 83 Linux
+/dev/sda6       18333696 20332543  1998848  976M 82 Linux swap / Solaris
+/dev/sda7       20334592 21352447  1017856  497M 83 Linux
+/dev/sda8       21354496 67106815 45752320 21.8G 83 Linux