Merge pull request #180 from kellyjonbrazil/dev

Dev v1.17.2

.github/workflows/pythonapp.yml

@@ -14,7 +14,7 @@ jobs:
     strategy:
       matrix:
         os: [macos-latest, ubuntu-latest, windows-latest]
-        python-version: [3.6, 3.7, 3.8, 3.9]
+        python-version: [3.6, 3.7, 3.8, 3.9, 3.10.0]
 
     steps:
     - uses: actions/checkout@v2

CHANGELOG

@@ -1,8 +1,65 @@
 jc changelog
 
+20211117 v1.17.2
+- Fix ping parser to add Alpine linux support
+- Fix netstat parser for older versions of netstat on linux
+- Fix df parser for cases where the filesystem field overflows the column length
+
+20211030 v1.17.1
+- Fix file parser for gzip files
+- Fix uname parser for cases where the 'processor' and/or 'hardware_platform' fields are missing on linux
+- Fix uname parser on FreeBSD
+- Add lsusb parser tested on linux
+- Add CSV file streaming parser
+- Add testing for Python 3.10.0
+
+20210923 v1.17.0
+- Note to Package Maintainers: please see note at 20210720 v1.16.0
+- Add wrapping of warning and error messages
+- Add vmstat parser tested on linux
+- Add support for streaming parsers
+- Add ls command streaming parser tested on linux, macOS, and freeBSD
+- Add ping command streaming parser tested on linux, macOS, and freeBSD
+- Add vmstat command streaming parser tested on linux
+- Add -u option to allow unbuffered output
+
+20210830 v1.16.2
+- Note to Package Maintainers: please see note at 20210720 v1.16.0
+- Update sfdisk parser to support the -F option and newer versions of sfdisk
+
+20210813 v1.16.1
+- Note to Package Maintainers: please see note at 20210720 v1.16.0
+- Fix issue with process substitution with the magic syntax
+- Fix issue with globs not including filenames with spaces with magic syntax
+- Fix stat parser to properly handle filenames with spaces on macOS/BSD
+
+20210720 v1.16.0
+- Note to Package Maintainers:
+    TL;DR: `/man/jc.1.gz` and `/jc/man/jc.1.gz` are deprecated and only `/man/jc.1` should be used.
+    
+    The Man page in the PyPi source packages will be moving from `/jc/man/jc.1.gz` to `/man/jc.1`
+    in version 1.17.0. For now the Man pages will be available in both locations, but be aware that
+    the Man page at `/jc/man/jc.1.gz` is now considered deprecated.
+
+    Also, starting in v1.17.0, the Man page will no longer be compressed in the source package,
+    therefore `/man/jc.1.gz` should also be considered deprecated and will no longer be available
+    after v1.17.0. Please use `/man/jc.1` and compress downstream if you would like.
+- Include CHANGELOG in source distribution
+- Fix Man page location in source packages
+- Add sfdisk command parser tested on linux
+- Update unit test files to change the timezone when needed (POSIX only)
+- Binaries and DEB/RPM/MSI packages now include Python 3.9.5 interpreter
+
+20210628 v1.15.6
+- Fix issue to only load local plugin parsers that have filenames that end in .py 
+
+20210520 v1.15.5
+- Fix issue where help and about information would not display if a 3rd party parser library was missing. (e.g. xmltodict)
+- Add more error message detail when encountering ParseError and LibraryNotFound exceptions
+
 20210517 v1.15.4
 - Update ping parser to support error responses in OSX and BSD
-- Update ping parser to be more resillient against parsing errors for unknown error types
+- Update ping parser to be more resilient against parsing errors for unknown error types
 - Update dig parser to support `+noall +answer` use case
 - Update dig parser compatibility to all platforms
 - Fix colors in Windows terminals (cmd.exe and PowerShell)

CONTRIBUTING.md

@@ -15,11 +15,12 @@ Pull requests are the best way to propose changes to the codebase (we use [Githu
 
 1. Open an issue to discuss the new feature, bug fix, or parser before opening a pull request. For new parsers, it is important to agree upon a schema before developing the parser.
 2. Fork the repo and create your branch from `dev`, if available, otherwise `master`.
-3. If you've added code that should be tested, add tests. All new parsers should have several sample outputs and tests.
-4. Documentation is auto-generated from docstrings, so ensure they are clear and accurate.
-5. Ensure the test suite passes. (Note: "**America/Los_Angeles**" timezone should be configured on the test system)
-6. Make sure your code lints.
-7. Issue that pull request!
+3. For new parsers: Use the [`jc/parsers/foo.py`](https://github.com/kellyjonbrazil/jc/blob/master/jc/parsers/foo.py) or [`jc/parsers/foo_s.py (streaming)`](https://github.com/kellyjonbrazil/jc/blob/master/jc/parsers/foo_s.py) parsers as a template to get started. You can even place a new parser python module file in the [parser plugin directory](https://github.com/kellyjonbrazil/jc#custom-parsers) to get started right away with just a standard `jc` installation.
+4. If you've added code that should be tested, add tests. All new parsers should have several sample outputs and tests.
+5. Documentation is auto-generated from docstrings, so ensure they are clear and accurate.
+6. Ensure the test suite passes. (Note: "**America/Los_Angeles**" timezone should be configured on the test system)
+7. Make sure your code lints.
+8. Issue that pull request!
 
 ## Parser Schema Guidelines
 - Try to keep the schema as flat as possible - typically a list of flat dictionaries
@@ -31,31 +32,31 @@ This will make it easier to use tools like `jq` without requiring escaping of sp
 **Examples**
 
 Bad:
-```
+```json
 {
   "Interface 1": [
-    192.168.1.1,
-    172.16.1.1
+    "192.168.1.1",
+    "172.16.1.1"
   ],
   "Wifi Interface 1": [
-    10.1.1.1
+    "10.1.1.1"
   ]
 }
 ```
 Good:
-```
+```json
 [
   {
     "interface": "Interface 1",
     "ip_addresses": [
-      192.168.1.1,
-      172.16.1.1
+      "192.168.1.1",
+      "172.16.1.1"
     ]
   },
   {
     "interface": "Wifi Interface 1",
     "ip_addresses": [
-      10.1.1.1
+      "10.1.1.1"
     ]
   }
 ]

EXAMPLES.md

@@ -1917,6 +1917,128 @@ lsof | jc --lsof -p          # or:  jc -p lsof
   }
 ]
 ```
+### lsusb
+```bash
+lsusb -v | jc --lsusb -p          # or: jc -p lsusb -v
+```
+```json
+[
+  {
+    "bus": "002",
+    "device": "001",
+    "id": "1d6b:0001",
+    "description": "Linux Foundation 1.1 root hub",
+    "device_descriptor": {
+      "bLength": {
+        "value": "18"
+      },
+      "bDescriptorType": {
+        "value": "1"
+      },
+      "bcdUSB": {
+        "value": "1.10"
+      },
+      ...
+      "bNumConfigurations": {
+        "value": "1"
+      },
+      "configuration_descriptor": {
+        "bLength": {
+          "value": "9"
+        },
+        ...
+        "iConfiguration": {
+          "value": "0"
+        },
+        "bmAttributes": {
+          "value": "0xe0",
+          "attributes": [
+            "Self Powered",
+            "Remote Wakeup"
+          ]
+        },
+        "MaxPower": {
+          "description": "0mA"
+        },
+        "interface_descriptors": [
+          {
+            "bLength": {
+              "value": "9"
+            },
+            ...
+            "bInterfaceProtocol": {
+              "value": "0",
+              "description": "Full speed (or root) hub"
+            },
+            "iInterface": {
+              "value": "0"
+            },
+            "endpoint_descriptors": [
+              {
+                "bLength": {
+                  "value": "7"
+                },
+                ...
+                "bmAttributes": {
+                  "value": "3",
+                  "attributes": [
+                    "Transfer Type  Interrupt",
+                    "Synch Type  None",
+                    "Usage Type  Data"
+                  ]
+                },
+                "wMaxPacketSize": {
+                  "value": "0x0002",
+                  "description": "1x 2 bytes"
+                },
+                "bInterval": {
+                  "value": "255"
+                }
+              }
+            ]
+          }
+        ]
+      }
+    },
+    "hub_descriptor": {
+      "bLength": {
+        "value": "9"
+      },
+      ...
+      "wHubCharacteristic": {
+        "value": "0x000a",
+        "attributes": [
+          "No power switching (usb 1.0)",
+          "Per-port overcurrent protection"
+        ]
+      },
+      ...
+      "hub_port_status": {
+        "Port 1": {
+          "value": "0000.0103",
+          "attributes": [
+            "power",
+            "enable",
+            "connect"
+          ]
+        },
+        "Port 2": {
+          "value": "0000.0103",
+          "attributes": [
+            "power",
+            "enable",
+            "connect"
+          ]
+        }
+      }
+    },
+    "device_status": {
+      "value": "0x0001",
+      "description": "Self Powered"
+    }
+  }
+]
+```
 ### mount
 ```bash
 mount | jc --mount -p          # or:  jc -p mount
@@ -2556,6 +2678,75 @@ rpm_qia | jc --rpm_qi -p          # or:  jc -p rpm -qia
   }
 ]
 ```
+### sfdisk
+```bash
+sfdisk -l | jc --sfdisk -p          # or jc -p sfdisk -l
+```
+```json
+[
+  {
+    "disk": "/dev/sda",
+    "cylinders": 2610,
+    "heads": 255,
+    "sectors_per_track": 63,
+    "units": "cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0",
+    "partitions": [
+      {
+        "device": "/dev/sda1",
+        "boot": true,
+        "start": 0,
+        "end": 130,
+        "cyls": 131,
+        "blocks": 1048576,
+        "id": "83",
+        "system": "Linux"
+      },
+      {
+        "device": "/dev/sda2",
+        "boot": false,
+        "start": 130,
+        "end": 2610,
+        "cyls": 2481,
+        "blocks": 19921920,
+        "id": "8e",
+        "system": "Linux LVM"
+      },
+      {
+        "device": "/dev/sda3",
+        "boot": false,
+        "start": 0,
+        "end": null,
+        "cyls": 0,
+        "blocks": 0,
+        "id": "0",
+        "system": "Empty"
+      },
+      {
+        "device": "/dev/sda4",
+        "boot": false,
+        "start": 0,
+        "end": null,
+        "cyls": 0,
+        "blocks": 0,
+        "id": "0",
+        "system": "Empty"
+      }
+    ]
+  },
+  {
+    "disk": "/dev/mapper/centos-root",
+    "cylinders": 2218,
+    "heads": 255,
+    "sectors_per_track": 63
+  },
+  {
+    "disk": "/dev/mapper/centos-swap",
+    "cylinders": 261,
+    "heads": 255,
+    "sectors_per_track": 63
+  }
+]
+```
 ### /etc/shadow file
 ```bash
 cat /etc/shadow | jc --shadow -p

MANIFEST.in

@@ -1 +1,3 @@
 include jc/man/jc.1.gz
+include man/jc.1
+include CHANGELOG

README.md

@@ -1,9 +1,9 @@
 ![Tests](https://github.com/kellyjonbrazil/jc/workflows/Tests/badge.svg?branch=master)
 ![Pypi](https://img.shields.io/pypi/v/jc.svg)
 
-> Try the new `jc` [web demo](https://jc-web-demo.herokuapp.com/)!
+> Try the `jc` [web demo](https://jc-web-demo.herokuapp.com/)
 
-> JC is [now available](https://galaxy.ansible.com/community/general) as an Ansible filter plugin in the `community.general` collection! See this [blog post](https://blog.kellybrazil.com/2020/08/30/parsing-command-output-in-ansible-with-jc/) for an example.
+> JC is [now available](https://galaxy.ansible.com/community/general) as an Ansible filter plugin in the `community.general` collection. See this [blog post](https://blog.kellybrazil.com/2020/08/30/parsing-command-output-in-ansible-with-jc/) for an example.
 
 # JC
 JSON CLI output utility
@@ -19,7 +19,7 @@ dig example.com | jc --dig
 39049,"data":"93.184.216.34"}],"query_time":49,"server":"2600:1700:bab0:d40::1#53(2600:1700:bab0:d40::1)","when":
 "Fri Apr 16 16:09:00 PDT 2021","rcvd":56,"when_epoch":1618614540,"when_epoch_utc":null}]
 ```
-This allows further command-line processing of output with tools like `jq` by piping commands:
+This allows further command-line processing of output with tools like `jq` or [`jello`](https://github.com/kellyjonbrazil/jello) by piping commands:
 ```bash
 $ dig example.com | jc --dig | jq -r '.[].answer[].data'
 93.184.216.34
@@ -31,28 +31,13 @@ $ jc dig example.com | jq -r '.[].answer[].data'
 ```
 The `jc` parsers can also be used as python modules. In this case the output will be a python dictionary, or list of dictionaries, instead of JSON:
 ```python
+>>> import subprocess
 >>> import jc.parsers.dig
 >>> 
->>> data = '''; <<>> DiG 9.10.6 <<>> example.com
-... ;; global options: +cmd
-... ;; Got answer:
-... ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64612
-... ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
-... 
-... ;; OPT PSEUDOSECTION:
-... ; EDNS: version: 0, flags:; udp: 4096
-... ;; QUESTION SECTION:
-... ;example.com.           IN  A
-... 
-... ;; ANSWER SECTION:
-... example.com.        29658   IN  A   93.184.216.34
-... 
-... ;; Query time: 52 msec
-... ;; SERVER: 2600:1700:bab0:d40::1#53(2600:1700:bab0:d40::1)
-... ;; WHEN: Fri Apr 16 16:13:00 PDT 2021
-... ;; MSG SIZE  rcvd: 56'''
+>>> cmd_output = subprocess.check_output(['dig', 'example.com'], text=True)
+>>> data = jc.parsers.dig.parse(cmd_output)
 >>>
->>> jc.parsers.dig.parse(data)
+>>> data
 [{'id': 64612, 'opcode': 'QUERY', 'status': 'NOERROR', 'flags': ['qr', 'rd', 'ra'], 'query_num': 1, 'answer_num':
 1, 'authority_num': 0, 'additional_num': 1, 'opt_pseudosection': {'edns': {'version': 0, 'flags': [], 'udp':
 4096}}, 'question': {'name': 'example.com.', 'class': 'IN', 'type': 'A'}, 'answer': [{'name': 'example.com.',
@@ -60,7 +45,7 @@ The `jc` parsers can also be used as python modules. In this case the output wil
 '2600:1700:bab0:d40::1#53(2600:1700:bab0:d40::1)', 'when': 'Fri Apr 16 16:13:00 PDT 2021', 'rcvd': 56,
 'when_epoch': 1618614780, 'when_epoch_utc': None}]
 ```
-Two representations of the data are possible. The default representation uses a strict schema per parser and converts known numbers to int/float JSON values. Certain known values of `None` are converted to JSON `null`, known boolean values are converted, and, in some cases, additional semantic context fields are added.
+Two representations of the data are available. The default representation uses a strict schema per parser and converts known numbers to int/float JSON values. Certain known values of `None` are converted to JSON `null`, known boolean values are converted, and, in some cases, additional semantic context fields are added.
 
 To access the raw, pre-processed JSON, use the `-r` cli option or the `raw=True` function parameter in `parse()`.
 
@@ -75,6 +60,8 @@ See also:
 - [libxo on FreeBSD](http://juniper.github.io/libxo/libxo-manual.html)
 - [powershell](https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/convertto-json?view=powershell-7)
 - [blog: linux apps should have a json flag](https://thomashunter.name/posts/2012-06-06-linux-cli-apps-should-have-a-json-flag)
+- [Hacker News discussion](https://news.ycombinator.com/item?id=28266193)
+- [Reddit discussion](https://www.reddit.com/r/programming/comments/pa4cbb/bringing_the_unix_philosophy_to_the_21st_century/)
 
 Use Cases:
 - [Bash scripting](https://blog.kellybrazil.com/2021/04/12/practical-json-at-the-command-line/)
@@ -83,7 +70,7 @@ Use Cases:
 - [Nornir command output parsing](https://blog.kellybrazil.com/2020/12/09/parsing-command-output-in-nornir-with-jc/)
 
 ## Installation
-There are several ways to get `jc`. You can install via `pip`; other OS package repositories like `apt-get`, `dnf`, `zypper`, `pacman`, `nix-env`, `guix`, `brew`, or `portsnap`; via DEB/RPM packaged binaries; or by downloading the correct binary for your architecture and running it anywhere on your filesystem.
+There are several ways to get `jc`. You can install via `pip`, OS package repositories, via DEB/RPM/MSI packaged binaries for linux and Windows, or by downloading the correct binary for your architecture and running it anywhere on your filesystem.
 
 ### Pip (macOS, linux, unix, Windows)
 ```bash
@@ -100,7 +87,7 @@ pip3 install jc
 | Arch linux            | `pacman -S jc`                                                                |
 | NixOS linux           | `nix-env -iA nixpkgs.jc` or `nix-env -iA nixos.jc`                            |
 | Guix System linux     | `guix install jc`                                                             |
-| MacOS                 | `brew install jc`                                                             |
+| macOS                 | `brew install jc`                                                             |
 | FreeBSD               | `portsnap fetch update && cd /usr/ports/textproc/py-jc && make install clean` |
 | Ansible filter plugin | `ansible-galaxy collection install community.general`                         |
 
@@ -128,6 +115,7 @@ The JSON output can be compact (default) or pretty formatted with the `-p` optio
 - `--crontab` enables the `crontab` command and file parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/crontab))
 - `--crontab-u` enables the `crontab` file parser with user support ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/crontab_u))
 - `--csv` enables the CSV file parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/csv))
+- `--csv-s` enables the CSV file streaming parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/csv_s))
 - `--date` enables the `date` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/date))
 - `--df` enables the `df` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/df))
 - `--dig` enables the `dig` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/dig))
@@ -156,19 +144,23 @@ The JSON output can be compact (default) or pretty formatted with the `-p` optio
 - `--kv` enables the Key/Value file parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/kv))
 - `--last` enables the `last` and `lastb` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/last))
 - `--ls` enables the `ls` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ls))
+- `--ls-s` enables the `ls` command streaming parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ls_s))
 - `--lsblk` enables the `lsblk` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/lsblk))
 - `--lsmod` enables the `lsmod` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/lsmod))
 - `--lsof` enables the `lsof` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/lsof))
+- `--lsusb` enables the `lsusb` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/lsusb))
 - `--mount` enables the `mount` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/mount))
 - `--netstat` enables the `netstat` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/netstat))
 - `--ntpq` enables the `ntpq -p` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ntpq))
 - `--passwd` enables the `/etc/passwd` file parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/passwd))
 - `--ping` enables the `ping` and `ping6` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ping))
+- `--ping-s` enables the `ping` and `ping6` command streaming parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ping_s))
 - `--pip-list` enables the `pip list` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/pip_list))
 - `--pip-show` enables the `pip show` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/pip_show))
 - `--ps` enables the `ps` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ps))
 - `--route` enables the `route` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/route))
 - `--rpm-qi` enables the `rpm -qi` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/rpm_qi))
+- `--sfdisk` enables the `sfdisk` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/sfdisk))
 - `--shadow` enables the `/etc/shadow` file parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/shadow))
 - `--ss` enables the `ss` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ss))
 - `--stat` enables the `stat` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/stat))
@@ -187,6 +179,8 @@ The JSON output can be compact (default) or pretty formatted with the `-p` optio
 - `--uname` enables the `uname -a` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/uname))
 - `--upower` enables the `upower` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/upower))
 - `--uptime` enables the `uptime` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/uptime))
+- `--vmstat` enables the `vmstat` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/vmstat))
+- `--vmstat-s` enables the `vmstat` command streaming parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/vmstat_s))
 - `--w` enables the `w` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/w))
 - `--wc` enables the `wc` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/wc))
 - `--who` enables the `who` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/who))
@@ -196,11 +190,12 @@ The JSON output can be compact (default) or pretty formatted with the `-p` optio
 ### Options
 - `-a` about `jc`. Prints information about `jc` and the parsers (in JSON, of course!)
 - `-d` debug mode. Prints trace messages if parsing issues are encountered (use `-dd` for verbose debugging)
-- `-h` `jc` help. Use `jc -h --parser_name` for parser documentation
+- `-h` help. Use `jc -h --parser_name` for parser documentation
 - `-m` monochrome JSON output
 - `-p` pretty format the JSON output
-- `-q` quiet mode. Suppresses parser warning messages
+- `-q` quiet mode. Suppresses parser warning messages (use `-qq` to ignore streaming parser errors)
 - `-r` raw output. Provides a more literal JSON output, typically with string values and no additional semantic processing
+- `-u` unbuffer output
 - `-v` version information
 
 ### Exit Codes
@@ -232,6 +227,64 @@ or
 JC_COLORS=default,default,default,default
 ```
 
+### Streaming Parsers
+Most parsers load all of the data from STDIN, parse it, then output the entire JSON document serially. There are some streaming parsers (e.g. `ls-s` and `ping-s`) that immediately start processing and outputing the data line-by-line as [JSON Lines](https://jsonlines.org/) (aka [NDJSON](http://ndjson.org/)) while it is being received from STDIN. This can significantly reduce the amount of memory required to parse large amounts of command output (e.g. `ls -lR /`) and can sometimes process the data more quickly. Streaming parsers have slightly different behavior than standard parsers as outlined below.
+
+> Note: Streaming parsers cannot be used with the "magic" syntax
+
+#### Ignoring Errors
+
+You may want to ignore parsing errors when using streaming parsers since these may be used in long-lived processing pipelines and errors can break the pipe. To ignore parsing errors, use the `-qq` cli option or the `ignore_exceptions=True` argument with the `parse()` function. This will add a `_jc_meta` object to the JSON output with a `success` attribute. If `success` is `true`, then there were no issues parsing the line. If `success` is `false`, then a parsing issue was found and `error` and `line` fields will be added to include a short error description and the contents of the unparsable line, respectively:
+
+Successfully parsed line with `-qq` option:
+```json
+{
+  "command_data": "data",
+  "_jc_meta": {
+    "success": true
+  }
+}
+```
+Unsuccessfully parsed line with `-qq` option:
+```json
+{
+  "_jc_meta": {
+    "success": false,
+    "error": "error message",
+    "line": "original line data"
+  }
+}
+```
+
+#### Unbuffering Output
+
+Most operating systems will buffer output that is being piped from process to process. The buffer is usually around 4KB. When viewing the output in the terminal the OS buffer is not engaged so output is immediately displayed on the screen. When piping multiple processes together, though, it may seem as if the output is hanging when the input data is very slow (e.g. `ping`):
+```
+$ ping 1.1.1.1 | jc --ping-s | jq
+<slow output>
+```
+This is because the OS engages the 4KB buffer between `jc` and `jq` in this example. To display the data on the terminal in realtime, you can disable the buffer with the `-u` (unbuffer) cli option:
+```
+$ ping 1.1.1.1 | jc --ping-s -u | jq
+{"type":"reply","pattern":null,"timestamp":null,"bytes":"64","response_ip":"1.1.1.1","icmp_seq":"1","ttl":"128","time_ms":"24.6","duplicate":false}
+{"type":"reply","pattern":null,"timestamp":null,"bytes":"64","response_ip":"1.1.1.1","icmp_seq":"2","ttl":"128","time_ms":"26.8","duplicate":false}
+...
+```
+> Note: Unbuffered output can be slower for large data streams.
+
+#### Using Streaming Parsers as Python Modules
+
+Streaming parsers accept any iterable object and return a generator iterator object allowing lazy processing of the data. The input data should iterate on lines of string data. Examples of good input data are `sys.stdin` or `str.splitlines()`.
+
+To use the generator object in your code, simply loop through it or use the [next()](https://docs.python.org/3/library/functions.html#next) builtin function:
+```python
+import jc.parsers.ls_s
+
+result = jc.parsers.ls_s.parse(ls_command_output.splitlines())
+for item in result:
+    print(item["filename"])
+```
+
 ### Custom Parsers
 Custom local parser plugins may be placed in a `jc/jcparsers` folder in your local **"App data directory"**:
 
@@ -246,9 +299,10 @@ Local plugin filenames must be valid python module names, therefore must consist
 > Note: The application data directory follows the [XDG Base Directory Specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html)
 
 ### Caveats
-**Locale:**
 
-For best results set the `LANG` locale environment variable to `C`. For example, either by setting directly on the command-line:
+#### Locale
+
+For best results set the `LANG` locale environment variable to `C` or `en_US.UTF-8`. For example, either by setting directly on the command-line:
 ```
 $ LANG=C date | jc --date
 ```
@@ -257,7 +311,7 @@ or by exporting to the environment before running commands:
 $ export LANG=C
 ```
 
-**Timezones:**
+#### Timezones
 
 Some parsers have calculated epoch timestamp fields added to the output. Unless a timestamp field name has a `_utc` suffix it is considered naive. (i.e. based on the local timezone of the system the `jc` parser was run on).
 
@@ -266,11 +320,16 @@ If a UTC timezone can be detected in the text of the command output, the timesta
 ## Compatibility
 Some parsers like `dig`, `xml`, `csv`, etc. will work on any platform. Other parsers that convert platform-specific output will generate a warning message if they are run on an unsupported platform. To see all parser information, including compatibility, run `jc -ap`.
 
-You may still use a parser on an unsupported platform - for example, you may want to parse a file with linux `lsof` output on an macOS laptop. In that case you can suppress the warning message with the `-q` cli option or the `quiet=True` function parameter in `parse()`:
+You may still use a parser on an unsupported platform - for example, you may want to parse a file with linux `lsof` output on an macOS or Windows laptop. In that case you can suppress the warning message with the `-q` cli option or the `quiet=True` function parameter in `parse()`:
 
+macOS:
 ```bash
 cat lsof.out | jc --lsof -q
 ```
+or Windows:
+```bash
+type lsof.out | jc --lsof -q
+```
 
 Tested on:
 - Centos 7.7
@@ -282,9 +341,11 @@ Tested on:
 - NixOS
 - FreeBSD12
 - Windows 10
+- Windows 2016 Server
+- Windows 2019 Server
 
 ## Contributions
-Feel free to add/improve code or parsers! You can use the [`jc/parsers/foo.py`](https://github.com/kellyjonbrazil/jc/blob/master/jc/parsers/foo.py) parser as a template and submit your parser with a pull request.
+Feel free to add/improve code or parsers! You can use the [`jc/parsers/foo.py`](https://github.com/kellyjonbrazil/jc/blob/master/jc/parsers/foo.py) or [`jc/parsers/foo_s.py (streaming)`](https://github.com/kellyjonbrazil/jc/blob/master/jc/parsers/foo_s.py) parsers as a template and submit your parser with a pull request.
 
 Please see the [Contributing Guidelines](https://github.com/kellyjonbrazil/jc/blob/master/CONTRIBUTING.md) for more information.
 

docs/parsers/csv_s.md

@@ -0,0 +1,77 @@
+[Home](https://kellyjonbrazil.github.io/jc/)
+
+# jc.parsers.csv_s
+jc - JSON CLI output utility `csv` file streaming parser
+
+> This streaming parser outputs JSON Lines
+
+The `csv` streaming parser will attempt to automatically detect the delimiter character. If the delimiter cannot be detected it will default to comma. The first row of the file must be a header row.
+
+Note: The first 100 rows are read into memory to enable delimiter detection, then the rest of the rows are loaded lazily.
+
+Usage (cli):
+
+    $ cat file.csv | jc --csv-s
+
+Usage (module):
+
+    import jc.parsers.csv_s
+    result = jc.parsers.csv_s.parse(csv_output)
+
+Schema:
+
+    csv file converted to a Dictionary: https://docs.python.org/3/library/csv.html
+
+    {
+      "column_name1":     string,
+      "column_name2":     string
+    }
+
+Examples:
+
+    $ cat homes.csv
+    "Sell", "List", "Living", "Rooms", "Beds", "Baths", "Age", "Acres", "Taxes"
+    142, 160, 28, 10, 5, 3,  60, 0.28,  3167
+    175, 180, 18,  8, 4, 1,  12, 0.43,  4033
+    129, 132, 13,  6, 3, 1,  41, 0.33,  1471
+    ...
+
+    $ cat homes.csv | jc --csv-s
+    {"Sell":"142","List":"160","Living":"28","Rooms":"10","Beds":"5","Baths":"3","Age":"60","Acres":"0.28","Taxes":"3167"}
+    {"Sell":"175","List":"180","Living":"18","Rooms":"8","Beds":"4","Baths":"1","Age":"12","Acres":"0.43","Taxes":"4033"}
+    {"Sell":"129","List":"132","Living":"13","Rooms":"6","Beds":"3","Baths":"1","Age":"41","Acres":"0.33","Taxes":"1471"}
+    ...
+
+
+## info
+```python
+info()
+```
+Provides parser metadata (version, author, etc.)
+
+## parse
+```python
+parse(data, raw=False, quiet=False, ignore_exceptions=False)
+```
+
+Main text parsing generator function. Returns an iterator object.
+
+Parameters:
+
+    data:              (iterable)  line-based text data to parse (e.g. sys.stdin or str.splitlines())
+    raw:               (boolean)   output preprocessed JSON if True
+    quiet:             (boolean)   suppress warning messages if True
+    ignore_exceptions: (boolean)   ignore parsing exceptions if True
+
+Yields:
+
+    Dictionary. Raw or processed structured data.
+
+Returns:
+
+    Iterator object
+
+## Parser Information
+Compatibility:  linux, darwin, cygwin, win32, aix, freebsd
+
+Version 1.0 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/df.md

@@ -122,4 +122,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux, darwin, freebsd
 
-Version 1.7 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.8 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/file.md

@@ -87,4 +87,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux, aix, freebsd, darwin
 
-Version 1.3 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.4 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/kv.md

@@ -38,7 +38,7 @@ Examples:
     # but can be preserved with the -r argument
     occupation:"Engineer"
 
-    $ cat keyvalue.txt | jc --ini -p
+    $ cat keyvalue.txt | jc --kv -p
     {
       "name": "John Doe",
       "address": "555 California Drive",

docs/parsers/ls_s.md

@@ -0,0 +1,94 @@
+[Home](https://kellyjonbrazil.github.io/jc/)
+
+# jc.parsers.ls_s
+jc - JSON CLI output utility `ls` and `vdir` command output streaming parser
+
+> This streaming parser outputs JSON Lines
+
+Requires the `-l` option to be used on `ls`. If there are newline characters in the filename, then make sure to use the `-b` option on `ls`.
+
+The `jc` `-qq` option can be used to ignore parsing errors. (e.g. filenames with newline characters, but `-b` was not used)
+
+The `epoch` calculated timestamp field is naive (i.e. based on the local time of the system the parser is run on)
+
+The `epoch_utc` calculated timestamp field is timezone-aware and is only available if the timezone field is UTC.
+
+Usage (cli):
+
+    $ ls | jc --ls-s
+
+Usage (module):
+
+    import jc.parsers.ls_s
+    result = jc.parsers.ls_s.parse(ls_command_output.splitlines())    # result is an iterable object
+    for item in result:
+        # do something
+
+Schema:
+
+    {
+      "filename":       string,
+      "flags":          string,
+      "links":          integer,
+      "parent":         string,
+      "owner":          string,
+      "group":          string,
+      "size":           integer,
+      "date":           string,
+      "epoch":          integer,     # naive timestamp if date field exists and can be converted
+      "epoch_utc":      integer,     # timezone aware timestamp if date field is in UTC and can be converted
+      "_jc_meta":                    # This object only exists if using -qq or ignore_exceptions=True
+        {
+          "success":    booean,      # true if successfully parsed, false if error
+          "error":      string,      # exists if "success" is false
+          "line":       string       # exists if "success" is false
+        }
+    }
+
+Examples:
+
+    $ ls -l /usr/bin | jc --ls-s
+    {"filename":"2to3-","flags":"-rwxr-xr-x","links":4,"owner":"root","group":"wheel","size":925,"date":"Feb 22 2019"}
+    {"filename":"2to3-2.7","link_to":"../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/2to3-2.7","flags":"lrwxr-xr-x","links":1,"owner":"root","group":"wheel","size":74,"date":"May 4 2019"}
+    {"filename":"AssetCacheLocatorUtil","flags":"-rwxr-xr-x","links":1,"owner":"root","group":"wheel","size":55152,"date":"May 3 2019"}
+    ...
+
+    $ ls -l /usr/bin | jc --ls-s -r
+    {"filename":"2to3-","flags":"-rwxr-xr-x","links":"4","owner":"root","group":"wheel","size":"925","date":"Feb 22 2019"}
+    {"filename":"2to3-2.7","link_to":"../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/2to3-2.7","flags":"lrwxr-xr-x","links":"1","owner":"root","group":"wheel","size":"74","date":"May 4 2019"}
+    {"filename":"AssetCacheLocatorUtil","flags":"-rwxr-xr-x","links":"1","owner":"root","group":"wheel","size":"55152","date":"May 3 2019"}
+    ...
+
+
+## info
+```python
+info()
+```
+Provides parser metadata (version, author, etc.)
+
+## parse
+```python
+parse(data, raw=False, quiet=False, ignore_exceptions=False)
+```
+
+Main text parsing generator function. Returns an iterator object.
+
+Parameters:
+
+    data:              (iterable)  line-based text data to parse (e.g. sys.stdin or str.splitlines())
+    raw:               (boolean)   output preprocessed JSON if True
+    quiet:             (boolean)   suppress warning messages if True
+    ignore_exceptions: (boolean)   ignore parsing exceptions if True
+
+Yields:
+
+    Dictionary. Raw or processed structured data.
+
+Returns:
+
+    Iterator object
+
+## Parser Information
+Compatibility:  linux, darwin, cygwin, aix, freebsd
+
+Version 0.5 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/lsusb.md

@@ -0,0 +1,288 @@
+[Home](https://kellyjonbrazil.github.io/jc/)
+
+# jc.parsers.lsusb
+jc - JSON CLI output utility `lsusb` command output parser
+
+Supports the `-v` option or no options.
+
+Usage (cli):
+
+    $ lsusb -v | jc --lsusb
+
+    or
+
+    $ jc lsusb -v
+
+Usage (module):
+
+    import jc.parsers.lsusb
+    result = jc.parsers.lsusb.parse(lsusb_command_output)
+
+Schema:
+
+    Note: <item> object keynames are assigned directly from the lsusb output.
+          If there are duplicate <item> names in a section, only the last one is converted.
+
+    [
+      {
+        "bus":                                string,
+        "device":                             string,
+        "id":                                 string,
+        "description":                        string,
+        "device_descriptor": {
+          "<item>": {
+            "value":                          string,
+            "description":                    string,
+            "attributes": [
+                                              string
+            ]
+          },
+          "configuration_descriptor": {
+            "<item>": {
+              "value":                        string,
+              "description":                  string,
+              "attributes": [
+                                              string
+              ]
+            },
+            "interface_association": {
+              "<item>": {
+                "value":                      string,
+                "description":                string,
+                "attributes": [
+                                              string
+                ]
+              }
+            },
+            "interface_descriptors": [
+              {
+                "<item>": {
+                  "value":                    string,
+                  "description":              string,
+                  "attributes": [
+                                              string
+                  ]
+                },
+                "cdc_header": {
+                  "<item>": {
+                    "value":                  string,
+                    "description":            string,
+                    "attributes": [
+                                              string
+                    ]
+                  }
+                },
+                "cdc_call_management": {
+                  "<item>": {
+                    "value":                  string,
+                    "description":            string,
+                    "attributes": [
+                                              string
+                    ]
+                  }
+                },
+                "cdc_acm": {
+                  "<item>": {
+                    "value":                  string,
+                    "description":            string,
+                    "attributes": [
+                                              string
+                    ]
+                  }
+                },
+                "cdc_union": {
+                  "<item>": {
+                    "value":                  string,
+                    "description":            string,
+                    "attributes": [
+                                              string
+                    ]
+                  }
+                },
+                "endpoint_descriptors": [
+                  {
+                    "<item>": {
+                      "value":                string,
+                      "description":          string,
+                      "attributes": [
+                                              string
+                      ]
+                    }
+                  }
+                ]
+              }
+            ]
+          }
+        },
+        "hub_descriptor": {
+          "<item>": {
+            "value":                          string,
+            "description":                    string,
+            "attributes": [
+                                              string,
+            ]
+          },
+          "hub_port_status": {
+            "<item>": {
+              "value":                        string,
+              "attributes": [
+                                              string
+              ]
+            }
+          }
+        },
+        "device_status": {
+          "value":                            string,
+          "description":                      string
+        }
+      }
+    ]
+
+Examples:
+
+    $ lsusb -v | jc --lsusb -p
+    [
+      {
+        "bus": "002",
+        "device": "001",
+        "id": "1d6b:0001",
+        "description": "Linux Foundation 1.1 root hub",
+        "device_descriptor": {
+          "bLength": {
+            "value": "18"
+          },
+          "bDescriptorType": {
+            "value": "1"
+          },
+          "bcdUSB": {
+            "value": "1.10"
+          },
+          ...
+          "bNumConfigurations": {
+            "value": "1"
+          },
+          "configuration_descriptor": {
+            "bLength": {
+              "value": "9"
+            },
+            ...
+            "iConfiguration": {
+              "value": "0"
+            },
+            "bmAttributes": {
+              "value": "0xe0",
+              "attributes": [
+                "Self Powered",
+                "Remote Wakeup"
+              ]
+            },
+            "MaxPower": {
+              "description": "0mA"
+            },
+            "interface_descriptors": [
+              {
+                "bLength": {
+                  "value": "9"
+                },
+                ...
+                "bInterfaceProtocol": {
+                  "value": "0",
+                  "description": "Full speed (or root) hub"
+                },
+                "iInterface": {
+                  "value": "0"
+                },
+                "endpoint_descriptors": [
+                  {
+                    "bLength": {
+                      "value": "7"
+                    },
+                    ...
+                    "bmAttributes": {
+                      "value": "3",
+                      "attributes": [
+                        "Transfer Type  Interrupt",
+                        "Synch Type  None",
+                        "Usage Type  Data"
+                      ]
+                    },
+                    "wMaxPacketSize": {
+                      "value": "0x0002",
+                      "description": "1x 2 bytes"
+                    },
+                    "bInterval": {
+                      "value": "255"
+                    }
+                  }
+                ]
+              }
+            ]
+          }
+        },
+        "hub_descriptor": {
+          "bLength": {
+            "value": "9"
+          },
+          ...
+          "wHubCharacteristic": {
+            "value": "0x000a",
+            "attributes": [
+              "No power switching (usb 1.0)",
+              "Per-port overcurrent protection"
+            ]
+          },
+          ...
+          "hub_port_status": {
+            "Port 1": {
+              "value": "0000.0103",
+              "attributes": [
+                "power",
+                "enable",
+                "connect"
+              ]
+            },
+            "Port 2": {
+              "value": "0000.0103",
+              "attributes": [
+                "power",
+                "enable",
+                "connect"
+              ]
+            }
+          }
+        },
+        "device_status": {
+          "value": "0x0001",
+          "description": "Self Powered"
+        }
+      }
+    ]
+
+
+## info
+```python
+info()
+```
+Provides parser metadata (version, author, etc.)
+
+## parse
+```python
+parse(data, raw=False, quiet=False)
+```
+
+Main text parsing function
+
+Parameters:
+
+    data:        (string)  text data to parse
+    raw:         (boolean) output preprocessed JSON if True
+    quiet:       (boolean) suppress warning messages if True
+
+Returns:
+
+    List of Dictionaries. Raw or processed structured data.
+
+## Parser Information
+Compatibility:  linux
+
+Version 1.0 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/netstat.md

@@ -379,4 +379,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux, darwin, freebsd
 
-Version 1.10 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.11 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/ping.md

@@ -181,4 +181,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux, darwin, freebsd
 
-Version 1.5 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.6 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/ping_s.md

@@ -0,0 +1,101 @@
+[Home](https://kellyjonbrazil.github.io/jc/)
+
+# jc.parsers.ping_s
+jc - JSON CLI output utility `ping` command output streaming parser
+
+> This streaming parser outputs JSON Lines
+
+Supports `ping` and `ping6` output.
+
+Usage (cli):
+
+    $ ping | jc --ping-s
+
+> Note: When piping `jc` converted `ping` output to other processes it may appear the output is hanging due to the OS pipe buffers. This is because `ping` output is too small to quickly fill up the buffer. Use the `-u` option to unbuffer the `jc` output if you would like immediate output. See the [readme](https://github.com/kellyjonbrazil/jc/tree/master#unbuffering-output) for more information.
+
+Usage (module):
+
+    import jc.parsers.ping_s
+    result = jc.parsers.ping_s.parse(ping_command_output.splitlines())    # result is an iterable object
+    for item in result:
+        # do something
+
+Schema:
+
+    {
+      "type":                        string,        # 'reply', 'timeout', 'summary', etc. See `_error_type.type_map` for all options.
+      "source_ip":                   string,
+      "destination_ip":              string,
+      "sent_bytes":                  integer,
+      "pattern":                     string,        # (null if not set)
+      "destination":                 string,
+      "timestamp":                   float,
+      "response_bytes":              integer,
+      "response_ip":                 string,
+      "icmp_seq":                    integer,
+      "ttl":                         integer,
+      "time_ms":                     float,
+      "duplicate":                   boolean,
+      "packets_transmitted":         integer,
+      "packets_received":            integer,
+      "packet_loss_percent":         float,
+      "duplicates":                  integer,
+      "round_trip_ms_min":           float,
+      "round_trip_ms_avg":           float,
+      "round_trip_ms_max":           float,
+      "round_trip_ms_stddev":        float,
+      "_jc_meta":                                  # This object only exists if using -qq or ignore_exceptions=True
+        {
+          "success":                 booean,       # true if successfully parsed, false if error
+          "error":                   string,       # exists if "success" is false
+          "line":                    string        # exists if "success" is false
+        }
+    }
+
+Examples:
+
+    $ ping 1.1.1.1 | jc --ping-s
+    {"type":"reply","destination_ip":"1.1.1.1","sent_bytes":56,"pattern":null,"response_bytes":64,"response_ip":"1.1.1.1","icmp_seq":0,"ttl":56,"time_ms":23.703}
+    {"type":"reply","destination_ip":"1.1.1.1","sent_bytes":56,"pattern":null,"response_bytes":64,"response_ip":"1.1.1.1","icmp_seq":1,"ttl":56,"time_ms":22.862}
+    {"type":"reply","destination_ip":"1.1.1.1","sent_bytes":56,"pattern":null,"response_bytes":64,"response_ip":"1.1.1.1","icmp_seq":2,"ttl":56,"time_ms":22.82}
+    ...
+
+    $ ping 1.1.1.1 | jc --ping-s -r
+    {"type":"reply","destination_ip":"1.1.1.1","sent_bytes":"56","pattern":null,"response_bytes":"64","response_ip":"1.1.1.1","icmp_seq":"0","ttl":"56","time_ms":"23.054"}
+    {"type":"reply","destination_ip":"1.1.1.1","sent_bytes":"56","pattern":null,"response_bytes":"64","response_ip":"1.1.1.1","icmp_seq":"1","ttl":"56","time_ms":"24.739"}
+    {"type":"reply","destination_ip":"1.1.1.1","sent_bytes":"56","pattern":null,"response_bytes":"64","response_ip":"1.1.1.1","icmp_seq":"2","ttl":"56","time_ms":"23.232"}
+    ...
+
+
+## info
+```python
+info()
+```
+Provides parser metadata (version, author, etc.)
+
+## parse
+```python
+parse(data, raw=False, quiet=False, ignore_exceptions=False)
+```
+
+Main text parsing generator function. Returns an iterator object.
+
+Parameters:
+
+    data:              (iterable)  line-based text data to parse (e.g. sys.stdin or str.splitlines())
+    raw:               (boolean)   output preprocessed JSON if True
+    quiet:             (boolean)   suppress warning messages if True
+    ignore_exceptions: (boolean)   ignore parsing exceptions if True
+
+Yields:
+
+    Dictionary. Raw or processed structured data.
+
+Returns:
+
+    Iterator object
+
+## Parser Information
+Compatibility:  linux, darwin, freebsd
+
+Version 0.5 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/route.md

@@ -25,7 +25,7 @@ Schema:
         "genmask":         string,
         "flags":           string,
         "flags_pretty": [
-                           string,
+                           string
         ]
         "metric":          integer,
         "ref":             integer,
@@ -76,7 +76,6 @@ Examples:
       }
     ]
 
-
     $ route -ee | jc --route -p -r
     [
       {
@@ -108,7 +107,6 @@ Examples:
     ]
 
 
-
 ## info
 ```python
 info()

docs/parsers/sfdisk.md

@@ -0,0 +1,228 @@
+[Home](https://kellyjonbrazil.github.io/jc/)
+
+# jc.parsers.sfdisk
+jc - JSON CLI output utility `sfdisk` command output parser
+
+Supports the following `sfdisk` options:
+- `-l`
+- `-F`
+- `-d`   (deprecated - only for older versions of util-linux)
+- `-uM`  (deprecated - only for older versions of util-linux)
+- `-uC`  (deprecated - only for older versions of util-linux)
+- `-uS`  (deprecated - only for older versions of util-linux)
+- `-uB`  (deprecated - only for older versions of util-linux)
+
+Usage (cli):
+
+    # sfdisk -l | jc --sfdisk
+
+    or
+
+    # jc sfdisk -l
+
+Usage (module):
+
+    import jc.parsers.sfdisk
+    result = jc.parsers.sfdisk.parse(sfdisk_command_output)
+
+Schema:
+
+    [
+      {
+        "disk":                     string,
+        "disk_size":                string,
+        "free_disk_size":           string,
+        "bytes":                    integer,
+        "free_bytes":               integer,
+        "sectors":                  integer,
+        "free_sectors":             integer,
+        "cylinders":                integer,
+        "heads":                    integer,
+        "sectors_per_track":        integer,
+        "units":                    string,
+        "logical_sector_size":      integer,
+        "physical_sector_size":     integer,
+        "min_io_size":              integer,
+        "optimal_io_size":          integer,
+        "disk_label_type":          string,
+        "disk_identifier":          string,
+        "disk_model":               string,
+        "partitions": [
+          {
+            "device":               string,
+            "boot":                 boolean,
+            "start":                integer,
+            "end":                  integer,
+            "size":                 string,    # Note: will be integer when using deprecated -d sfdisk option
+            "cyls":                 integer,
+            "mib":                  integer,
+            "blocks":               integer,
+            "sectors":              integer,
+            "id":                   string,
+            "system":               string,
+            "type":                 string
+          }
+        ]
+      }
+    ]
+
+Examples:
+
+    # sfdisk -l | jc --sfdisk -p
+    [
+      {
+        "disk": "/dev/sda",
+        "cylinders": 2610,
+        "heads": 255,
+        "sectors_per_track": 63,
+        "units": "cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0",
+        "partitions": [
+          {
+            "device": "/dev/sda1",
+            "boot": true,
+            "start": 0,
+            "end": 130,
+            "cyls": 131,
+            "blocks": 1048576,
+            "id": "83",
+            "system": "Linux"
+          },
+          {
+            "device": "/dev/sda2",
+            "boot": false,
+            "start": 130,
+            "end": 2610,
+            "cyls": 2481,
+            "blocks": 19921920,
+            "id": "8e",
+            "system": "Linux LVM"
+          },
+          {
+            "device": "/dev/sda3",
+            "boot": false,
+            "start": 0,
+            "end": null,
+            "cyls": 0,
+            "blocks": 0,
+            "id": "0",
+            "system": "Empty"
+          },
+          {
+            "device": "/dev/sda4",
+            "boot": false,
+            "start": 0,
+            "end": null,
+            "cyls": 0,
+            "blocks": 0,
+            "id": "0",
+            "system": "Empty"
+          }
+        ]
+      },
+      {
+        "disk": "/dev/mapper/centos-root",
+        "cylinders": 2218,
+        "heads": 255,
+        "sectors_per_track": 63
+      },
+      {
+        "disk": "/dev/mapper/centos-swap",
+        "cylinders": 261,
+        "heads": 255,
+        "sectors_per_track": 63
+      }
+    ]
+
+    # sfdisk -l | jc --sfdisk -p -r
+    [
+      {
+        "disk": "/dev/sda",
+        "cylinders": "2610",
+        "heads": "255",
+        "sectors_per_track": "63",
+        "units": "cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0",
+        "partitions": [
+          {
+            "device": "/dev/sda1",
+            "boot": "*",
+            "start": "0+",
+            "end": "130-",
+            "cyls": "131-",
+            "blocks": "1048576",
+            "id": "83",
+            "system": "Linux"
+          },
+          {
+            "device": "/dev/sda2",
+            "boot": null,
+            "start": "130+",
+            "end": "2610-",
+            "cyls": "2481-",
+            "blocks": "19921920",
+            "id": "8e",
+            "system": "Linux LVM"
+          },
+          {
+            "device": "/dev/sda3",
+            "boot": null,
+            "start": "0",
+            "end": "-",
+            "cyls": "0",
+            "blocks": "0",
+            "id": "0",
+            "system": "Empty"
+          },
+          {
+            "device": "/dev/sda4",
+            "boot": null,
+            "start": "0",
+            "end": "-",
+            "cyls": "0",
+            "blocks": "0",
+            "id": "0",
+            "system": "Empty"
+          }
+        ]
+      },
+      {
+        "disk": "/dev/mapper/centos-root",
+        "cylinders": "2218",
+        "heads": "255",
+        "sectors_per_track": "63"
+      },
+      {
+        "disk": "/dev/mapper/centos-swap",
+        "cylinders": "261",
+        "heads": "255",
+        "sectors_per_track": "63"
+      }
+    ]
+
+
+## info
+```python
+info()
+```
+Provides parser metadata (version, author, etc.)
+
+## parse
+```python
+parse(data, raw=False, quiet=False)
+```
+
+Main text parsing function
+
+Parameters:
+
+    data:        (string)  text data to parse
+    raw:         (boolean) output preprocessed JSON if True
+    quiet:       (boolean) suppress warning messages if True
+
+Returns:
+
+    List of Dictionaries. Raw or processed structured data.
+
+## Parser Information
+Compatibility:  linux
+
+Version 1.1 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/stat.md

@@ -193,4 +193,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux, darwin, freebsd
 
-Version 1.8 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.9 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/uname.md

@@ -72,4 +72,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux, darwin, freebsd
 
-Version 1.5 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.6 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/vmstat.md

@@ -0,0 +1,149 @@
+[Home](https://kellyjonbrazil.github.io/jc/)
+
+# jc.parsers.vmstat
+jc - JSON CLI output utility `vmstat` command output parser
+
+Options supported: `-a`, `-w`, `-d`, `-t`
+
+The `epoch` calculated timestamp field is naive (i.e. based on the local time of the system the parser is run on)
+
+The `epoch_utc` calculated timestamp field is timezone-aware and is only available if the timezone field is UTC.
+
+Usage (cli):
+
+    $ vmstat | jc --vmstat
+
+    or
+
+    $ jc vmstat
+
+Usage (module):
+
+    import jc.parsers.vmstat
+    result = jc.parsers.vmstat.parse(vmstat_command_output)
+
+Schema:
+
+    [
+      {
+        "runnable_procs":                    integer,
+        "uninterruptible_sleeping_procs":    integer,
+        "virtual_mem_used":                  integer,
+        "free_mem":                          integer,
+        "buffer_mem":                        integer,
+        "cache_mem":                         integer,
+        "inactive_mem":                      integer,
+        "active_mem":                        integer,
+        "swap_in":                           integer,
+        "swap_out":                          integer,
+        "blocks_in":                         integer,
+        "blocks_out":                        integer,
+        "interrupts":                        integer,
+        "context_switches":                  integer,
+        "user_time":                         integer,
+        "system_time":                       integer,
+        "idle_time":                         integer,
+        "io_wait_time":                      integer,
+        "stolen_time":                       integer,
+        "disk":                              string,
+        "total_reads":                       integer,
+        "merged_reads":                      integer,
+        "sectors_read":                      integer,
+        "reading_ms":                        integer,
+        "total_writes":                      integer,
+        "merged_writes":                     integer,
+        "sectors_written":                   integer,
+        "writing_ms":                        integer,
+        "current_io":                        integer,
+        "io_seconds":                        integer,
+        "timestamp":                         string,
+        "timezone":                          string,
+        "epoch":                             integer,     # naive timestamp if -t flag is used
+        "epoch_utc":                         integer      # aware timestamp if -t flag is used and UTC TZ
+      }
+    ]
+
+Examples:
+
+    $ vmstat | jc --vmstat -p
+    [
+      {
+        "runnable_procs": 2,
+        "uninterruptible_sleeping_procs": 0,
+        "virtual_mem_used": 0,
+        "free_mem": 2794468,
+        "buffer_mem": 2108,
+        "cache_mem": 741208,
+        "inactive_mem": null,
+        "active_mem": null,
+        "swap_in": 0,
+        "swap_out": 0,
+        "blocks_in": 1,
+        "blocks_out": 3,
+        "interrupts": 29,
+        "context_switches": 57,
+        "user_time": 0,
+        "system_time": 0,
+        "idle_time": 99,
+        "io_wait_time": 0,
+        "stolen_time": 0,
+        "timestamp": null,
+        "timezone": null
+      }
+    ]
+
+    $ vmstat | jc --vmstat -p -r
+    [
+      {
+        "runnable_procs": "2",
+        "uninterruptible_sleeping_procs": "0",
+        "virtual_mem_used": "0",
+        "free_mem": "2794468",
+        "buffer_mem": "2108",
+        "cache_mem": "741208",
+        "inactive_mem": null,
+        "active_mem": null,
+        "swap_in": "0",
+        "swap_out": "0",
+        "blocks_in": "1",
+        "blocks_out": "3",
+        "interrupts": "29",
+        "context_switches": "57",
+        "user_time": "0",
+        "system_time": "0",
+        "idle_time": "99",
+        "io_wait_time": "0",
+        "stolen_time": "0",
+        "timestamp": null,
+        "timezone": null
+      }
+    ]
+
+
+## info
+```python
+info()
+```
+Provides parser metadata (version, author, etc.)
+
+## parse
+```python
+parse(data, raw=False, quiet=False)
+```
+
+Main text parsing function
+
+Parameters:
+
+    data:        (string)  text data to parse
+    raw:         (boolean) output preprocessed JSON if True
+    quiet:       (boolean) suppress warning messages if True
+
+Returns:
+
+    List of Dictionaries. Raw or processed structured data.
+
+## Parser Information
+Compatibility:  linux
+
+Version 1.0 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/vmstat_s.md

@@ -0,0 +1,114 @@
+[Home](https://kellyjonbrazil.github.io/jc/)
+
+# jc.parsers.vmstat_s
+jc - JSON CLI output utility `vmstat` command output streaming parser
+
+> This streaming parser outputs JSON Lines
+
+Options supported: `-a`, `-w`, `-d`, `-t`
+
+The `epoch` calculated timestamp field is naive (i.e. based on the local time of the system the parser is run on)
+
+The `epoch_utc` calculated timestamp field is timezone-aware and is only available if the timezone field is UTC.
+
+Usage (cli):
+
+    $ vmstat | jc --vmstat-s
+
+> Note: When piping `jc` converted `vmstat` output to other processes it may appear the output is hanging due to the OS pipe buffers. This is because `vmstat` output is too small to quickly fill up the buffer. Use the `-u` option to unbuffer the `jc` output if you would like immediate output. See the [readme](https://github.com/kellyjonbrazil/jc/tree/master#unbuffering-output) for more information.
+
+Usage (module):
+
+    import jc.parsers.vmstat_s
+    result = jc.parsers.vmstat_s.parse(vmstat_command_output.splitlines())    # result is an iterable object
+    for item in result:
+        # do something
+
+Schema:
+
+    {
+      "runnable_procs":                    integer,
+      "uninterruptible_sleeping_procs":    integer,
+      "virtual_mem_used":                  integer,
+      "free_mem":                          integer,
+      "buffer_mem":                        integer,
+      "cache_mem":                         integer,
+      "inactive_mem":                      integer,
+      "active_mem":                        integer,
+      "swap_in":                           integer,
+      "swap_out":                          integer,
+      "blocks_in":                         integer,
+      "blocks_out":                        integer,
+      "interrupts":                        integer,
+      "context_switches":                  integer,
+      "user_time":                         integer,
+      "system_time":                       integer,
+      "idle_time":                         integer,
+      "io_wait_time":                      integer,
+      "stolen_time":                       integer,
+      "disk":                              string,
+      "total_reads":                       integer,
+      "merged_reads":                      integer,
+      "sectors_read":                      integer,
+      "reading_ms":                        integer,
+      "total_writes":                      integer,
+      "merged_writes":                     integer,
+      "sectors_written":                   integer,
+      "writing_ms":                        integer,
+      "current_io":                        integer,
+      "io_seconds":                        integer,
+      "timestamp":                         string,
+      "timezone":                          string,
+      "epoch":                             integer,     # naive timestamp if -t flag is used
+      "epoch_utc":                         integer      # aware timestamp if -t flag is used and UTC TZ
+      "_jc_meta":                                       # This object only exists if using -qq or ignore_exceptions=True
+        {
+          "success":                       booean,      # true if successfully parsed, false if error
+          "error":                         string,      # exists if "success" is false
+          "line":                          string       # exists if "success" is false
+        }
+    }
+
+Examples:
+
+    $ vmstat | jc --vmstat-s
+    {"runnable_procs":2,"uninterruptible_sleeping_procs":0,"virtual_mem_used":0,"free_mem":2794468,"buffer_mem":2108,"cache_mem":741208,"inactive_mem":null,"active_mem":null,"swap_in":0,"swap_out":0,"blocks_in":1,"blocks_out":3,"interrupts":29,"context_switches":57,"user_time":0,"system_time":0,"idle_time":99,"io_wait_time":0,"stolen_time":0,"timestamp":null,"timezone":null}
+    ...
+
+    $ vmstat | jc --vmstat-s -r
+    {"runnable_procs":"2","uninterruptible_sleeping_procs":"0","virtual_mem_used":"0","free_mem":"2794468","buffer_mem":"2108","cache_mem":"741208","inactive_mem":null,"active_mem":null,"swap_in":"0","swap_out":"0","blocks_in":"1","blocks_out":"3","interrupts":"29","context_switches":"57","user_time":"0","system_time":"0","idle_time":"99","io_wait_time":"0","stolen_time":"0","timestamp":null,"timezone":null}
+    ...
+
+
+## info
+```python
+info()
+```
+Provides parser metadata (version, author, etc.)
+
+## parse
+```python
+parse(data, raw=False, quiet=False, ignore_exceptions=False)
+```
+
+Main text parsing generator function. Returns an iterator object.
+
+Parameters:
+
+    data:              (iterable)  line-based text data to parse (e.g. sys.stdin or str.splitlines())
+    raw:               (boolean)   output preprocessed JSON if True
+    quiet:             (boolean)   suppress warning messages if True
+    ignore_exceptions: (boolean)   ignore parsing exceptions if True
+
+Yields:
+
+    Dictionary. Raw or processed structured data.
+
+Returns:
+
+    Iterator object
+
+## Parser Information
+Compatibility:  linux
+
+Version 0.5 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/xml.md

@@ -95,4 +95,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux, darwin, cygwin, win32, aix, freebsd
 
-Version 1.4 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.5 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/yaml.md

@@ -109,4 +109,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux, darwin, cygwin, win32, aix, freebsd
 
-Version 1.4 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.5 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/readme.md

@@ -59,26 +59,13 @@ Module Example:
 
     >>> import jc.parsers.dig
     >>>
-    >>> data = '''; <<>> DiG 9.10.6 <<>> example.com
-    ... ;; global options: +cmd
-    ... ;; Got answer:
-    ... ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64612
-    ... ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
-    ...
-    ... ;; OPT PSEUDOSECTION:
-    ... ; EDNS: version: 0, flags:; udp: 4096
-    ... ;; QUESTION SECTION:
-    ... ;example.com.           IN  A
-    ...
-    ... ;; ANSWER SECTION:
-    ... example.com.        29658   IN  A   93.184.216.34
-    ...
-    ... ;; Query time: 52 msec
-    ... ;; SERVER: 2600:1700:bab0:d40::1#53(2600:1700:bab0:d40::1)
-    ... ;; WHEN: Fri Apr 16 16:13:00 PDT 2021
-    ... ;; MSG SIZE  rcvd: 56'''
+    >>> import subprocess
+    >>> import jc.parsers.dig
     >>>
-    >>> jc.parsers.dig.parse(data)
+    >>> cmd_output = subprocess.check_output(['dig', 'example.com'], text=True)
+    >>> data = jc.parsers.dig.parse(cmd_output)
+    >>>
+    >>> data
     [{'id': 64612, 'opcode': 'QUERY', 'status': 'NOERROR', 'flags': ['qr', 'rd', 'ra'], 'query_num': 1, 'answer_num':
     1, 'authority_num': 0, 'additional_num': 1, 'opt_pseudosection': {'edns': {'version': 0, 'flags': [], 'udp':
     4096}}, 'question': {'name': 'example.com.', 'class': 'IN', 'type': 'A'}, 'answer': [{'name': 'example.com.',

docs/utils.md

@@ -4,14 +4,16 @@ jc - JSON CLI output utility utils
 
 ## warning_message
 ```python
-warning_message(message)
+warning_message(message_lines)
 ```
 
-Prints a warning message for non-fatal issues
+Prints warning message for non-fatal issues. The first line is prepended with
+'jc:  Warning - ' and subsequent lines are indented. Wraps text as needed based
+on the terminal width.
 
 Parameters:
 
-    message:        (string) text of message
+    message:   (list) list of string lines
 
 Returns:
 
@@ -20,14 +22,16 @@ Returns:
 
 ## error_message
 ```python
-error_message(message)
+error_message(message_lines)
 ```
 
-Prints an error message for fatal issues
+Prints an error message for fatal issues. The first line is prepended with
+'jc:  Error - ' and subsequent lines are indented. Wraps text as needed based
+on the terminal width.
 
 Parameters:
 
-    message:        (string) text of message
+    message:   (list) list of string lines
 
 Returns:
 
@@ -117,6 +121,20 @@ Returns:
     True/False      False unless a 'truthy' number or string is found ('y', 'yes', 'true', '1', 1, -1, etc.)
 
 
+## stream_success
+```python
+stream_success(output_line, ignore_exceptions)
+```
+Add `_jc_meta` object to output line if `ignore_exceptions=True`
+
+## stream_error
+```python
+stream_error(e, ignore_exceptions, line)
+```
+Reraise the stream exception with annotation or print an error `_jc_meta`
+field if `ignore_exceptions=True`
+
+
 ## timestamp
 ```python
 timestamp(datetime_string)
@@ -131,7 +149,7 @@ Parameters:
 Attributes:
 
     string              (str)   the input datetime string
-    format              (int)   the format rule that was used to decode the datetime string
+    format              (int)   the format rule that was used to decode the datetime string. None if conversion fails
     naive               (int)   timestamp based on locally configured timezone. None if conversion fails
     utc                 (int)   aware timestamp only if UTC timezone detected in datetime string. None if conversion fails
 

jc/__init__.py

@@ -57,26 +57,13 @@ Module Example:
 
     >>> import jc.parsers.dig
     >>>
-    >>> data = '''; <<>> DiG 9.10.6 <<>> example.com
-    ... ;; global options: +cmd
-    ... ;; Got answer:
-    ... ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64612
-    ... ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
-    ...
-    ... ;; OPT PSEUDOSECTION:
-    ... ; EDNS: version: 0, flags:; udp: 4096
-    ... ;; QUESTION SECTION:
-    ... ;example.com.           IN  A
-    ...
-    ... ;; ANSWER SECTION:
-    ... example.com.        29658   IN  A   93.184.216.34
-    ...
-    ... ;; Query time: 52 msec
-    ... ;; SERVER: 2600:1700:bab0:d40::1#53(2600:1700:bab0:d40::1)
-    ... ;; WHEN: Fri Apr 16 16:13:00 PDT 2021
-    ... ;; MSG SIZE  rcvd: 56'''
+    >>> import subprocess
+    >>> import jc.parsers.dig
     >>>
-    >>> jc.parsers.dig.parse(data)
+    >>> cmd_output = subprocess.check_output(['dig', 'example.com'], text=True)
+    >>> data = jc.parsers.dig.parse(cmd_output)
+    >>>
+    >>> data
     [{'id': 64612, 'opcode': 'QUERY', 'status': 'NOERROR', 'flags': ['qr', 'rd', 'ra'], 'query_num': 1, 'answer_num':
     1, 'authority_num': 0, 'additional_num': 1, 'opt_pseudosection': {'edns': {'version': 0, 'flags': [], 'udp':
     4096}}, 'question': {'name': 'example.com.', 'class': 'IN', 'type': 'A'}, 'answer': [{'name': 'example.com.',
@@ -86,4 +73,4 @@ Module Example:
 """
 
 name = 'jc'
-__version__ = '1.15.4'
+__version__ = '1.17.2'

jc/cli.py

@@ -6,16 +6,17 @@ import sys
 import os
 import os.path
 import re
-import shlex
 import importlib
 import textwrap
 import signal
+import shlex
 import subprocess
 import json
 import jc
-import jc.appdirs as appdirs
+from jc import appdirs
 import jc.utils
 import jc.tracebackplus
+from jc.exceptions import LibraryNotInstalled, ParseError
 
 # make pygments import optional
 try:
@@ -25,9 +26,9 @@ try:
     from pygments.token import (Name, Number, String, Keyword)
     from pygments.lexers import JsonLexer
     from pygments.formatters import Terminal256Formatter
-    pygments_installed = True
+    PYGMENTS_INSTALLED = True
 except Exception:
-    pygments_installed = False
+    PYGMENTS_INSTALLED = False
 
 
 class info():
@@ -52,6 +53,7 @@ parsers = [
     'crontab',
     'crontab-u',
     'csv',
+    'csv-s',
     'date',
     'df',
     'dig',
@@ -80,19 +82,23 @@ parsers = [
     'kv',
     'last',
     'ls',
+    'ls-s',
     'lsblk',
     'lsmod',
     'lsof',
+    'lsusb',
     'mount',
     'netstat',
     'ntpq',
     'passwd',
     'ping',
+    'ping-s',
     'pip-list',
     'pip-show',
     'ps',
     'route',
     'rpm-qi',
+    'sfdisk',
     'shadow',
     'ss',
     'stat',
@@ -111,6 +117,8 @@ parsers = [
     'uname',
     'upower',
     'uptime',
+    'vmstat',
+    'vmstat-s',
     'w',
     'wc',
     'who',
@@ -129,7 +137,7 @@ local_parsers_dir = os.path.join(data_dir, 'jcparsers')
 if os.path.isdir(local_parsers_dir):
     sys.path.append(data_dir)
     for name in os.listdir(local_parsers_dir):
-        if re.match(r'\w+\.py', name) and os.path.isfile(os.path.join(local_parsers_dir, name)):
+        if re.match(r'\w+\.py$', name) and os.path.isfile(os.path.join(local_parsers_dir, name)):
             plugin_name = name[0:-3]
             local_parsers.append(plugin_name)
             if plugin_name not in parsers:
@@ -138,7 +146,7 @@ if os.path.isdir(local_parsers_dir):
 
 # We only support 2.3.0+, pygments changed color names in 2.4.0.
 # startswith is sufficient and avoids potential exceptions from split and int.
-if pygments_installed:
+if PYGMENTS_INSTALLED:
     if pygments.__version__.startswith('2.3.'):
         PYGMENT_COLOR = {
             'black': '#ansiblack',
@@ -213,7 +221,7 @@ def set_env_colors(env_colors=None):
 
     # if there is an issue with the env variable, just set all colors to default and move on
     if input_error:
-        jc.utils.warning_message('Could not parse JC_COLORS environment variable')
+        jc.utils.warning_message(['Could not parse JC_COLORS environment variable'])
         color_list = ['default', 'default', 'default', 'default']
 
     # Try the color set in the JC_COLORS env variable first. If it is set to default, then fall back to default colors
@@ -227,7 +235,7 @@ def set_env_colors(env_colors=None):
 
 def piped_output():
     """Return False if stdout is a TTY. True if output is being piped to another program"""
-    return False if sys.stdout.isatty() else True
+    return not sys.stdout.isatty()
 
 
 def ctrlc(signum, frame):
@@ -235,9 +243,9 @@ def ctrlc(signum, frame):
     sys.exit(JC_ERROR_EXIT)
 
 
-def parser_shortname(parser_argument):
+def parser_shortname(parser_arg):
     """Return short name of the parser with dashes and no -- prefix"""
-    return parser_argument[2:]
+    return parser_arg[2:]
 
 
 def parser_argument(parser):
@@ -324,14 +332,15 @@ def helptext():
     Parsers:
 {parsers_string}
     Options:
-            -a               about jc
-            -d               debug (-dd for verbose debug)
-            -h               help (-h --parser_name for parser documentation)
-            -m               monochrome output
-            -p               pretty print output
-            -q               quiet - suppress parser warnings
-            -r               raw JSON output
-            -v               version info
+            -a    about jc
+            -d    debug (-dd for verbose debug)
+            -h    help (-h --parser_name for parser documentation)
+            -m    monochrome output
+            -p    pretty print output
+            -q    quiet - suppress parser warnings (-qq to ignore streaming errors)
+            -r    raw JSON output
+            -u    unbuffer output
+            -v    version info
 
     Examples:
             Standard Syntax:
@@ -358,11 +367,10 @@ def help_doc(options):
             # load parser module just in time so we don't need to load all modules
             parser = parser_module(arg)
             compatible = ', '.join(parser.info.compatible)
-            doc_text = f'''{parser.__doc__}
-Compatibility:  {compatible}
-
-Version {parser.info.version} by {parser.info.author} ({parser.info.author_email})
-'''
+            doc_text = \
+                f'{parser.__doc__}\n'\
+                f'Compatibility:  {compatible}\n\n'\
+                f'Version {parser.info.version} by {parser.info.author} ({parser.info.author_email})\n'
 
             return doc_text
 
@@ -395,8 +403,7 @@ def json_out(data, pretty=False, env_colors=None, mono=False, piped_out=False):
         return str(highlight(json.dumps(data, indent=indent, separators=separators, ensure_ascii=False),
                              JsonLexer(), Terminal256Formatter(style=JcStyle))[0:-1])
 
-    else:
-        return json.dumps(data, indent=indent, separators=separators, ensure_ascii=False)
+    return json.dumps(data, indent=indent, separators=separators, ensure_ascii=False)
 
 
 def magic_parser(args):
@@ -413,8 +420,7 @@ def magic_parser(args):
     if len(args) <= 1 or args[1].startswith('--'):
         return False, None, None, []
 
-    # correctly parse escape characters and spaces with shlex
-    args_given = ' '.join(map(shlex.quote, args[1:])).split()
+    args_given = args[1:]
     options = []
 
     # find the options
@@ -424,7 +430,7 @@ def magic_parser(args):
             return False, None, None, []
 
         # option found - populate option list
-        elif arg.startswith('-'):
+        if arg.startswith('-'):
             options.extend(args_given.pop(0)[1:])
 
         # command found if iterator didn't already stop - stop iterating
@@ -455,7 +461,7 @@ def magic_parser(args):
     found_parser = magic_dict.get(two_word_command, magic_dict.get(one_word_command))
 
     return (
-        True if found_parser else False,    # was a suitable parser found?
+        bool(found_parser),                 # was a suitable parser found?
         args_given,                         # run_command
         found_parser,                       # the parser selected
         options                             # jc options to preserve
@@ -464,7 +470,11 @@ def magic_parser(args):
 
 def run_user_command(command):
     """Use subprocess to run the user's command. Returns the STDOUT, STDERR, and the Exit Code as a tuple."""
-    proc = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
+    proc = subprocess.Popen(command,
+                            stdout=subprocess.PIPE,
+                            stderr=subprocess.PIPE,
+                            close_fds=False,            # Allows inheriting file descriptors. Useful for process substitution
+                            universal_newlines=True)
     stdout, stderr = proc.communicate()
 
     return (
@@ -476,8 +486,7 @@ def run_user_command(command):
 
 def combined_exit_code(program_exit=0, jc_exit=0):
     exit_code = program_exit + jc_exit
-    if exit_code > 255:
-        exit_code = 255
+    exit_code = min(exit_code, 255)
     return exit_code
 
 
@@ -514,18 +523,20 @@ def main():
 
     about = 'a' in options
     debug = 'd' in options
-    verbose_debug = True if options.count('d') > 1 else False
+    verbose_debug = options.count('d') > 1
     mono = 'm' in options
     help_me = 'h' in options
     pretty = 'p' in options
     quiet = 'q' in options
+    ignore_exceptions = options.count('q') > 1
     raw = 'r' in options
+    unbuffer = 'u' in options
     version_info = 'v' in options
 
     if verbose_debug:
         jc.tracebackplus.enable(context=11)
 
-    if not pygments_installed:
+    if not PYGMENTS_INSTALLED:
         mono = True
 
     if about:
@@ -543,7 +554,10 @@ def main():
     # if magic syntax used, try to run the command and error if it's not found, etc.
     magic_stdout, magic_stderr, magic_exit_code = None, None, 0
     if run_command:
-        run_command_str = ' '.join(run_command)
+        try:
+            run_command_str = shlex.join(run_command)      # python 3.8+
+        except AttributeError:
+            run_command_str = ' '.join(run_command)        # older python versions
 
     if valid_command:
         try:
@@ -554,19 +568,26 @@ def main():
         except FileNotFoundError:
             if debug:
                 raise
-            else:
-                jc.utils.error_message(f'"{run_command_str}" command could not be found. For details use the -d or -dd option.')
-                sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
+
+            jc.utils.error_message([f'"{run_command_str}" command could not be found. For details use the -d or -dd option.'])
+            sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
+
+        except OSError:
+            if debug:
+                raise
+
+            jc.utils.error_message([f'"{run_command_str}" command could not be run due to too many open files. For details use the -d or -dd option.'])
+            sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
 
         except Exception:
             if debug:
                 raise
-            else:
-                jc.utils.error_message(f'"{run_command_str}" command could not be run. For details use the -d or -dd option.')
-                sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
+
+            jc.utils.error_message([f'"{run_command_str}" command could not be run. For details use the -d or -dd option.'])
+            sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
 
     elif run_command is not None:
-        jc.utils.error_message(f'"{run_command_str}" cannot be used with Magic syntax. Use "jc -h" for help.')
+        jc.utils.error_message([f'"{run_command_str}" cannot be used with Magic syntax. Use "jc -h" for help.'])
         sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
 
     # find the correct parser
@@ -585,46 +606,79 @@ def main():
                 break
 
         if not found:
-            jc.utils.error_message('Missing or incorrect arguments. Use "jc -h" for help.')
+            jc.utils.error_message(['Missing or incorrect arguments. Use "jc -h" for help.'])
             sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
 
     # check for input errors (pipe vs magic)
     if not sys.stdin.isatty() and magic_stdout:
-        jc.utils.error_message('Piped data and Magic syntax used simultaneously. Use "jc -h" for help.')
+        jc.utils.error_message(['Piped data and Magic syntax used simultaneously. Use "jc -h" for help.'])
         sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
 
     elif sys.stdin.isatty() and magic_stdout is None:
-        jc.utils.error_message('Missing piped data. Use "jc -h" for help.')
+        jc.utils.error_message(['Missing piped data. Use "jc -h" for help.'])
         sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
 
-    # parse the data
-    data = magic_stdout or sys.stdin.read()
-
+    # parse and print to stdout
     try:
-        result = parser.parse(data, raw=raw, quiet=quiet)
+        # differentiate between regular and streaming parsers
 
-    except Exception:
-        if debug:
-            raise
+        # streaming
+        if getattr(parser.info, 'streaming', None):
+            result = parser.parse(sys.stdin, raw=raw, quiet=quiet, ignore_exceptions=ignore_exceptions)
+            for line in result:
+                print(json_out(line,
+                               pretty=pretty,
+                               env_colors=jc_colors,
+                               mono=mono,
+                               piped_out=piped_output()),
+                      flush=unbuffer)
+
+            sys.exit(combined_exit_code(magic_exit_code, 0))
+
+        # regular
         else:
-            jc.utils.error_message(
-                f'{parser_name} parser could not parse the input data. Did you use the correct parser?\n'
-                '             For details use the -d or -dd option. Use "jc -h" for help.')
-            sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
+            data = magic_stdout or sys.stdin.read()
+            result = parser.parse(data, raw=raw, quiet=quiet)
+            print(json_out(result,
+                           pretty=pretty,
+                           env_colors=jc_colors,
+                           mono=mono,
+                           piped_out=piped_output()),
+                  flush=unbuffer)
 
-    # output the json
-    try:
-        print(json_out(result, pretty=pretty, env_colors=jc_colors, mono=mono, piped_out=piped_output()))
         sys.exit(combined_exit_code(magic_exit_code, 0))
 
+    except (ParseError, LibraryNotInstalled) as e:
+        if debug:
+            raise
+
+        jc.utils.error_message([f'Parser issue with {parser_name}:',
+                                f'{e.__class__.__name__}: {e}',
+                                'For details use the -d or -dd option. Use "jc -h" for help.'])
+        sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
+
+    except json.JSONDecodeError:
+        if debug:
+            raise
+
+        jc.utils.error_message(['There was an issue generating the JSON output.',
+                                'For details use the -d or -dd option.'])
+        sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
+
     except Exception:
         if debug:
             raise
-        else:
-            jc.utils.error_message(
-                'There was an issue generating the JSON output.\n'
-                '             For details use the -d or -dd option.')
-            sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
+
+        streaming_msg = ''
+        if getattr(parser.info, 'streaming', None):
+            streaming_msg = 'Use the -qq option to ignore streaming parser errors.'
+
+        jc.utils.error_message([
+            f'{parser_name} parser could not parse the input data. Did you use the correct parser?',
+            f'{streaming_msg}',
+            'For details use the -d or -dd option. Use "jc -h" for help.'
+        ])
+        sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
 
 
 if __name__ == '__main__':

jc/exceptions.py

@@ -3,3 +3,7 @@
 
 class ParseError(Exception):
     pass
+
+
+class LibraryNotInstalled(Exception):
+    pass

jc/parsers/csv_s.py

@@ -0,0 +1,131 @@
+"""jc - JSON CLI output utility `csv` file streaming parser
+
+> This streaming parser outputs JSON Lines
+
+The `csv` streaming parser will attempt to automatically detect the delimiter character. If the delimiter cannot be detected it will default to comma. The first row of the file must be a header row.
+
+Note: The first 100 rows are read into memory to enable delimiter detection, then the rest of the rows are loaded lazily.
+
+Usage (cli):
+
+    $ cat file.csv | jc --csv-s
+
+Usage (module):
+
+    import jc.parsers.csv_s
+    result = jc.parsers.csv_s.parse(csv_output)
+
+Schema:
+
+    csv file converted to a Dictionary: https://docs.python.org/3/library/csv.html
+
+    {
+      "column_name1":     string,
+      "column_name2":     string
+    }
+
+Examples:
+
+    $ cat homes.csv
+    "Sell", "List", "Living", "Rooms", "Beds", "Baths", "Age", "Acres", "Taxes"
+    142, 160, 28, 10, 5, 3,  60, 0.28,  3167
+    175, 180, 18,  8, 4, 1,  12, 0.43,  4033
+    129, 132, 13,  6, 3, 1,  41, 0.33,  1471
+    ...
+
+    $ cat homes.csv | jc --csv-s
+    {"Sell":"142","List":"160","Living":"28","Rooms":"10","Beds":"5","Baths":"3","Age":"60","Acres":"0.28","Taxes":"3167"}
+    {"Sell":"175","List":"180","Living":"18","Rooms":"8","Beds":"4","Baths":"1","Age":"12","Acres":"0.43","Taxes":"4033"}
+    {"Sell":"129","List":"132","Living":"13","Rooms":"6","Beds":"3","Baths":"1","Age":"41","Acres":"0.33","Taxes":"1471"}
+    ...
+"""
+import itertools
+import csv
+import jc.utils
+from jc.utils import stream_success, stream_error
+from jc.exceptions import ParseError
+
+
+class info():
+    """Provides parser metadata (version, author, etc.)"""
+    version = '1.0'
+    description = 'CSV file streaming parser'
+    author = 'Kelly Brazil'
+    author_email = 'kellyjonbrazil@gmail.com'
+    details = 'Using the python standard csv library'
+    compatible = ['linux', 'darwin', 'cygwin', 'win32', 'aix', 'freebsd']
+    streaming = True
+
+
+__version__ = info.version
+
+
+def _process(proc_data):
+    """
+    Final processing to conform to the schema.
+
+    Parameters:
+
+        proc_data:   (List of Dictionaries) raw structured data to process
+
+    Returns:
+
+        List of Dictionaries. Each Dictionary represents a row in the csv file.
+    """
+    # No further processing
+    return proc_data
+
+
+def parse(data, raw=False, quiet=False, ignore_exceptions=False):
+    """
+    Main text parsing generator function. Returns an iterator object.
+
+    Parameters:
+
+        data:              (iterable)  line-based text data to parse (e.g. sys.stdin or str.splitlines())
+        raw:               (boolean)   output preprocessed JSON if True
+        quiet:             (boolean)   suppress warning messages if True
+        ignore_exceptions: (boolean)   ignore parsing exceptions if True
+
+    Yields:
+
+        Dictionary. Raw or processed structured data.
+
+    Returns:
+
+        Iterator object
+    """
+    if not quiet:
+        jc.utils.compatibility(__name__, info.compatible)
+
+    # convert data to an iterable in case a sequence like a list is used as input.
+    # this allows the exhaustion of the input so we don't double-process later.
+    data = iter(data)
+    temp_list = []
+
+    # first, load the first 100 lines into a list to detect the CSV dialect
+    for line in itertools.islice(data, 100):
+        temp_list.append(line)
+
+    # check for Python bug that does not split on `\r` newlines from sys.stdin correctly
+    # https://bugs.python.org/issue45617
+    if len(temp_list) == 1:
+        raise ParseError('Unable to detect line endings. Please try the non-streaming CSV parser instead.')
+
+    sniffdata = '\n'.join(temp_list)
+
+    dialect = None
+    try:
+        dialect = csv.Sniffer().sniff(sniffdata)
+    except Exception:
+        pass
+
+    # chain `temp_list` and `data` together to lazy load the rest of the CSV data
+    new_data = itertools.chain(temp_list, data)
+    reader = csv.DictReader(new_data, dialect=dialect)
+
+    for row in reader:
+        try:
+            yield stream_success(row, ignore_exceptions) if raw else stream_success(_process(row), ignore_exceptions)
+        except Exception as e:
+            yield stream_error(e, ignore_exceptions, row)

jc/parsers/df.py

@@ -92,13 +92,14 @@ Examples:
       ...
     ]
 """
+import hashlib
 import jc.utils
 import jc.parsers.universal
 
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.7'
+    version = '1.8'
     description = '`df` command parser'
     author = 'Kelly Brazil'
     author_email = 'kellyjonbrazil@gmail.com'
@@ -165,6 +166,29 @@ def _process(proc_data):
     return proc_data
 
 
+def _long_filesystem_hash(header, line):
+    """Returns truncated hash and value of the filesystem field if it is too long for the column"""
+    filesystem_field = line.split()[0]
+
+    # get length of filesystem column
+    space_count = 0
+    for char in header[10:]:
+        if char == ' ':
+            space_count += 1
+            continue
+
+        break
+
+    filesystem_col_len = space_count + 9
+
+    # return the hash and value if the field data is longer than the column length
+    if len(filesystem_field) > filesystem_col_len:
+        truncated_hash = hashlib.sha256(filesystem_field.encode('utf-8')).hexdigest()[:filesystem_col_len]
+        return truncated_hash, filesystem_field
+
+    return None, None
+
+
 def parse(data, raw=False, quiet=False):
     """
     Main text parsing function
@@ -184,7 +208,9 @@ def parse(data, raw=False, quiet=False):
         jc.utils.compatibility(__name__, info.compatible)
 
     cleandata = data.splitlines()
+    fix_data = []
     raw_output = []
+    filesystem_map = {}
 
     if jc.utils.has_data(data):
 
@@ -193,8 +219,26 @@ def parse(data, raw=False, quiet=False):
         cleandata[0] = cleandata[0].replace('-', '_')
         cleandata[0] = cleandata[0].replace('mounted on', 'mounted_on')
 
+        # fix long filesystem data in some older versions of df
+        header = cleandata[0]
+        fix_data.append(header)
+        for line in cleandata[1:]:
+            field_hash, field_value = _long_filesystem_hash(header, line)
+            if field_hash:
+                filesystem_map.update({field_hash: field_value})
+                newline = line.replace(field_value, field_hash)
+                fix_data.append(newline)
+            else:
+                fix_data.append(line)
+
         # parse the data
-        raw_output = jc.parsers.universal.sparse_table_parse(cleandata)
+        raw_output = jc.parsers.universal.sparse_table_parse(fix_data)
+
+        # replace hash values with real values to fix long filesystem data in some older versions of df
+        for item in raw_output:
+            if 'filesystem' in item:
+                if item['filesystem'] in filesystem_map:
+                    item['filesystem'] = filesystem_map[item['filesystem']]
 
     if raw:
         return raw_output

jc/parsers/file.py

@@ -63,7 +63,7 @@ import jc.parsers.universal
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.3'
+    version = '1.4'
     description = '`file` command parser'
     author = 'Kelly Brazil'
     author_email = 'kellyjonbrazil@gmail.com'
@@ -116,7 +116,14 @@ def parse(data, raw=False, quiet=False):
     if jc.utils.has_data(data):
 
         for line in filter(None, data.splitlines()):
-            linedata = line.rsplit(': ', maxsplit=1)
+
+            # fix case for gzip files where description contains ': ' delimiter
+            if 'gzip compressed data, last modified: ' in line:
+                linedata = line.split(': ', maxsplit=1)
+
+            # use rsplit to correctly grab filenames containing ': ' delimiter text
+            else:
+                linedata = line.rsplit(': ', maxsplit=1)
 
             try:
                 filename = linedata[0].strip()
@@ -130,7 +137,7 @@ def parse(data, raw=False, quiet=False):
                 )
             except IndexError:
                 if not warned:
-                    jc.utils.warning_message('Filenames with newline characters detected. Some filenames may be truncated.')
+                    jc.utils.warning_message(['Filenames with newline characters detected. Some filenames may be truncated.'])
                     warned = True
 
     if raw:

jc/parsers/foo.py

@@ -65,6 +65,7 @@ def _process(proc_data):
         List of Dictionaries. Structured to conform to the schema.
     """
 
+    # process the data here
     # rebuild output for added semantic information
     # use helper functions in jc.utils for int, float, bool conversions and timestamps
 
@@ -93,10 +94,9 @@ def parse(data, raw=False, quiet=False):
     if jc.utils.has_data(data):
 
         for line in filter(None, data.splitlines()):
-            # parse the content
+            #
+            # parse the content here
+            #
             pass
 
-    if raw:
-        return raw_output
-    else:
-        return _process(raw_output)
+    return raw_output if raw else _process(raw_output)

jc/parsers/foo_s.py

@@ -0,0 +1,117 @@
+"""jc - JSON CLI output utility `foo` command output streaming parser
+
+> This streaming parser outputs JSON Lines
+
+<<Short foo description and caveats>>
+
+Usage (cli):
+
+    $ foo | jc --foo-s
+
+Usage (module):
+
+    import jc.parsers.foo_s
+    result = jc.parsers.foo_s.parse(foo_command_output.splitlines())    # result is an iterable object
+    for item in result:
+        # do something
+
+Schema:
+
+    {
+      "foo":            string,
+      "_jc_meta":                    # This object only exists if using -qq or ignore_exceptions=True
+        {
+          "success":    booean,      # true if successfully parsed, false if error
+          "error":      string,      # exists if "success" is false
+          "line":       string       # exists if "success" is false
+        }
+    }
+
+Examples:
+
+    $ foo | jc --foo-s
+    {example output}
+    ...
+
+    $ foo | jc --foo-s -r
+    {example output}
+    ...
+"""
+import jc.utils
+from jc.utils import stream_success, stream_error
+from jc.exceptions import ParseError
+
+
+class info():
+    """Provides parser metadata (version, author, etc.)"""
+    version = '1.0'
+    description = '`foo` command streaming parser'
+    author = 'John Doe'
+    author_email = 'johndoe@gmail.com'
+
+    # compatible options: linux, darwin, cygwin, win32, aix, freebsd
+    compatible = ['linux', 'darwin', 'cygwin', 'aix', 'freebsd']
+    streaming = True
+
+
+__version__ = info.version
+
+
+def _process(proc_data):
+    """
+    Final processing to conform to the schema.
+
+    Parameters:
+
+        proc_data:   (Dictionary) raw structured data to process
+
+    Returns:
+
+        Dictionary. Structured data to conform to the schema.
+    """
+    #
+    # process the data here
+    # rebuild output for added semantic information
+    # use helper functions in jc.utils for int, float, bool conversions and timestamps
+    #
+
+    return proc_data
+
+
+def parse(data, raw=False, quiet=False, ignore_exceptions=False):
+    """
+    Main text parsing generator function. Returns an iterator object.
+
+    Parameters:
+
+        data:              (iterable)  line-based text data to parse (e.g. sys.stdin or str.splitlines())
+        raw:               (boolean)   output preprocessed JSON if True
+        quiet:             (boolean)   suppress warning messages if True
+        ignore_exceptions: (boolean)   ignore parsing exceptions if True
+
+    Yields:
+
+        Dictionary. Raw or processed structured data.
+
+    Returns:
+
+        Iterator object
+    """
+    if not quiet:
+        jc.utils.compatibility(__name__, info.compatible)
+
+    for line in data:
+        try:
+            output_line = {}
+
+            #
+            # parse the input here
+            #
+
+            if output_line:
+                yield stream_success(output_line, ignore_exceptions) if raw else stream_success(_process(output_line), ignore_exceptions)
+            else:
+                raise ParseError('Not foo data')
+
+        except Exception as e:
+            yield stream_error(e, ignore_exceptions, line)

jc/parsers/kv.py

@@ -35,7 +35,7 @@ Examples:
     # but can be preserved with the -r argument
     occupation:"Engineer"
 
-    $ cat keyvalue.txt | jc --ini -p
+    $ cat keyvalue.txt | jc --kv -p
     {
       "name": "John Doe",
       "address": "555 California Drive",

jc/parsers/ls.py

@@ -255,7 +255,7 @@ def parse(data, raw=False, quiet=False):
                     continue
 
                 if not quiet and next_is_parent and not entry.endswith(':') and not warned:
-                    jc.utils.warning_message('Newline characters detected. Filenames probably corrupted. Use ls -l or -b instead.')
+                    jc.utils.warning_message(['Newline characters detected. Filenames probably corrupted. Use ls -l or -b instead.'])
                     warned = True
 
                 output_line['filename'] = entry

jc/parsers/ls_s.py

@@ -0,0 +1,180 @@
+"""jc - JSON CLI output utility `ls` and `vdir` command output streaming parser
+
+> This streaming parser outputs JSON Lines
+
+Requires the `-l` option to be used on `ls`. If there are newline characters in the filename, then make sure to use the `-b` option on `ls`.
+
+The `jc` `-qq` option can be used to ignore parsing errors. (e.g. filenames with newline characters, but `-b` was not used)
+
+The `epoch` calculated timestamp field is naive (i.e. based on the local time of the system the parser is run on)
+
+The `epoch_utc` calculated timestamp field is timezone-aware and is only available if the timezone field is UTC.
+
+Usage (cli):
+
+    $ ls | jc --ls-s
+
+Usage (module):
+
+    import jc.parsers.ls_s
+    result = jc.parsers.ls_s.parse(ls_command_output.splitlines())    # result is an iterable object
+    for item in result:
+        # do something
+
+Schema:
+
+    {
+      "filename":       string,
+      "flags":          string,
+      "links":          integer,
+      "parent":         string,
+      "owner":          string,
+      "group":          string,
+      "size":           integer,
+      "date":           string,
+      "epoch":          integer,     # naive timestamp if date field exists and can be converted
+      "epoch_utc":      integer,     # timezone aware timestamp if date field is in UTC and can be converted
+      "_jc_meta":                    # This object only exists if using -qq or ignore_exceptions=True
+        {
+          "success":    booean,      # true if successfully parsed, false if error
+          "error":      string,      # exists if "success" is false
+          "line":       string       # exists if "success" is false
+        }
+    }
+
+Examples:
+
+    $ ls -l /usr/bin | jc --ls-s
+    {"filename":"2to3-","flags":"-rwxr-xr-x","links":4,"owner":"root","group":"wheel","size":925,"date":"Feb 22 2019"}
+    {"filename":"2to3-2.7","link_to":"../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/2to3-2.7","flags":"lrwxr-xr-x","links":1,"owner":"root","group":"wheel","size":74,"date":"May 4 2019"}
+    {"filename":"AssetCacheLocatorUtil","flags":"-rwxr-xr-x","links":1,"owner":"root","group":"wheel","size":55152,"date":"May 3 2019"}
+    ...
+
+    $ ls -l /usr/bin | jc --ls-s -r
+    {"filename":"2to3-","flags":"-rwxr-xr-x","links":"4","owner":"root","group":"wheel","size":"925","date":"Feb 22 2019"}
+    {"filename":"2to3-2.7","link_to":"../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/2to3-2.7","flags":"lrwxr-xr-x","links":"1","owner":"root","group":"wheel","size":"74","date":"May 4 2019"}
+    {"filename":"AssetCacheLocatorUtil","flags":"-rwxr-xr-x","links":"1","owner":"root","group":"wheel","size":"55152","date":"May 3 2019"}
+    ...
+"""
+import re
+import jc.utils
+from jc.utils import stream_success, stream_error
+from jc.exceptions import ParseError
+
+
+class info():
+    """Provides parser metadata (version, author, etc.)"""
+    version = '0.5'
+    description = '`ls` command streaming parser'
+    author = 'Kelly Brazil'
+    author_email = 'kellyjonbrazil@gmail.com'
+
+    # compatible options: linux, darwin, cygwin, win32, aix, freebsd
+    compatible = ['linux', 'darwin', 'cygwin', 'aix', 'freebsd']
+    streaming = True
+
+
+__version__ = info.version
+
+
+def _process(proc_data):
+    """
+    Final processing to conform to the schema.
+
+    Parameters:
+
+        proc_data:   (Dictionary) raw structured data to process
+
+    Returns:
+
+        Dictionary. Structured data to conform to the schema.
+    """
+    int_list = ['links', 'size']
+    for key in proc_data:
+        if key in int_list:
+            proc_data[key] = jc.utils.convert_to_int(proc_data[key])
+
+    if 'date' in proc_data:
+        # to speed up processing only try to convert the date if it's not the default format
+        if not re.match(r'[a-zA-Z]{3}\s{1,2}\d{1,2}\s{1,2}[0-9:]{4,5}', proc_data['date']):
+            ts = jc.utils.timestamp(proc_data['date'])
+            proc_data['epoch'] = ts.naive
+            proc_data['epoch_utc'] = ts.utc
+
+    return proc_data
+
+
+def parse(data, raw=False, quiet=False, ignore_exceptions=False):
+    """
+    Main text parsing generator function. Returns an iterator object.
+
+    Parameters:
+
+        data:              (iterable)  line-based text data to parse (e.g. sys.stdin or str.splitlines())
+        raw:               (boolean)   output preprocessed JSON if True
+        quiet:             (boolean)   suppress warning messages if True
+        ignore_exceptions: (boolean)   ignore parsing exceptions if True
+
+    Yields:
+
+        Dictionary. Raw or processed structured data.
+
+    Returns:
+
+        Iterator object
+    """
+    if not quiet:
+        jc.utils.compatibility(__name__, info.compatible)
+
+    parent = ''
+
+    for line in data:
+        try:
+
+            # skip line if it starts with 'total 1234'
+            if re.match(r'total [0-9]+', line):
+                continue
+
+            # skip blank lines
+            if line.strip() == '':
+                continue
+
+            # Look for parent line if glob or -R is used
+            if not re.match(r'[-dclpsbDCMnP?]([-r][-w][-xsS]){2}([-r][-w][-xtT])[+]?', line) \
+               and line.strip().endswith(':'):
+                parent = line.strip()[:-1]
+                continue
+
+            if not re.match(r'[-dclpsbDCMnP?]([-r][-w][-xsS]){2}([-r][-w][-xtT])[+]?', line):
+                raise ParseError('Not ls -l data')
+
+            parsed_line = line.strip().split(maxsplit=8)
+            output_line = {}
+
+            # split filenames and links
+            if len(parsed_line) == 9:
+                filename_field = parsed_line[8].split(' -> ')
+            else:
+                # in case of filenames starting with a newline character
+                filename_field = ['']
+
+            # create output object
+            output_line['filename'] = filename_field[0]
+
+            if len(filename_field) > 1:
+                output_line['link_to'] = filename_field[1]
+
+            if parent:
+                output_line['parent'] = parent
+
+            output_line['flags'] = parsed_line[0]
+            output_line['links'] = parsed_line[1]
+            output_line['owner'] = parsed_line[2]
+            output_line['group'] = parsed_line[3]
+            output_line['size'] = parsed_line[4]
+            output_line['date'] = ' '.join(parsed_line[5:8])
+
+            yield stream_success(output_line, ignore_exceptions) if raw else stream_success(_process(output_line), ignore_exceptions)
+
+        except Exception as e:
+            yield stream_error(e, ignore_exceptions, line)

jc/parsers/lsusb.py

@@ -0,0 +1,844 @@
+"""jc - JSON CLI output utility `lsusb` command output parser
+
+Supports the `-v` option or no options.
+
+Usage (cli):
+
+    $ lsusb -v | jc --lsusb
+
+    or
+
+    $ jc lsusb -v
+
+Usage (module):
+
+    import jc.parsers.lsusb
+    result = jc.parsers.lsusb.parse(lsusb_command_output)
+
+Schema:
+
+    Note: <item> object keynames are assigned directly from the lsusb output.
+          If there are duplicate <item> names in a section, only the last one is converted.
+
+    [
+      {
+        "bus":                                string,
+        "device":                             string,
+        "id":                                 string,
+        "description":                        string,
+        "device_descriptor": {
+          "<item>": {
+            "value":                          string,
+            "description":                    string,
+            "attributes": [
+                                              string
+            ]
+          },
+          "configuration_descriptor": {
+            "<item>": {
+              "value":                        string,
+              "description":                  string,
+              "attributes": [
+                                              string
+              ]
+            },
+            "interface_association": {
+              "<item>": {
+                "value":                      string,
+                "description":                string,
+                "attributes": [
+                                              string
+                ]
+              }
+            },
+            "interface_descriptors": [
+              {
+                "<item>": {
+                  "value":                    string,
+                  "description":              string,
+                  "attributes": [
+                                              string
+                  ]
+                },
+                "cdc_header": {
+                  "<item>": {
+                    "value":                  string,
+                    "description":            string,
+                    "attributes": [
+                                              string
+                    ]
+                  }
+                },
+                "cdc_call_management": {
+                  "<item>": {
+                    "value":                  string,
+                    "description":            string,
+                    "attributes": [
+                                              string
+                    ]
+                  }
+                },
+                "cdc_acm": {
+                  "<item>": {
+                    "value":                  string,
+                    "description":            string,
+                    "attributes": [
+                                              string
+                    ]
+                  }
+                },
+                "cdc_union": {
+                  "<item>": {
+                    "value":                  string,
+                    "description":            string,
+                    "attributes": [
+                                              string
+                    ]
+                  }
+                },
+                "endpoint_descriptors": [
+                  {
+                    "<item>": {
+                      "value":                string,
+                      "description":          string,
+                      "attributes": [
+                                              string
+                      ]
+                    }
+                  }
+                ]
+              }
+            ]
+          }
+        },
+        "hub_descriptor": {
+          "<item>": {
+            "value":                          string,
+            "description":                    string,
+            "attributes": [
+                                              string,
+            ]
+          },
+          "hub_port_status": {
+            "<item>": {
+              "value":                        string,
+              "attributes": [
+                                              string
+              ]
+            }
+          }
+        },
+        "device_status": {
+          "value":                            string,
+          "description":                      string
+        }
+      }
+    ]
+
+Examples:
+
+    $ lsusb -v | jc --lsusb -p
+    [
+      {
+        "bus": "002",
+        "device": "001",
+        "id": "1d6b:0001",
+        "description": "Linux Foundation 1.1 root hub",
+        "device_descriptor": {
+          "bLength": {
+            "value": "18"
+          },
+          "bDescriptorType": {
+            "value": "1"
+          },
+          "bcdUSB": {
+            "value": "1.10"
+          },
+          ...
+          "bNumConfigurations": {
+            "value": "1"
+          },
+          "configuration_descriptor": {
+            "bLength": {
+              "value": "9"
+            },
+            ...
+            "iConfiguration": {
+              "value": "0"
+            },
+            "bmAttributes": {
+              "value": "0xe0",
+              "attributes": [
+                "Self Powered",
+                "Remote Wakeup"
+              ]
+            },
+            "MaxPower": {
+              "description": "0mA"
+            },
+            "interface_descriptors": [
+              {
+                "bLength": {
+                  "value": "9"
+                },
+                ...
+                "bInterfaceProtocol": {
+                  "value": "0",
+                  "description": "Full speed (or root) hub"
+                },
+                "iInterface": {
+                  "value": "0"
+                },
+                "endpoint_descriptors": [
+                  {
+                    "bLength": {
+                      "value": "7"
+                    },
+                    ...
+                    "bmAttributes": {
+                      "value": "3",
+                      "attributes": [
+                        "Transfer Type  Interrupt",
+                        "Synch Type  None",
+                        "Usage Type  Data"
+                      ]
+                    },
+                    "wMaxPacketSize": {
+                      "value": "0x0002",
+                      "description": "1x 2 bytes"
+                    },
+                    "bInterval": {
+                      "value": "255"
+                    }
+                  }
+                ]
+              }
+            ]
+          }
+        },
+        "hub_descriptor": {
+          "bLength": {
+            "value": "9"
+          },
+          ...
+          "wHubCharacteristic": {
+            "value": "0x000a",
+            "attributes": [
+              "No power switching (usb 1.0)",
+              "Per-port overcurrent protection"
+            ]
+          },
+          ...
+          "hub_port_status": {
+            "Port 1": {
+              "value": "0000.0103",
+              "attributes": [
+                "power",
+                "enable",
+                "connect"
+              ]
+            },
+            "Port 2": {
+              "value": "0000.0103",
+              "attributes": [
+                "power",
+                "enable",
+                "connect"
+              ]
+            }
+          }
+        },
+        "device_status": {
+          "value": "0x0001",
+          "description": "Self Powered"
+        }
+      }
+    ]
+"""
+import jc.utils
+from jc.parsers.universal import sparse_table_parse
+from jc.exceptions import ParseError
+
+
+class info():
+    """Provides parser metadata (version, author, etc.)"""
+    version = '1.0'
+    description = '`lsusb` command parser'
+    author = 'Kelly Brazil'
+    author_email = 'kellyjonbrazil@gmail.com'
+    compatible = ['linux']
+    magic_commands = ['lsusb']
+
+
+__version__ = info.version
+
+
+def _process(proc_data):
+    """
+    Final processing to conform to the schema.
+
+    Parameters:
+
+        proc_data:   (List of Dictionaries) raw structured data to process
+
+    Returns:
+
+        List of Dictionaries. Structured to conform to the schema.
+    """
+    # no further processing
+    return proc_data
+
+
+class _NestedDict(dict):
+    # for ease of creating/updating nested dictionary structures
+    # https://stackoverflow.com/questions/5369723/multi-level-defaultdict-with-variable-depth
+    # https://ohuiginn.net/mt/2010/07/nested_dictionaries_in_python.html
+    def __getitem__(self, key):
+        if key in self:
+            return self.get(key)
+        return self.setdefault(key, _NestedDict())
+
+
+class _LsUsb():
+    def __init__(self):
+        self.raw_output = []
+        self.output_line = _NestedDict()
+
+        self.section = ''
+        self.old_section = ''
+        self.bus_idx = -1
+        self.interface_descriptor_idx = -1
+        self.endpoint_descriptor_idx = -1
+        self.last_item = ''
+        self.last_indent = 0
+        self.attribute_value = False
+
+        self.bus_list = []
+        self.device_descriptor_list = []
+        self.configuration_descriptor_list = []
+        self.interface_association_list = []
+        self.interface_descriptor_list = []
+        self.interface_descriptor_attribute_list = []
+        self.cdc_header_list = []
+        self.cdc_call_management_list = []
+        self.cdc_acm_list = []
+        self.cdc_union_list = []
+        self.endpoint_descriptor_list = []
+        self.hid_device_descriptor_list = []
+        self.report_descriptors_list = []
+        self.hub_descriptor_list = []
+        self.hub_port_status_list = []
+        self.device_status_list = []
+
+    @staticmethod
+    def _count_indent(line):
+        indent = 0
+        for char in line:
+            if char == ' ':
+                indent += 1
+                continue
+            break
+        return indent
+
+    def _add_attributes(self, line):
+        indent = self._count_indent(line)
+
+        # determine whether this is a top-level value item or lower-level attribute
+        if indent > self.last_indent and self.old_section == self.section:
+            self.attribute_value = True
+        elif indent == self.last_indent and self.attribute_value and self.old_section == self.section:
+            self.attribute_value = True
+        else:
+            self.attribute_value = False
+
+        # section_header is formatted with the correct spacing to be used with
+        # jc.parsers.universal.sparse_table_parse(). Pad end of string to be at least len of 25
+        section_header = 'key                   val description'
+
+        temp_obj = [section_header, line.strip() + (' ' * 25)]
+        temp_obj = sparse_table_parse(temp_obj)
+        temp_obj = temp_obj[0]
+
+        line_obj = {
+            temp_obj['key']: {
+                'value': temp_obj['val'],
+                'description': temp_obj['description'],
+                '_state': {
+                    'attribute_value': self.attribute_value,
+                    'last_item': self.last_item,
+                    'bus_idx': self.bus_idx,
+                    'interface_descriptor_idx': self.interface_descriptor_idx,
+                    'endpoint_descriptor_idx': self.endpoint_descriptor_idx
+                }
+            }
+        }
+
+        if line_obj[temp_obj['key']]['value'] is None:
+            del line_obj[temp_obj['key']]['value']
+
+        if line_obj[temp_obj['key']]['description'] is None:
+            del line_obj[temp_obj['key']]['description']
+
+        self.old_section = self.section
+        self.last_indent = indent
+
+        if not self.attribute_value:
+            self.last_item = temp_obj['key']
+
+        return line_obj
+
+    def _add_hub_port_status_attributes(self, line):
+        # Port 1: 0000.0103 power enable connect
+        first_split = line.split(': ', maxsplit=1)
+        port_field = first_split[0].strip()
+        second_split = first_split[1].split(maxsplit=1)
+        port_val = second_split[0]
+        attributes = second_split[1].split()
+
+        return {
+            port_field: {
+                'value': port_val,
+                'attributes': attributes,
+                '_state': {
+                    'bus_idx': self.bus_idx
+                }
+            }
+        }
+
+    def _add_device_status_attributes(self, line):
+        return {
+            'description': line.strip(),
+            '_state': {
+                'bus_idx': self.bus_idx
+            }
+        }
+
+    def _set_sections(self, line):
+        # ignore blank lines
+        if not line:
+            self.section = ''
+            self.attribute_value = False
+            return True
+
+        # bus information is on the same line so need to extract data immediately and set indexes
+        if line.startswith('Bus '):
+            self.section = 'bus'
+            self.bus_idx += 1
+            self.interface_descriptor_idx = -1
+            self.endpoint_descriptor_idx = -1
+            self.attribute_value = False
+            line_split = line.strip().split(maxsplit=6)
+            self.bus_list.append(
+                {
+                    'bus': line_split[1],
+                    'device': line_split[3][:-1],
+                    'id': line_split[5],
+                    'description': (line_split[6:7] or [None])[0],     # way to get a list item or None
+                    '_state': {
+                        'bus_idx': self.bus_idx
+                    }
+                }
+            )
+            return True
+
+        # This section is a list, so need to update indexes
+        if line.startswith('    Interface Descriptor:'):
+            self.section = 'interface_descriptor'
+            self.interface_descriptor_idx += 1
+            self.endpoint_descriptor_idx = -1
+            self.attribute_value = False
+            return True
+
+        # This section is a list, so need to update the index
+        if line.startswith('      Endpoint Descriptor:'):
+            self.section = 'endpoint_descriptor'
+            self.endpoint_descriptor_idx += 1
+            self.attribute_value = False
+            return True
+
+        # some device status information is displayed on the initial line so need to extract immediately
+        if line.startswith('Device Status:'):
+            self.section = 'device_status'
+            self.attribute_value = False
+            line_split = line.strip().split(':', maxsplit=1)
+            self.device_status_list.append(
+                {
+                    'value': line_split[1].strip(),
+                    '_state': {
+                        'bus_idx': self.bus_idx
+                    }
+                }
+            )
+            return True
+
+        # set the rest of the sections
+        string_section_map = {
+            'Device Descriptor:': 'device_descriptor',
+            '  Configuration Descriptor:': 'configuration_descriptor',
+            '    Interface Association:': 'interface_association',
+            '      CDC Header:': 'cdc_header',
+            '      CDC Call Management:': 'cdc_call_management',
+            '      CDC ACM:': 'cdc_acm',
+            '      CDC Union:': 'cdc_union',
+            '        HID Device Descriptor:': 'hid_device_descriptor',
+            '         Report Descriptors:': 'report_descriptors',
+            'Hub Descriptor:': 'hub_descriptor',
+            ' Hub Port Status:': 'hub_port_status'
+        }
+
+        for sec_string, section_val in string_section_map.items():
+            if line.startswith(sec_string):
+                self.section = section_val
+                self.attribute_value = False
+                return True
+
+        return False
+
+    def _populate_lists(self, line):
+        section_list_map = {
+            'device_descriptor': self.device_descriptor_list,
+            'configuration_descriptor': self.configuration_descriptor_list,
+            'interface_association': self.interface_association_list,
+            'interface_descriptor': self.interface_descriptor_list,
+            'cdc_header': self.cdc_header_list,
+            'cdc_call_management': self.cdc_call_management_list,
+            'cdc_acm': self.cdc_acm_list,
+            'cdc_union': self.cdc_union_list,
+            'hid_device_descriptor': self.hid_device_descriptor_list,
+            'report_descriptors': self.report_descriptors_list,
+            'endpoint_descriptor': self.endpoint_descriptor_list,
+            'hub_descriptor': self.hub_descriptor_list
+        }
+
+        for sec in section_list_map:
+            if line.startswith(' ') and self.section == sec:
+                section_list_map[self.section].append(self._add_attributes(line))
+                return True
+
+        # special handling of these sections
+        if line.startswith(' ') and self.section == 'hub_port_status':
+            self.hub_port_status_list.append(self._add_hub_port_status_attributes(line))
+            return True
+
+        if line.startswith(' ') and self.section == 'device_status':
+            self.device_status_list.append(self._add_device_status_attributes(line))
+            return True
+
+        return False
+
+    def _populate_schema(self):
+        """
+        Schema:
+        = {}
+        ['device_descriptor'] = {}
+        ['device_descriptor']['configuration_descriptor'] = {}
+        ['device_descriptor']['configuration_descriptor']['interface_association'] = {}
+        ['device_descriptor']['configuration_descriptor']['interface_descriptors'] = []
+        ['device_descriptor']['configuration_descriptor']['interface_descriptors'][0] = {}
+        ['device_descriptor']['configuration_descriptor']['interface_descriptors'][0]['cdc_header'] = {}
+        ['device_descriptor']['configuration_descriptor']['interface_descriptors'][0]['cdc_call_management'] = {}
+        ['device_descriptor']['configuration_descriptor']['interface_descriptors'][0]['cdc_acm'] = {}
+        ['device_descriptor']['configuration_descriptor']['interface_descriptors'][0]['cdc_union'] = {}
+        ['device_descriptor']['configuration_descriptor']['interface_descriptors'][0]['hid_device_descriptor'] = {}
+        ['device_descriptor']['configuration_descriptor']['interface_descriptors'][0]['endpoint_descriptors'] = []
+        ['device_descriptor']['configuration_descriptor']['interface_descriptors'][0]['endpoint_descriptors'][0] = {}
+        ['hub_descriptor'] = {}
+        ['hub_descriptor']['hub_port_status'] = {}
+        ['device_status'] = {}
+        """
+        for idx, item in enumerate(self.bus_list):
+            if self.output_line:
+                self.raw_output.append(self.output_line)
+
+            self.output_line = _NestedDict()
+
+            del item['_state']
+            self.output_line.update(item)
+
+            for dd in self.device_descriptor_list:
+                keyname = tuple(dd.keys())[0]
+                if '_state' in dd[keyname] and dd[keyname]['_state']['bus_idx'] == idx:
+
+                    # is this a top level value or an attribute?
+                    if dd[keyname]['_state']['attribute_value']:
+                        last_item = dd[keyname]['_state']['last_item']
+                        if 'attributes' not in self.output_line['device_descriptor'][last_item]:
+                            self.output_line['device_descriptor'][last_item]['attributes'] = []
+
+                        this_attribute = f'{keyname} {dd[keyname].get("value", "")} {dd[keyname].get("description", "")}'.strip()
+                        self.output_line['device_descriptor'][last_item]['attributes'].append(this_attribute)
+                        continue
+
+                    self.output_line['device_descriptor'].update(dd)
+                    del self.output_line['device_descriptor'][keyname]['_state']
+
+            for cd in self.configuration_descriptor_list:
+                keyname = tuple(cd.keys())[0]
+                if '_state' in cd[keyname] and cd[keyname]['_state']['bus_idx'] == idx:
+
+                    # is this a top level value or an attribute?
+                    if cd[keyname]['_state']['attribute_value']:
+                        last_item = cd[keyname]['_state']['last_item']
+                        if 'attributes' not in self.output_line['device_descriptor']['configuration_descriptor'][last_item]:
+                            self.output_line['device_descriptor']['configuration_descriptor'][last_item]['attributes'] = []
+
+                        this_attribute = f'{keyname} {cd[keyname].get("value", "")} {cd[keyname].get("description", "")}'.strip()
+                        self.output_line['device_descriptor']['configuration_descriptor'][last_item]['attributes'].append(this_attribute)
+                        continue
+
+                    self.output_line['device_descriptor']['configuration_descriptor'].update(cd)
+                    del self.output_line['device_descriptor']['configuration_descriptor'][keyname]['_state']
+
+            for ia in self.interface_association_list:
+                keyname = tuple(ia.keys())[0]
+                if '_state' in ia[keyname] and ia[keyname]['_state']['bus_idx'] == idx:
+
+                    # is this a top level value or an attribute?
+                    if ia[keyname]['_state']['attribute_value']:
+                        last_item = ia[keyname]['_state']['last_item']
+                        if 'attributes' not in self.output_line['device_descriptor']['configuration_descriptor']['interface_association'][last_item]:
+                            self.output_line['device_descriptor']['configuration_descriptor']['interface_association'][last_item]['attributes'] = []
+
+                        this_attribute = f'{keyname} {ia[keyname].get("value", "")} {ia[keyname].get("description", "")}'.strip()
+                        self.output_line['device_descriptor']['configuration_descriptor']['interface_association'][last_item]['attributes'].append(this_attribute)
+                        continue
+
+                    self.output_line['device_descriptor']['configuration_descriptor']['interface_association'].update(ia)
+                    del self.output_line['device_descriptor']['configuration_descriptor']['interface_association'][keyname]['_state']
+
+            # add interface_descriptor key if it doesn't exist and there are entries for this bus
+            for iface_attrs in self.interface_descriptor_list:
+                keyname = tuple(iface_attrs.keys())[0]
+                if '_state' in iface_attrs[keyname] and iface_attrs[keyname]['_state']['bus_idx'] == idx:
+                    self.output_line['device_descriptor']['configuration_descriptor']['interface_descriptors'] = []
+
+            # find max index for this bus idx, then iterate over that range
+            i_desc_iters = -1
+            for iface_attrs in self.interface_descriptor_list:
+                keyname = tuple(iface_attrs.keys())[0]
+                if '_state' in iface_attrs[keyname] and iface_attrs[keyname]['_state']['bus_idx'] == idx:
+                    i_desc_iters = iface_attrs[keyname]['_state']['interface_descriptor_idx']
+
+            # create the interface descriptor object
+            if i_desc_iters > -1:
+                for iface_idx in range(i_desc_iters + 1):
+                    i_desc_obj = _NestedDict()
+                    for iface_attrs in self.interface_descriptor_list:
+                        keyname = tuple(iface_attrs.keys())[0]
+                        if '_state' in iface_attrs[keyname] and iface_attrs[keyname]['_state']['bus_idx'] == idx and iface_attrs[keyname]['_state']['interface_descriptor_idx'] == iface_idx:
+
+                            # is this a top level value or an attribute?
+                            if iface_attrs[keyname]['_state']['attribute_value']:
+                                last_item = iface_attrs[keyname]['_state']['last_item']
+                                if 'attributes' not in i_desc_obj[last_item]:
+                                    i_desc_obj[last_item]['attributes'] = []
+
+                                this_attribute = f'{keyname} {iface_attrs[keyname].get("value", "")} {iface_attrs[keyname].get("description", "")}'.strip()
+                                i_desc_obj[last_item]['attributes'].append(this_attribute)
+                                continue
+
+                            del iface_attrs[keyname]['_state']
+                            i_desc_obj.update(iface_attrs)
+
+                    # add other nodes to the object (cdc_header, endpoint descriptors, etc.)
+                    for ch in self.cdc_header_list:
+                        keyname = tuple(ch.keys())[0]
+                        if '_state' in ch[keyname] and ch[keyname]['_state']['bus_idx'] == idx and ch[keyname]['_state']['interface_descriptor_idx'] == iface_idx:
+
+                            # is this a top level value or an attribute?
+                            if ch[keyname]['_state']['attribute_value']:
+                                last_item = ch[keyname]['_state']['last_item']
+                                if 'attributes' not in i_desc_obj['cdc_header'][last_item]:
+                                    i_desc_obj['cdc_header'][last_item]['attributes'] = []
+
+                                this_attribute = f'{keyname} {ch[keyname].get("value", "")} {ch[keyname].get("description", "")}'.strip()
+                                i_desc_obj['cdc_header'][last_item]['attributes'].append(this_attribute)
+                                continue
+
+                            i_desc_obj['cdc_header'].update(ch)
+                            del i_desc_obj['cdc_header'][keyname]['_state']
+
+                    for ccm in self.cdc_call_management_list:
+                        keyname = tuple(ccm.keys())[0]
+                        if '_state' in ccm[keyname] and ccm[keyname]['_state']['bus_idx'] == idx and ccm[keyname]['_state']['interface_descriptor_idx'] == iface_idx:
+
+                            # is this a top level value or an attribute?
+                            if ccm[keyname]['_state']['attribute_value']:
+                                last_item = ccm[keyname]['_state']['last_item']
+                                if 'attributes' not in i_desc_obj['cdc_call_management'][last_item]:
+                                    i_desc_obj['cdc_call_management'][last_item]['attributes'] = []
+
+                                this_attribute = f'{keyname} {ccm[keyname].get("value", "")} {ccm[keyname].get("description", "")}'.strip()
+                                i_desc_obj['cdc_call_management'][last_item]['attributes'].append(this_attribute)
+                                continue
+
+                            i_desc_obj['cdc_call_management'].update(ccm)
+                            del i_desc_obj['cdc_call_management'][keyname]['_state']
+
+                    for ca in self.cdc_acm_list:
+                        keyname = tuple(ca.keys())[0]
+                        if '_state' in ca[keyname] and ca[keyname]['_state']['bus_idx'] == idx and ca[keyname]['_state']['interface_descriptor_idx'] == iface_idx:
+
+                            # is this a top level value or an attribute?
+                            if ca[keyname]['_state']['attribute_value']:
+                                last_item = ca[keyname]['_state']['last_item']
+                                if 'attributes' not in i_desc_obj['cdc_acm'][last_item]:
+                                    i_desc_obj['cdc_acm'][last_item]['attributes'] = []
+
+                                this_attribute = f'{keyname} {ca[keyname].get("value", "")} {ca[keyname].get("description", "")}'.strip()
+                                i_desc_obj['cdc_acm'][last_item]['attributes'].append(this_attribute)
+                                continue
+
+                            i_desc_obj['cdc_acm'].update(ca)
+                            del i_desc_obj['cdc_acm'][keyname]['_state']
+
+                    for cu in self.cdc_union_list:
+                        keyname = tuple(cu.keys())[0]
+                        if '_state' in cu[keyname] and cu[keyname]['_state']['bus_idx'] == idx and cu[keyname]['_state']['interface_descriptor_idx'] == iface_idx:
+
+                            # is this a top level value or an attribute?
+                            if cu[keyname]['_state']['attribute_value']:
+                                last_item = cu[keyname]['_state']['last_item']
+                                if 'attributes' not in i_desc_obj['cdc_union'][last_item]:
+                                    i_desc_obj['cdc_union'][last_item]['attributes'] = []
+
+                                this_attribute = f'{keyname} {cu[keyname].get("value", "")} {cu[keyname].get("description", "")}'.strip()
+                                i_desc_obj['cdc_union'][last_item]['attributes'].append(this_attribute)
+                                continue
+
+                            i_desc_obj['cdc_union'].update(cu)
+                            del i_desc_obj['cdc_union'][keyname]['_state']
+
+                    for hidd in self.hid_device_descriptor_list:
+                        keyname = tuple(hidd.keys())[0]
+                        if '_state' in hidd[keyname] and hidd[keyname]['_state']['bus_idx'] == idx and hidd[keyname]['_state']['interface_descriptor_idx'] == iface_idx:
+
+                            # is this a top level value or an attribute?
+                            if hidd[keyname]['_state']['attribute_value']:
+                                last_item = hidd[keyname]['_state']['last_item']
+                                if 'attributes' not in i_desc_obj['hid_device_descriptor'][last_item]:
+                                    i_desc_obj['hid_device_descriptor'][last_item]['attributes'] = []
+
+                                this_attribute = f'{keyname} {hidd[keyname].get("value", "")} {hidd[keyname].get("description", "")}'.strip()
+                                i_desc_obj['hid_device_descriptor'][last_item]['attributes'].append(this_attribute)
+                                continue
+
+                            i_desc_obj['hid_device_descriptor'].update(hidd)
+                            del i_desc_obj['hid_device_descriptor'][keyname]['_state']
+
+                        # Not Implemented: Report Descriptors (need more samples)
+                        # for rd in self.report_descriptors_list:
+                        #     keyname = tuple(rd.keys())[0]
+                        #     if '_state' in rd[keyname] and rd[keyname]['_state']['bus_idx'] == idx and rd[keyname]['_state']['interface_descriptor_idx'] == iface_idx:
+                        #         i_desc_obj['hid_device_descriptor']['report_descriptors'].update(rd)
+                        #         del i_desc_obj['hid_device_descriptor']['report_descriptors'][keyname]['_state']
+
+                    # add endpoint_descriptor key if it doesn't exist and there are entries for this interface_descriptor
+                    for endpoint_attrs in self.endpoint_descriptor_list:
+                        keyname = tuple(endpoint_attrs.keys())[0]
+                        if '_state' in endpoint_attrs[keyname] and endpoint_attrs[keyname]['_state']['bus_idx'] == idx and endpoint_attrs[keyname]['_state']['interface_descriptor_idx'] == iface_idx:
+                            i_desc_obj['endpoint_descriptors'] = []
+
+                    # find max index for this endpoint_descriptor idx, then iterate over that range
+                    e_desc_iters = -1
+                    for endpoint_attrs in self.endpoint_descriptor_list:
+                        keyname = tuple(endpoint_attrs.keys())[0]
+                        if '_state' in endpoint_attrs[keyname] and endpoint_attrs[keyname]['_state']['bus_idx'] == idx and endpoint_attrs[keyname]['_state']['interface_descriptor_idx'] == iface_idx:
+                            e_desc_iters = endpoint_attrs[keyname]['_state']['endpoint_descriptor_idx']
+
+                    # create the endpoint descriptor object
+                    if e_desc_iters > -1:
+                        for endpoint_idx in range(e_desc_iters + 1):
+                            e_desc_obj = {}
+                            for endpoint_attrs in self.endpoint_descriptor_list:
+                                keyname = tuple(endpoint_attrs.keys())[0]
+                                if '_state' in endpoint_attrs[keyname] and endpoint_attrs[keyname]['_state']['bus_idx'] == idx and endpoint_attrs[keyname]['_state']['interface_descriptor_idx'] == iface_idx and endpoint_attrs[keyname]['_state']['endpoint_descriptor_idx'] == endpoint_idx:
+
+                                    # is this a top level value or an attribute?
+                                    if endpoint_attrs[keyname]['_state']['attribute_value']:
+                                        last_item = endpoint_attrs[keyname]['_state']['last_item']
+                                        if 'attributes' not in e_desc_obj[last_item]:
+                                            e_desc_obj[last_item]['attributes'] = []
+
+                                        this_attribute = f'{keyname} {endpoint_attrs[keyname].get("value", "")} {endpoint_attrs[keyname].get("description", "")}'.strip()
+                                        e_desc_obj[last_item]['attributes'].append(this_attribute)
+                                        continue
+
+                                    e_desc_obj.update(endpoint_attrs)
+                                    del endpoint_attrs[keyname]['_state']
+
+                            i_desc_obj['endpoint_descriptors'].append(e_desc_obj)
+
+                    # add the object to the list of interface descriptors
+                    self.output_line['device_descriptor']['configuration_descriptor']['interface_descriptors'].append(i_desc_obj)
+
+            for hd in self.hub_descriptor_list:
+                keyname = tuple(hd.keys())[0]
+                if '_state' in hd[keyname] and hd[keyname]['_state']['bus_idx'] == idx:
+
+                    # is this a top level value or an attribute?
+                    if hd[keyname]['_state']['attribute_value']:
+                        last_item = hd[keyname]['_state']['last_item']
+                        if 'attributes' not in self.output_line['hub_descriptor'][last_item]:
+                            self.output_line['hub_descriptor'][last_item]['attributes'] = []
+
+                        this_attribute = f'{keyname} {hd[keyname].get("value", "")} {hd[keyname].get("description", "")}'.strip()
+                        self.output_line['hub_descriptor'][last_item]['attributes'].append(this_attribute)
+                        continue
+
+                    self.output_line['hub_descriptor'].update(hd)
+                    del self.output_line['hub_descriptor'][keyname]['_state']
+
+            for hps in self.hub_port_status_list:
+                keyname = tuple(hps.keys())[0]
+                if '_state' in hps[keyname] and hps[keyname]['_state']['bus_idx'] == idx:
+                    self.output_line['hub_descriptor']['hub_port_status'].update(hps)
+                    del self.output_line['hub_descriptor']['hub_port_status'][keyname]['_state']
+
+            for ds in self.device_status_list:
+                if '_state' in ds and ds['_state']['bus_idx'] == idx:
+                    self.output_line['device_status'].update(ds)
+                    del self.output_line['device_status']['_state']
+
+
+def parse(data, raw=False, quiet=False):
+    """
+    Main text parsing function
+
+    Parameters:
+
+        data:        (string)  text data to parse
+        raw:         (boolean) output preprocessed JSON if True
+        quiet:       (boolean) suppress warning messages if True
+
+    Returns:
+
+        List of Dictionaries. Raw or processed structured data.
+    """
+    if not quiet:
+        jc.utils.compatibility(__name__, info.compatible)
+
+    lsusb = _LsUsb()
+
+    if jc.utils.has_data(data):
+        for line in data.splitlines():
+            # only -v option or no options are supported
+            if line.startswith('/'):
+                raise ParseError('Only `lsusb` or `lsusb -v` are supported.')
+
+            # sections
+            if lsusb._set_sections(line):
+                continue
+
+            # create section lists and schema
+            if lsusb._populate_lists(line):
+                continue
+
+    # populate the schema
+    lsusb._populate_schema()
+
+    # add any final output object if it exists and return the raw_output list
+    if lsusb.output_line:
+        lsusb.raw_output.append(lsusb.output_line)
+
+    return lsusb.raw_output if raw else _process(lsusb.raw_output)

jc/parsers/netstat.py

@@ -354,7 +354,7 @@ import jc.utils
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.10'
+    version = '1.11'
     description = '`netstat` command parser'
     author = 'Kelly Brazil'
     author_email = 'kellyjonbrazil@gmail.com'

jc/parsers/netstat_linux.py

@@ -8,7 +8,7 @@ def normalize_headers(header):
     header = header.replace('foreign address', 'foreign_address')
     header = header.replace('pid/program name', 'program_name')
     header = header.replace('security context', 'security_context')
-    header = header.replace('i-node', 'inode')
+    header = header.replace('i-node', ' inode')
     header = header.replace('-', '_')
 
     return header

jc/parsers/ping.py

@@ -158,7 +158,7 @@ import jc.utils
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.5'
+    version = '1.6'
     description = '`ping` and `ping6` command parser'
     author = 'Kelly Brazil'
     author_email = 'kellyjonbrazil@gmail.com'
@@ -480,7 +480,7 @@ def _bsd_parse(data):
                         'round_trip_ms_min': split_line[0],
                         'round_trip_ms_avg': split_line[1],
                         'round_trip_ms_max': split_line[2],
-                        'round_trip_ms_stddev': split_line[3].replace(' ms', '')
+                        'round_trip_ms_stddev': split_line[3].replace(' ms', '') if len(split_line) == 4 else None
                     }
                 )
 

jc/parsers/ping_s.py

@@ -0,0 +1,526 @@
+"""jc - JSON CLI output utility `ping` command output streaming parser
+
+> This streaming parser outputs JSON Lines
+
+Supports `ping` and `ping6` output.
+
+Usage (cli):
+
+    $ ping | jc --ping-s
+
+> Note: When piping `jc` converted `ping` output to other processes it may appear the output is hanging due to the OS pipe buffers. This is because `ping` output is too small to quickly fill up the buffer. Use the `-u` option to unbuffer the `jc` output if you would like immediate output. See the [readme](https://github.com/kellyjonbrazil/jc/tree/master#unbuffering-output) for more information.
+
+Usage (module):
+
+    import jc.parsers.ping_s
+    result = jc.parsers.ping_s.parse(ping_command_output.splitlines())    # result is an iterable object
+    for item in result:
+        # do something
+
+Schema:
+
+    {
+      "type":                        string,        # 'reply', 'timeout', 'summary', etc. See `_error_type.type_map` for all options.
+      "source_ip":                   string,
+      "destination_ip":              string,
+      "sent_bytes":                  integer,
+      "pattern":                     string,        # (null if not set)
+      "destination":                 string,
+      "timestamp":                   float,
+      "response_bytes":              integer,
+      "response_ip":                 string,
+      "icmp_seq":                    integer,
+      "ttl":                         integer,
+      "time_ms":                     float,
+      "duplicate":                   boolean,
+      "packets_transmitted":         integer,
+      "packets_received":            integer,
+      "packet_loss_percent":         float,
+      "duplicates":                  integer,
+      "round_trip_ms_min":           float,
+      "round_trip_ms_avg":           float,
+      "round_trip_ms_max":           float,
+      "round_trip_ms_stddev":        float,
+      "_jc_meta":                                  # This object only exists if using -qq or ignore_exceptions=True
+        {
+          "success":                 booean,       # true if successfully parsed, false if error
+          "error":                   string,       # exists if "success" is false
+          "line":                    string        # exists if "success" is false
+        }
+    }
+
+Examples:
+
+    $ ping 1.1.1.1 | jc --ping-s
+    {"type":"reply","destination_ip":"1.1.1.1","sent_bytes":56,"pattern":null,"response_bytes":64,"response_ip":"1.1.1.1","icmp_seq":0,"ttl":56,"time_ms":23.703}
+    {"type":"reply","destination_ip":"1.1.1.1","sent_bytes":56,"pattern":null,"response_bytes":64,"response_ip":"1.1.1.1","icmp_seq":1,"ttl":56,"time_ms":22.862}
+    {"type":"reply","destination_ip":"1.1.1.1","sent_bytes":56,"pattern":null,"response_bytes":64,"response_ip":"1.1.1.1","icmp_seq":2,"ttl":56,"time_ms":22.82}
+    ...
+
+    $ ping 1.1.1.1 | jc --ping-s -r
+    {"type":"reply","destination_ip":"1.1.1.1","sent_bytes":"56","pattern":null,"response_bytes":"64","response_ip":"1.1.1.1","icmp_seq":"0","ttl":"56","time_ms":"23.054"}
+    {"type":"reply","destination_ip":"1.1.1.1","sent_bytes":"56","pattern":null,"response_bytes":"64","response_ip":"1.1.1.1","icmp_seq":"1","ttl":"56","time_ms":"24.739"}
+    {"type":"reply","destination_ip":"1.1.1.1","sent_bytes":"56","pattern":null,"response_bytes":"64","response_ip":"1.1.1.1","icmp_seq":"2","ttl":"56","time_ms":"23.232"}
+    ...
+"""
+import string
+import ipaddress
+import jc.utils
+from jc.exceptions import ParseError
+from jc.utils import stream_success, stream_error
+
+
+class info():
+    """Provides parser metadata (version, author, etc.)"""
+    version = '0.5'
+    description = '`ping` and `ping6` command streaming parser'
+    author = 'Kelly Brazil'
+    author_email = 'kellyjonbrazil@gmail.com'
+
+    # compatible options: linux, darwin, cygwin, win32, aix, freebsd
+    compatible = ['linux', 'darwin', 'freebsd']
+    streaming = True
+
+
+__version__ = info.version
+
+
+def _process(proc_data):
+    """
+    Final processing to conform to the schema.
+
+    Parameters:
+
+        proc_data:   (Dictionary) raw structured data to process
+
+    Returns:
+
+        Dictionary. Structured data to conform to the schema.
+    """
+    int_list = ['sent_bytes', 'packets_transmitted', 'packets_received', 'response_bytes', 'icmp_seq', 'ttl',
+                'duplicates', 'vr', 'hl', 'tos', 'len', 'id', 'flg', 'off', 'pro', 'cks']
+    float_list = ['packet_loss_percent', 'round_trip_ms_min', 'round_trip_ms_avg', 'round_trip_ms_max',
+                  'round_trip_ms_stddev', 'timestamp', 'time_ms']
+
+    for key in proc_data:
+        if key in int_list:
+            proc_data[key] = jc.utils.convert_to_int(proc_data[key])
+
+        if key in float_list:
+            proc_data[key] = jc.utils.convert_to_float(proc_data[key])
+
+    return proc_data
+
+
+class _state:
+    os_detected = None
+    linux = None
+    bsd = None
+    ipv4 = None
+    hostname = None
+    destination_ip = None
+    sent_bytes = None
+    pattern = None
+    footer = False
+    packets_transmitted = None
+    packets_received = None
+    packet_loss_percent = None
+    time_ms = None
+    duplicates = None
+
+
+def _ipv6_in(line):
+    line_list = line.replace('(', ' ').replace(')', ' ').replace(',', ' ').replace('%', ' ').split()
+    ipv6 = False
+    for item in line_list:
+        try:
+            _ = ipaddress.IPv6Address(item)
+            ipv6 = True
+        except Exception:
+            pass
+    return ipv6
+
+
+def _error_type(line):
+    # from https://github.com/dgibson/iputils/blob/master/ping.c
+    # https://android.googlesource.com/platform/external/ping/+/8fc3c91cf9e7f87bc20b9e6d3ea2982d87b70d9a/ping.c
+    # https://opensource.apple.com/source/network_cmds/network_cmds-328/ping.tproj/ping.c
+    type_map = {
+        'Destination Net Unreachable': 'destination_net_unreachable',
+        'Destination Host Unreachable': 'destination_host_unreachable',
+        'Destination Protocol Unreachable': 'destination_protocol_unreachable',
+        'Destination Port Unreachable': 'destination_port_unreachable',
+        'Frag needed and DF set': 'frag_needed_and_df_set',
+        'Source Route Failed': 'source_route_failed',
+        'Destination Net Unknown': 'destination_net_unknown',
+        'Destination Host Unknown': 'destination_host_unknown',
+        'Source Host Isolated': 'source_host_isolated',
+        'Destination Net Prohibited': 'destination_net_prohibited',
+        'Destination Host Prohibited': 'destination_host_prohibited',
+        'Destination Net Unreachable for Type of Service': 'destination_net_unreachable_for_type_of_service',
+        'Destination Host Unreachable for Type of Service': 'destination_host_unreachable_for_type_of_service',
+        'Packet filtered': 'packet_filtered',
+        'Precedence Violation': 'precedence_violation',
+        'Precedence Cutoff': 'precedence_cutoff',
+        'Dest Unreachable, Bad Code': 'dest_unreachable_bad_code',
+        'Redirect Network': 'redirect_network',
+        'Redirect Host': 'redirect_host',
+        'Redirect Type of Service and Network': 'redirect_type_of_service_and_network',
+        'Redirect, Bad Code': 'redirect_bad_code',
+        'Time to live exceeded': 'time_to_live_exceeded',
+        'Frag reassembly time exceeded': 'frag_reassembly_time_exceeded',
+        'Time exceeded, Bad Code': 'time_exceeded_bad_code'
+    }
+
+    for err_type, code in type_map.items():
+        if err_type in line:
+            return code
+
+    return None
+
+
+def _bsd_parse(line, s):
+    output_line = {}
+
+    if line.startswith('PING '):
+        s.destination_ip = line.split()[2].lstrip('(').rstrip(':').rstrip(')')
+        s.sent_bytes = line.split()[3]
+        return None
+
+    if line.startswith('PING6('):
+        line = line.replace('(', ' ').replace(')', ' ').replace('=', ' ')
+        s.source_ip = line.split()[4]
+        s.destination_ip = line.split()[6]
+        s.sent_bytes = line.split()[1]
+        return None
+
+    if line.startswith('---'):
+        s.footer = True
+        return None
+
+    if s.footer:
+        if 'packets transmitted' in line:
+            if ' duplicates,' in line:
+                s.packets_transmitted = line.split()[0]
+                s.packets_received = line.split()[3]
+                s.packet_loss_percent = line.split()[8].rstrip('%')
+                s.duplicates = line.split()[6].lstrip('+')
+                return None
+
+            s.packets_transmitted = line.split()[0]
+            s.packets_received = line.split()[3]
+            s.packet_loss_percent = line.split()[6].rstrip('%')
+            s.duplicates = '0'
+            return None
+
+        split_line = line.split(' = ')[1]
+        split_line = split_line.split('/')
+
+        output_line = {
+            'type': 'summary',
+            'destination_ip': s.destination_ip or None,
+            'sent_bytes': s.sent_bytes or None,
+            'pattern': s.pattern or None,
+            'packets_transmitted': s.packets_transmitted or None,
+            'packets_received': s.packets_received or None,
+            'packet_loss_percent': s.packet_loss_percent or None,
+            'duplicates': s.duplicates or None,
+            'round_trip_ms_min': split_line[0],
+            'round_trip_ms_avg': split_line[1],
+            'round_trip_ms_max': split_line[2],
+            'round_trip_ms_stddev': split_line[3].replace(' ms', '')
+        }
+
+        return output_line
+
+    # ping response lines
+
+    # ipv4 lines
+    if not _ipv6_in(line):
+
+        # request timeout
+        if line.startswith('Request timeout for '):
+            output_line = {
+                'type': 'timeout',
+                'destination_ip': s.destination_ip or None,
+                'sent_bytes': s.sent_bytes or None,
+                'pattern': s.pattern or None,
+                'icmp_seq': line.split()[4]
+            }
+
+            return output_line
+
+        # catch error responses
+        err = _error_type(line)
+        if err:
+            output_line = {
+                'type': err
+            }
+
+            try:
+                output_line['bytes'] = line.split()[0]
+                output_line['destination_ip'] = s.destination_ip
+                output_line['response_ip'] = line.split()[4].strip(':').strip('(').strip(')')
+            except Exception:
+                pass
+
+            return output_line
+
+        # normal response
+        if ' bytes from ' in line:
+            line = line.replace(':', ' ').replace('=', ' ')
+
+            output_line = {
+                'type': 'reply',
+                'destination_ip': s.destination_ip or None,
+                'sent_bytes': s.sent_bytes or None,
+                'pattern': s.pattern or None,
+                'response_bytes': line.split()[0],
+                'response_ip': line.split()[3],
+                'icmp_seq': line.split()[5],
+                'ttl': line.split()[7],
+                'time_ms': line.split()[9]
+            }
+
+            return output_line
+
+    # ipv6 lines
+    elif ' bytes from ' in line:
+        line = line.replace(',', ' ').replace('=', ' ')
+
+        output_line = {
+            'type': 'reply',
+            'destination_ip': s.destination_ip or None,
+            'sent_bytes': s.sent_bytes or None,
+            'pattern': s.pattern or None,
+            'bytes': line.split()[0],
+            'response_ip': line.split()[3],
+            'icmp_seq': line.split()[5],
+            'ttl': line.split()[7],
+            'time_ms': line.split()[9]
+        }
+
+        return output_line
+
+
+def _linux_parse(line, s):
+    """
+    Linux ping line parsing function.
+
+    Parameters:
+
+        line:        (string)        line of text data to parse
+        s:           (state object)  global state
+
+    Returns:
+
+        Dictionary. Raw structured data.
+    """
+    output_line = {}
+
+    if line.startswith('PING '):
+        s.ipv4 = 'bytes of data' in line
+
+        if s.ipv4 and line[5] not in string.digits:
+            s.hostname = True
+        elif s.ipv4 and line[5] in string.digits:
+            s.hostname = False
+        elif not s.ipv4 and ' (' in line:
+            s.hostname = True
+        else:
+            s.hostname = False
+
+        if s.ipv4 and not s.hostname:
+            dst_ip, dta_byts = (2, 3)
+        elif s.ipv4 and s.hostname:
+            dst_ip, dta_byts = (2, 3)
+        elif not s.ipv4 and not s.hostname:
+            dst_ip, dta_byts = (2, 3)
+        else:
+            dst_ip, dta_byts = (3, 4)
+
+        line = line.replace('(', ' ').replace(')', ' ')
+        s.destination_ip = line.split()[dst_ip].lstrip('(').rstrip(')')
+        s.sent_bytes = line.split()[dta_byts]
+
+        return None
+
+    if line.startswith('---'):
+        s.footer = True
+        return None
+
+    if s.footer:
+        if 'packets transmitted' in line:
+            if ' duplicates,' in line:
+                s.packets_transmitted = line.split()[0]
+                s.packets_received = line.split()[3]
+                s.packet_loss_percent = line.split()[7].rstrip('%')
+                s.duplicates = line.split()[5].lstrip('+')
+                s.time_ms = line.split()[11].replace('ms', '')
+                return None
+
+            s.packets_transmitted = line.split()[0]
+            s.packets_received = line.split()[3]
+            s.packet_loss_percent = line.split()[5].rstrip('%')
+            s.duplicates = '0'
+            s.time_ms = line.split()[9].replace('ms', '')
+            return None
+
+        split_line = line.split(' = ')[1]
+        split_line = split_line.split('/')
+        output_line = {
+            'type': 'summary',
+            'destination_ip': s.destination_ip or None,
+            'sent_bytes': s.sent_bytes or None,
+            'pattern': s.pattern or None,
+            'packets_transmitted': s.packets_transmitted or None,
+            'packets_received': s.packets_received or None,
+            'packet_loss_percent': s.packet_loss_percent or None,
+            'duplicates': s.duplicates or None,
+            'time_ms': s.time_ms or None,
+            'round_trip_ms_min': split_line[0],
+            'round_trip_ms_avg': split_line[1],
+            'round_trip_ms_max': split_line[2],
+            'round_trip_ms_stddev': split_line[3].split()[0]
+        }
+
+        return output_line
+
+    # ping response lines
+
+    # request timeout
+    if 'no answer yet for icmp_seq=' in line:
+        timestamp = False
+        isequence = 5
+
+        # if timestamp option is specified, then shift icmp sequence field right by one
+        if line[0] == '[':
+            timestamp = True
+            isequence = 6
+
+        output_line = {
+            'type': 'timeout',
+            'destination_ip': s.destination_ip or None,
+            'sent_bytes': s.sent_bytes or None,
+            'pattern': s.pattern or None,
+            'timestamp': line.split()[0].lstrip('[').rstrip(']') if timestamp else None,
+            'icmp_seq': line.replace('=', ' ').split()[isequence]
+        }
+
+        return output_line
+
+    # normal responses
+    if ' bytes from ' in line:
+
+        line = line.replace('(', ' ').replace(')', ' ').replace('=', ' ')
+
+        # positions of items depend on whether ipv4/ipv6 and/or ip/hostname is used
+        if s.ipv4 and not s.hostname:
+            bts, rip, iseq, t2l, tms = (0, 3, 5, 7, 9)
+        elif s.ipv4 and s.hostname:
+            bts, rip, iseq, t2l, tms = (0, 4, 7, 9, 11)
+        elif not s.ipv4 and not s.hostname:
+            bts, rip, iseq, t2l, tms = (0, 3, 5, 7, 9)
+        elif not s.ipv4 and s.hostname:
+            bts, rip, iseq, t2l, tms = (0, 4, 7, 9, 11)
+
+        # if timestamp option is specified, then shift everything right by one
+        timestamp = False
+        if line[0] == '[':
+            timestamp = True
+            bts, rip, iseq, t2l, tms = (bts + 1, rip + 1, iseq + 1, t2l + 1, tms + 1)
+
+        output_line = {
+            'type': 'reply',
+            'destination_ip': s.destination_ip or None,
+            'sent_bytes': s.sent_bytes or None,
+            'pattern': s.pattern or None,
+            'timestamp': line.split()[0].lstrip('[').rstrip(']') if timestamp else None,
+            'response_bytes': line.split()[bts],
+            'response_ip': line.split()[rip].rstrip(':'),
+            'icmp_seq': line.split()[iseq],
+            'ttl': line.split()[t2l],
+            'time_ms': line.split()[tms],
+            'duplicate': 'DUP!' in line
+        }
+
+        return output_line
+
+
+def parse(data, raw=False, quiet=False, ignore_exceptions=False):
+    """
+    Main text parsing generator function. Returns an iterator object.
+
+    Parameters:
+
+        data:              (iterable)  line-based text data to parse (e.g. sys.stdin or str.splitlines())
+        raw:               (boolean)   output preprocessed JSON if True
+        quiet:             (boolean)   suppress warning messages if True
+        ignore_exceptions: (boolean)   ignore parsing exceptions if True
+
+    Yields:
+
+        Dictionary. Raw or processed structured data.
+
+    Returns:
+
+        Iterator object
+    """
+    s = _state()
+
+    if not quiet:
+        jc.utils.compatibility(__name__, info.compatible)
+
+    for line in data:
+
+        output_line = {}
+
+        try:
+            # skip blank lines
+            if line.strip() == '':
+                continue
+
+            # skip warning lines
+            if line.startswith('WARNING: '):
+                continue
+
+            # check for PATTERN
+            if line.startswith('PATTERN: '):
+                s.pattern = line.strip().split(': ')[1]
+                continue
+
+            # detect Linux vs. BSD ping
+            if not s.os_detected and line.strip().endswith('bytes of data.'):
+                s.os_detected = True
+                s.linux = True
+
+            elif not s.os_detected and '-->' in line:
+                s.os_detected = True
+                s.bsd = True
+
+            elif not s.os_detected and _ipv6_in(line) and line.strip().endswith('data bytes'):
+                s.os_detected = True
+                s.linux = True
+
+            elif not s.os_detected and not _ipv6_in(line) and line.strip().endswith('data bytes'):
+                s.os_detected = True
+                s.bsd = True
+
+            # parse the data
+            if s.os_detected and s.linux:
+                output_line = _linux_parse(line, s)
+
+            elif s.os_detected and s.bsd:
+                output_line = _bsd_parse(line, s)
+
+            else:
+                raise ParseError('Could not detect ping OS')
+
+            # yield the output line if it has data
+            if output_line:
+                yield stream_success(output_line, ignore_exceptions) if raw else stream_success(_process(output_line), ignore_exceptions)
+            else:
+                continue
+
+        except Exception as e:
+            yield stream_error(e, ignore_exceptions, line)

jc/parsers/route.py

@@ -22,7 +22,7 @@ Schema:
         "genmask":         string,
         "flags":           string,
         "flags_pretty": [
-                           string,
+                           string
         ]
         "metric":          integer,
         "ref":             integer,
@@ -73,7 +73,6 @@ Examples:
       }
     ]
 
-
     $ route -ee | jc --route -p -r
     [
       {
@@ -103,7 +102,6 @@ Examples:
         "irtt": "0"
       }
     ]
-
 """
 import jc.utils
 import jc.parsers.universal

jc/parsers/sfdisk.py

@@ -0,0 +1,407 @@
+"""jc - JSON CLI output utility `sfdisk` command output parser
+
+Supports the following `sfdisk` options:
+- `-l`
+- `-F`
+- `-d`   (deprecated - only for older versions of util-linux)
+- `-uM`  (deprecated - only for older versions of util-linux)
+- `-uC`  (deprecated - only for older versions of util-linux)
+- `-uS`  (deprecated - only for older versions of util-linux)
+- `-uB`  (deprecated - only for older versions of util-linux)
+
+Usage (cli):
+
+    # sfdisk -l | jc --sfdisk
+
+    or
+
+    # jc sfdisk -l
+
+Usage (module):
+
+    import jc.parsers.sfdisk
+    result = jc.parsers.sfdisk.parse(sfdisk_command_output)
+
+Schema:
+
+    [
+      {
+        "disk":                     string,
+        "disk_size":                string,
+        "free_disk_size":           string,
+        "bytes":                    integer,
+        "free_bytes":               integer,
+        "sectors":                  integer,
+        "free_sectors":             integer,
+        "cylinders":                integer,
+        "heads":                    integer,
+        "sectors_per_track":        integer,
+        "units":                    string,
+        "logical_sector_size":      integer,
+        "physical_sector_size":     integer,
+        "min_io_size":              integer,
+        "optimal_io_size":          integer,
+        "disk_label_type":          string,
+        "disk_identifier":          string,
+        "disk_model":               string,
+        "partitions": [
+          {
+            "device":               string,
+            "boot":                 boolean,
+            "start":                integer,
+            "end":                  integer,
+            "size":                 string,    # Note: will be integer when using deprecated -d sfdisk option
+            "cyls":                 integer,
+            "mib":                  integer,
+            "blocks":               integer,
+            "sectors":              integer,
+            "id":                   string,
+            "system":               string,
+            "type":                 string
+          }
+        ]
+      }
+    ]
+
+Examples:
+
+    # sfdisk -l | jc --sfdisk -p
+    [
+      {
+        "disk": "/dev/sda",
+        "cylinders": 2610,
+        "heads": 255,
+        "sectors_per_track": 63,
+        "units": "cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0",
+        "partitions": [
+          {
+            "device": "/dev/sda1",
+            "boot": true,
+            "start": 0,
+            "end": 130,
+            "cyls": 131,
+            "blocks": 1048576,
+            "id": "83",
+            "system": "Linux"
+          },
+          {
+            "device": "/dev/sda2",
+            "boot": false,
+            "start": 130,
+            "end": 2610,
+            "cyls": 2481,
+            "blocks": 19921920,
+            "id": "8e",
+            "system": "Linux LVM"
+          },
+          {
+            "device": "/dev/sda3",
+            "boot": false,
+            "start": 0,
+            "end": null,
+            "cyls": 0,
+            "blocks": 0,
+            "id": "0",
+            "system": "Empty"
+          },
+          {
+            "device": "/dev/sda4",
+            "boot": false,
+            "start": 0,
+            "end": null,
+            "cyls": 0,
+            "blocks": 0,
+            "id": "0",
+            "system": "Empty"
+          }
+        ]
+      },
+      {
+        "disk": "/dev/mapper/centos-root",
+        "cylinders": 2218,
+        "heads": 255,
+        "sectors_per_track": 63
+      },
+      {
+        "disk": "/dev/mapper/centos-swap",
+        "cylinders": 261,
+        "heads": 255,
+        "sectors_per_track": 63
+      }
+    ]
+
+    # sfdisk -l | jc --sfdisk -p -r
+    [
+      {
+        "disk": "/dev/sda",
+        "cylinders": "2610",
+        "heads": "255",
+        "sectors_per_track": "63",
+        "units": "cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0",
+        "partitions": [
+          {
+            "device": "/dev/sda1",
+            "boot": "*",
+            "start": "0+",
+            "end": "130-",
+            "cyls": "131-",
+            "blocks": "1048576",
+            "id": "83",
+            "system": "Linux"
+          },
+          {
+            "device": "/dev/sda2",
+            "boot": null,
+            "start": "130+",
+            "end": "2610-",
+            "cyls": "2481-",
+            "blocks": "19921920",
+            "id": "8e",
+            "system": "Linux LVM"
+          },
+          {
+            "device": "/dev/sda3",
+            "boot": null,
+            "start": "0",
+            "end": "-",
+            "cyls": "0",
+            "blocks": "0",
+            "id": "0",
+            "system": "Empty"
+          },
+          {
+            "device": "/dev/sda4",
+            "boot": null,
+            "start": "0",
+            "end": "-",
+            "cyls": "0",
+            "blocks": "0",
+            "id": "0",
+            "system": "Empty"
+          }
+        ]
+      },
+      {
+        "disk": "/dev/mapper/centos-root",
+        "cylinders": "2218",
+        "heads": "255",
+        "sectors_per_track": "63"
+      },
+      {
+        "disk": "/dev/mapper/centos-swap",
+        "cylinders": "261",
+        "heads": "255",
+        "sectors_per_track": "63"
+      }
+    ]
+"""
+import jc.utils
+import jc.parsers.universal
+
+
+class info():
+    """Provides parser metadata (version, author, etc.)"""
+    version = '1.1'
+    description = '`sfdisk` command parser'
+    author = 'Kelly Brazil'
+    author_email = 'kellyjonbrazil@gmail.com'
+    compatible = ['linux']
+    magic_commands = ['sfdisk']
+
+
+__version__ = info.version
+
+
+def _process(proc_data):
+    """
+    Final processing to conform to the schema.
+
+    Parameters:
+
+        proc_data:   (List of Dictionaries) raw structured data to process
+
+    Returns:
+
+        List of Dictionaries. Structured to conform to the schema.
+    """
+    int_list = ['cylinders', 'heads', 'sectors_per_track', 'start', 'end', 'cyls', 'mib',
+                'blocks', 'sectors', 'bytes', 'logical_sector_size', 'physical_sector_size',
+                'min_io_size', 'optimal_io_size', 'free_bytes', 'free_sectors']
+    bool_list = ['boot']
+
+    for entry in proc_data:
+        for key in entry:
+            if key in int_list:
+                entry[key] = jc.utils.convert_to_int(entry[key].replace('-', ''))
+
+        if 'partitions' in entry:
+            for p in entry['partitions']:
+                for key in p:
+                    # legacy conversion for -d option
+                    if key == 'size':
+                        if p[key].isnumeric():
+                            p[key] = jc.utils.convert_to_int(p[key])
+
+                    # normal conversions
+                    if key in int_list:
+                        p[key] = jc.utils.convert_to_int(p[key].replace('-', ''))
+                    if key in bool_list:
+                        p[key] = jc.utils.convert_to_bool(p[key])
+
+    return proc_data
+
+
+def parse(data, raw=False, quiet=False):
+    """
+    Main text parsing function
+
+    Parameters:
+
+        data:        (string)  text data to parse
+        raw:         (boolean) output preprocessed JSON if True
+        quiet:       (boolean) suppress warning messages if True
+
+    Returns:
+
+        List of Dictionaries. Raw or processed structured data.
+    """
+    if not quiet:
+        jc.utils.compatibility(__name__, info.compatible)
+
+    raw_output = []
+    item = {}
+    partitions = []
+    option = ''
+    section = ''
+
+    if jc.utils.has_data(data):
+
+        for line in data.splitlines():
+            # deprecated - only for older versions of util-linux
+            if line.startswith('# partition table of'):
+                if item:
+                    raw_output.append(item)
+
+                item = {}
+                partitions = []
+                option = 'd'
+                item['disk'] = line.split()[4]
+                continue
+
+            # deprecated - only for older versions of util-linux
+            if option == 'd':
+                if line.startswith('unit: '):
+                    item['units'] = line.split()[1]
+                    section = 'partitions'
+                    continue
+
+                if section == 'partitions' and line:
+                    part = {}
+                    part['device'] = line.split()[0]
+                    line = line.replace(',', ' ').replace('=', ' ')
+                    part['start'] = line.split()[3]
+                    part['size'] = line.split()[5]
+                    part['id'] = line.split()[7]
+                    part['boot'] = '*' if 'bootable' in line else None
+                    partitions.append(part)
+                    item['partitions'] = partitions
+                    continue
+
+            else:
+                # older versions of util-linux
+                # Disk /dev/sda: 2610 cylinders, 255 heads, 63 sectors/track
+                if line.startswith('Disk ') and 'sectors/track' in line:
+                    if item:
+                        raw_output.append(item)
+
+                    item = {}
+                    partitions = []
+                    line = line.replace(':', '').replace(',', '')
+                    fields = line.split()
+                    item['disk'] = fields[1]
+                    item['cylinders'] = fields[2]
+                    item['heads'] = fields[4]
+                    item['sectors_per_track'] = fields[6]
+                    continue
+
+                # util-linux v2.32.0+ (?)
+                # Disk /dev/sda: 20 GiB, 21474836480 bytes, 41943040 sectors
+                if line.startswith('Disk ') and line.endswith('sectors'):
+                    if item:
+                        raw_output.append(item)
+
+                    item = {}
+                    partitions = []
+                    line = line.replace(':', '').replace(',', '')
+                    fields = line.split()
+                    item['disk'] = fields[1]
+                    item['disk_size'] = ' '.join(fields[2:4])
+                    item['bytes'] = fields[4]
+                    item['sectors'] = fields[6]
+                    continue
+
+                if line.startswith('Disk model: '):
+                    item['disk_model'] = line.split(':', maxsplit=1)[1].strip()
+                    continue
+
+                if line.startswith('Sector size (logical/physical)'):
+                    fields = line.split()
+                    item['logical_sector_size'] = fields[3]
+                    item['physical_sector_size'] = fields[6]
+                    continue
+
+                if line.startswith('I/O size (minimum/optimal)'):
+                    fields = line.split()
+                    item['min_io_size'] = fields[3]
+                    item['optimal_io_size'] = fields[6]
+                    continue
+
+                if line.startswith('Disklabel type'):
+                    item['disk_label_type'] = line.split(':', maxsplit=1)[1].strip()
+                    continue
+
+                if line.startswith('Disk identifier'):
+                    item['disk_identifier'] = line.split(':', maxsplit=1)[1].strip()
+                    continue
+
+                if line.startswith('Units: '):
+                    item['units'] = line.split(':')[1].strip()
+                    continue
+
+                # sfdisk -F
+                if line.startswith('Unpartitioned space'):
+                    line = line.replace(':', '').replace(',', '')
+                    fields = line.split()
+                    item['disk'] = fields[2]
+                    item['free_disk_size'] = ' '.join(fields[3:5])
+                    item['free_bytes'] = fields[5]
+                    item['free_sectors'] = fields[7]
+                    continue
+
+                # partition lines
+                if 'Start' in line and 'End' in line and ('Sectors' in line or 'Device' in line):
+                    section = 'partitions'
+                    partitions.append(line.lower().replace('#', ' '))
+                    continue
+
+                if section == 'partitions' and line:
+                    partitions.append(line)
+                    continue
+
+                if section == 'partitions' and line == '':
+                    item['partitions'] = jc.parsers.universal.sparse_table_parse(partitions)
+                    section = ''
+                    partitions = []
+                    continue
+
+        # get final partitions if there are any left over
+        if section == 'partitions' and option != 'd' and partitions:
+            item['partitions'] = jc.parsers.universal.sparse_table_parse(partitions)
+
+        if item:
+            raw_output.append(item)
+
+    if raw:
+        return raw_output
+    else:
+        return _process(raw_output)

jc/parsers/stat.py

@@ -169,7 +169,7 @@ import jc.utils
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.8'
+    version = '1.9'
     description = '`stat` command parser'
     author = 'Kelly Brazil'
     author_email = 'kellyjonbrazil@gmail.com'
@@ -322,7 +322,7 @@ def parse(data, raw=False, quiet=False):
             for line in cleandata:
                 value = shlex.split(line)
                 output_line = {
-                    'file': value[15],
+                    'file': ' '.join(value[15:]),
                     'unix_device': value[0],
                     'inode': value[1],
                     'flags': value[2],

jc/parsers/traceroute.py

@@ -389,7 +389,7 @@ def parse(data, raw=False, quiet=False):
 
             # print warning to STDERR
             if not quiet:
-                jc.utils.warning_message('No header row found. For destination info redirect STDERR to STDOUT')
+                jc.utils.warning_message(['No header row found. For destination info redirect STDERR to STDOUT'])
 
         data = '\n'.join(new_data)
 

jc/parsers/uname.py

@@ -48,7 +48,7 @@ from jc.exceptions import ParseError
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.5'
+    version = '1.6'
     description = '`uname -a` command parser'
     author = 'Kelly Brazil'
     author_email = 'kellyjonbrazil@gmail.com'
@@ -98,8 +98,8 @@ def parse(data, raw=False, quiet=False):
 
     if jc.utils.has_data(data):
 
-        # check for OSX output
-        if data.startswith('Darwin'):
+        # check for macOS or FreeBSD output
+        if data.startswith('Darwin') or data.startswith('FreeBSD'):
             parsed_line = data.split()
 
             if len(parsed_line) < 5:
@@ -113,6 +113,19 @@ def parse(data, raw=False, quiet=False):
 
         # otherwise use linux parser
         else:
+            # fixup for cases where 'machine' exists but 'processor' and 'hardware_platform' fields are blank.
+            # if the fields exist then at least two of the three will be the same.
+            # if the fields do not exist then none of the fields in those positions will be the same.
+            # case of only two existing is undefined. Must either be one or all three existing, otherwise
+            # there will be unexpected results during parsing.
+            fixup = data.split()
+            if len(fixup) >= 4:
+                fixup_set = set([fixup[-2], fixup[-3], fixup[-4]])
+                if len(fixup_set) > 2:
+                    fixup.insert(-1, 'unknown')
+                    fixup.insert(-1, 'unknown')
+                    data = ' '.join(fixup)
+            
             parsed_line = data.split(maxsplit=3)
 
             if len(parsed_line) < 3:
@@ -125,8 +138,8 @@ def parse(data, raw=False, quiet=False):
             parsed_line = parsed_line[-1].rsplit(maxsplit=4)
 
             raw_output['operating_system'] = parsed_line.pop(-1)
-            raw_output['hardware_platform'] = parsed_line.pop(-1)
             raw_output['processor'] = parsed_line.pop(-1)
+            raw_output['hardware_platform'] = parsed_line.pop(-1)
             raw_output['machine'] = parsed_line.pop(-1)
 
             raw_output['kernel_version'] = parsed_line.pop(0)

jc/parsers/vmstat.py

@@ -0,0 +1,279 @@
+"""jc - JSON CLI output utility `vmstat` command output parser
+
+Options supported: `-a`, `-w`, `-d`, `-t`
+
+The `epoch` calculated timestamp field is naive (i.e. based on the local time of the system the parser is run on)
+
+The `epoch_utc` calculated timestamp field is timezone-aware and is only available if the timezone field is UTC.
+
+Usage (cli):
+
+    $ vmstat | jc --vmstat
+
+    or
+
+    $ jc vmstat
+
+Usage (module):
+
+    import jc.parsers.vmstat
+    result = jc.parsers.vmstat.parse(vmstat_command_output)
+
+Schema:
+
+    [
+      {
+        "runnable_procs":                    integer,
+        "uninterruptible_sleeping_procs":    integer,
+        "virtual_mem_used":                  integer,
+        "free_mem":                          integer,
+        "buffer_mem":                        integer,
+        "cache_mem":                         integer,
+        "inactive_mem":                      integer,
+        "active_mem":                        integer,
+        "swap_in":                           integer,
+        "swap_out":                          integer,
+        "blocks_in":                         integer,
+        "blocks_out":                        integer,
+        "interrupts":                        integer,
+        "context_switches":                  integer,
+        "user_time":                         integer,
+        "system_time":                       integer,
+        "idle_time":                         integer,
+        "io_wait_time":                      integer,
+        "stolen_time":                       integer,
+        "disk":                              string,
+        "total_reads":                       integer,
+        "merged_reads":                      integer,
+        "sectors_read":                      integer,
+        "reading_ms":                        integer,
+        "total_writes":                      integer,
+        "merged_writes":                     integer,
+        "sectors_written":                   integer,
+        "writing_ms":                        integer,
+        "current_io":                        integer,
+        "io_seconds":                        integer,
+        "timestamp":                         string,
+        "timezone":                          string,
+        "epoch":                             integer,     # naive timestamp if -t flag is used
+        "epoch_utc":                         integer      # aware timestamp if -t flag is used and UTC TZ
+      }
+    ]
+
+Examples:
+
+    $ vmstat | jc --vmstat -p
+    [
+      {
+        "runnable_procs": 2,
+        "uninterruptible_sleeping_procs": 0,
+        "virtual_mem_used": 0,
+        "free_mem": 2794468,
+        "buffer_mem": 2108,
+        "cache_mem": 741208,
+        "inactive_mem": null,
+        "active_mem": null,
+        "swap_in": 0,
+        "swap_out": 0,
+        "blocks_in": 1,
+        "blocks_out": 3,
+        "interrupts": 29,
+        "context_switches": 57,
+        "user_time": 0,
+        "system_time": 0,
+        "idle_time": 99,
+        "io_wait_time": 0,
+        "stolen_time": 0,
+        "timestamp": null,
+        "timezone": null
+      }
+    ]
+
+    $ vmstat | jc --vmstat -p -r
+    [
+      {
+        "runnable_procs": "2",
+        "uninterruptible_sleeping_procs": "0",
+        "virtual_mem_used": "0",
+        "free_mem": "2794468",
+        "buffer_mem": "2108",
+        "cache_mem": "741208",
+        "inactive_mem": null,
+        "active_mem": null,
+        "swap_in": "0",
+        "swap_out": "0",
+        "blocks_in": "1",
+        "blocks_out": "3",
+        "interrupts": "29",
+        "context_switches": "57",
+        "user_time": "0",
+        "system_time": "0",
+        "idle_time": "99",
+        "io_wait_time": "0",
+        "stolen_time": "0",
+        "timestamp": null,
+        "timezone": null
+      }
+    ]
+"""
+import jc.utils
+
+
+class info():
+    """Provides parser metadata (version, author, etc.)"""
+    version = '1.0'
+    description = '`vmstat` command parser'
+    author = 'Kelly Brazil'
+    author_email = 'kellyjonbrazil@gmail.com'
+    # details = 'enter any other details here'
+
+    # compatible options: linux, darwin, cygwin, win32, aix, freebsd
+    compatible = ['linux']
+    magic_commands = ['vmstat']
+
+
+__version__ = info.version
+
+
+def _process(proc_data):
+    """
+    Final processing to conform to the schema.
+
+    Parameters:
+
+        proc_data:   (List of Dictionaries) raw structured data to process
+
+    Returns:
+
+        List of Dictionaries. Structured to conform to the schema.
+    """
+
+    int_list = ['runnable_procs', 'uninterruptible_sleeping_procs', 'virtual_mem_used', 'free_mem', 'buffer_mem',
+                'cache_mem', 'inactive_mem', 'active_mem', 'swap_in', 'swap_out', 'blocks_in', 'blocks_out',
+                'interrupts', 'context_switches', 'user_time', 'system_time', 'idle_time', 'io_wait_time',
+                'stolen_time', 'total_reads', 'merged_reads', 'sectors_read', 'reading_ms', 'total_writes',
+                'merged_writes', 'sectors_written', 'writing_ms', 'current_io', 'io_seconds']
+
+    for entry in proc_data:
+        for key in entry:
+            if key in int_list:
+                entry[key] = jc.utils.convert_to_int(entry[key])
+
+        if entry['timestamp']:
+            ts = jc.utils.timestamp(f'{entry["timestamp"]} {entry["timezone"]}')
+            entry['epoch'] = ts.naive
+            entry['epoch_utc'] = ts.utc
+
+    return proc_data
+
+
+def parse(data, raw=False, quiet=False):
+    """
+    Main text parsing function
+
+    Parameters:
+
+        data:        (string)  text data to parse
+        raw:         (boolean) output preprocessed JSON if True
+        quiet:       (boolean) suppress warning messages if True
+
+    Returns:
+
+        List of Dictionaries. Raw or processed structured data.
+    """
+    if not quiet:
+        jc.utils.compatibility(__name__, info.compatible)
+
+    raw_output = []
+    output_line = {}
+    procs = None
+    buff_cache = None
+    disk = None
+    tstamp = None
+    tz = None
+
+    if jc.utils.has_data(data):
+
+        for line in filter(None, data.splitlines()):
+
+            # detect output type
+            if not procs and not disk and line.startswith('procs'):
+                procs = True
+                tstamp = '-timestamp-' in line
+                continue
+
+            if not procs and not disk and line.startswith('disk'):
+                disk = True
+                tstamp = '-timestamp-' in line
+                continue
+
+            # skip header rows
+            if (procs or disk) and (line.startswith('procs') or line.startswith('disk')):
+                continue
+
+            if 'swpd' in line and 'free' in line and 'buff' in line and 'cache' in line:
+                buff_cache = True
+                tz = line.strip().split()[-1] if tstamp else None
+                continue
+
+            if 'swpd' in line and 'free' in line and 'inact' in line and 'active' in line:
+                buff_cache = False
+                tz = line.strip().split()[-1] if tstamp else None
+                continue
+
+            if 'total' in line and 'merged' in line and 'sectors' in line:
+                tz = line.strip().split()[-1] if tstamp else None
+                continue
+
+            # line parsing
+            if procs:
+                line_list = line.strip().split(maxsplit=17)
+
+                output_line = {
+                    'runnable_procs': line_list[0],
+                    'uninterruptible_sleeping_procs': line_list[1],
+                    'virtual_mem_used': line_list[2],
+                    'free_mem': line_list[3],
+                    'buffer_mem': line_list[4] if buff_cache else None,
+                    'cache_mem': line_list[5] if buff_cache else None,
+                    'inactive_mem': line_list[4] if not buff_cache else None,
+                    'active_mem': line_list[5] if not buff_cache else None,
+                    'swap_in': line_list[6],
+                    'swap_out': line_list[7],
+                    'blocks_in': line_list[8],
+                    'blocks_out': line_list[9],
+                    'interrupts': line_list[10],
+                    'context_switches': line_list[11],
+                    'user_time': line_list[12],
+                    'system_time': line_list[13],
+                    'idle_time': line_list[14],
+                    'io_wait_time': line_list[15],
+                    'stolen_time': line_list[16],
+                    'timestamp': line_list[17] if tstamp else None,
+                    'timezone': tz or None
+                }
+
+                raw_output.append(output_line)
+
+            if disk:
+                line_list = line.strip().split(maxsplit=11)
+
+                output_line = {
+                    'disk': line_list[0],
+                    'total_reads': line_list[1],
+                    'merged_reads': line_list[2],
+                    'sectors_read': line_list[3],
+                    'reading_ms': line_list[4],
+                    'total_writes': line_list[5],
+                    'merged_writes': line_list[6],
+                    'sectors_written': line_list[7],
+                    'writing_ms': line_list[8],
+                    'current_io': line_list[9],
+                    'io_seconds': line_list[10],
+                    'timestamp': line_list[11] if tstamp else None,
+                    'timezone': tz or None
+                }
+
+                raw_output.append(output_line)
+
+    return raw_output if raw else _process(raw_output)

jc/parsers/vmstat_s.py

@@ -0,0 +1,246 @@
+"""jc - JSON CLI output utility `vmstat` command output streaming parser
+
+> This streaming parser outputs JSON Lines
+
+Options supported: `-a`, `-w`, `-d`, `-t`
+
+The `epoch` calculated timestamp field is naive (i.e. based on the local time of the system the parser is run on)
+
+The `epoch_utc` calculated timestamp field is timezone-aware and is only available if the timezone field is UTC.
+
+Usage (cli):
+
+    $ vmstat | jc --vmstat-s
+
+> Note: When piping `jc` converted `vmstat` output to other processes it may appear the output is hanging due to the OS pipe buffers. This is because `vmstat` output is too small to quickly fill up the buffer. Use the `-u` option to unbuffer the `jc` output if you would like immediate output. See the [readme](https://github.com/kellyjonbrazil/jc/tree/master#unbuffering-output) for more information.
+
+Usage (module):
+
+    import jc.parsers.vmstat_s
+    result = jc.parsers.vmstat_s.parse(vmstat_command_output.splitlines())    # result is an iterable object
+    for item in result:
+        # do something
+
+Schema:
+
+    {
+      "runnable_procs":                    integer,
+      "uninterruptible_sleeping_procs":    integer,
+      "virtual_mem_used":                  integer,
+      "free_mem":                          integer,
+      "buffer_mem":                        integer,
+      "cache_mem":                         integer,
+      "inactive_mem":                      integer,
+      "active_mem":                        integer,
+      "swap_in":                           integer,
+      "swap_out":                          integer,
+      "blocks_in":                         integer,
+      "blocks_out":                        integer,
+      "interrupts":                        integer,
+      "context_switches":                  integer,
+      "user_time":                         integer,
+      "system_time":                       integer,
+      "idle_time":                         integer,
+      "io_wait_time":                      integer,
+      "stolen_time":                       integer,
+      "disk":                              string,
+      "total_reads":                       integer,
+      "merged_reads":                      integer,
+      "sectors_read":                      integer,
+      "reading_ms":                        integer,
+      "total_writes":                      integer,
+      "merged_writes":                     integer,
+      "sectors_written":                   integer,
+      "writing_ms":                        integer,
+      "current_io":                        integer,
+      "io_seconds":                        integer,
+      "timestamp":                         string,
+      "timezone":                          string,
+      "epoch":                             integer,     # naive timestamp if -t flag is used
+      "epoch_utc":                         integer      # aware timestamp if -t flag is used and UTC TZ
+      "_jc_meta":                                       # This object only exists if using -qq or ignore_exceptions=True
+        {
+          "success":                       booean,      # true if successfully parsed, false if error
+          "error":                         string,      # exists if "success" is false
+          "line":                          string       # exists if "success" is false
+        }
+    }
+
+Examples:
+
+    $ vmstat | jc --vmstat-s
+    {"runnable_procs":2,"uninterruptible_sleeping_procs":0,"virtual_mem_used":0,"free_mem":2794468,"buffer_mem":2108,"cache_mem":741208,"inactive_mem":null,"active_mem":null,"swap_in":0,"swap_out":0,"blocks_in":1,"blocks_out":3,"interrupts":29,"context_switches":57,"user_time":0,"system_time":0,"idle_time":99,"io_wait_time":0,"stolen_time":0,"timestamp":null,"timezone":null}
+    ...
+
+    $ vmstat | jc --vmstat-s -r
+    {"runnable_procs":"2","uninterruptible_sleeping_procs":"0","virtual_mem_used":"0","free_mem":"2794468","buffer_mem":"2108","cache_mem":"741208","inactive_mem":null,"active_mem":null,"swap_in":"0","swap_out":"0","blocks_in":"1","blocks_out":"3","interrupts":"29","context_switches":"57","user_time":"0","system_time":"0","idle_time":"99","io_wait_time":"0","stolen_time":"0","timestamp":null,"timezone":null}
+    ...
+"""
+import jc.utils
+from jc.utils import stream_success, stream_error
+from jc.exceptions import ParseError
+
+
+class info():
+    """Provides parser metadata (version, author, etc.)"""
+    version = '0.5'
+    description = '`vmstat` command streaming parser'
+    author = 'Kelly Brazil'
+    author_email = 'kellyjonbrazil@gmail.com'
+
+    # compatible options: linux, darwin, cygwin, win32, aix, freebsd
+    compatible = ['linux']
+    streaming = True
+
+
+__version__ = info.version
+
+
+def _process(proc_data):
+    """
+    Final processing to conform to the schema.
+
+    Parameters:
+
+        proc_data:   (Dictionary) raw structured data to process
+
+    Returns:
+
+        Dictionary. Structured data to conform to the schema.
+    """
+    int_list = ['runnable_procs', 'uninterruptible_sleeping_procs', 'virtual_mem_used', 'free_mem', 'buffer_mem',
+                'cache_mem', 'inactive_mem', 'active_mem', 'swap_in', 'swap_out', 'blocks_in', 'blocks_out',
+                'interrupts', 'context_switches', 'user_time', 'system_time', 'idle_time', 'io_wait_time',
+                'stolen_time', 'total_reads', 'merged_reads', 'sectors_read', 'reading_ms', 'total_writes',
+                'merged_writes', 'sectors_written', 'writing_ms', 'current_io', 'io_seconds']
+
+    for key in proc_data:
+        if key in int_list:
+            proc_data[key] = jc.utils.convert_to_int(proc_data[key])
+
+    if proc_data['timestamp']:
+        ts = jc.utils.timestamp(f'{proc_data["timestamp"]} {proc_data["timezone"]}')
+        proc_data['epoch'] = ts.naive
+        proc_data['epoch_utc'] = ts.utc
+
+    return proc_data
+
+
+def parse(data, raw=False, quiet=False, ignore_exceptions=False):
+    """
+    Main text parsing generator function. Returns an iterator object.
+
+    Parameters:
+
+        data:              (iterable)  line-based text data to parse (e.g. sys.stdin or str.splitlines())
+        raw:               (boolean)   output preprocessed JSON if True
+        quiet:             (boolean)   suppress warning messages if True
+        ignore_exceptions: (boolean)   ignore parsing exceptions if True
+
+    Yields:
+
+        Dictionary. Raw or processed structured data.
+
+    Returns:
+
+        Iterator object
+    """
+    if not quiet:
+        jc.utils.compatibility(__name__, info.compatible)
+
+    output_line = {}
+    procs = None
+    buff_cache = None
+    disk = None
+    tstamp = None
+    tz = None
+
+    for line in data:
+        try:
+            # skip blank lines
+            if line.strip() == '':
+                continue
+
+            # detect output type
+            if not procs and not disk and line.startswith('procs'):
+                procs = True
+                tstamp = '-timestamp-' in line
+                continue
+
+            if not procs and not disk and line.startswith('disk'):
+                disk = True
+                tstamp = '-timestamp-' in line
+                continue
+
+            # skip header rows
+            if (procs or disk) and (line.startswith('procs') or line.startswith('disk')):
+                continue
+
+            if 'swpd' in line and 'free' in line and 'buff' in line and 'cache' in line:
+                buff_cache = True
+                tz = line.strip().split()[-1] if tstamp else None
+                continue
+
+            if 'swpd' in line and 'free' in line and 'inact' in line and 'active' in line:
+                buff_cache = False
+                tz = line.strip().split()[-1] if tstamp else None
+                continue
+
+            if 'total' in line and 'merged' in line and 'sectors' in line:
+                tz = line.strip().split()[-1] if tstamp else None
+                continue
+
+            # line parsing
+            if procs:
+                line_list = line.strip().split(maxsplit=17)
+
+                output_line = {
+                    'runnable_procs': line_list[0],
+                    'uninterruptible_sleeping_procs': line_list[1],
+                    'virtual_mem_used': line_list[2],
+                    'free_mem': line_list[3],
+                    'buffer_mem': line_list[4] if buff_cache else None,
+                    'cache_mem': line_list[5] if buff_cache else None,
+                    'inactive_mem': line_list[4] if not buff_cache else None,
+                    'active_mem': line_list[5] if not buff_cache else None,
+                    'swap_in': line_list[6],
+                    'swap_out': line_list[7],
+                    'blocks_in': line_list[8],
+                    'blocks_out': line_list[9],
+                    'interrupts': line_list[10],
+                    'context_switches': line_list[11],
+                    'user_time': line_list[12],
+                    'system_time': line_list[13],
+                    'idle_time': line_list[14],
+                    'io_wait_time': line_list[15],
+                    'stolen_time': line_list[16],
+                    'timestamp': line_list[17] if tstamp else None,
+                    'timezone': tz or None
+                }
+
+            if disk:
+                line_list = line.strip().split(maxsplit=11)
+
+                output_line = {
+                    'disk': line_list[0],
+                    'total_reads': line_list[1],
+                    'merged_reads': line_list[2],
+                    'sectors_read': line_list[3],
+                    'reading_ms': line_list[4],
+                    'total_writes': line_list[5],
+                    'merged_writes': line_list[6],
+                    'sectors_written': line_list[7],
+                    'writing_ms': line_list[8],
+                    'current_io': line_list[9],
+                    'io_seconds': line_list[10],
+                    'timestamp': line_list[11] if tstamp else None,
+                    'timezone': tz or None
+                }
+
+            if output_line:
+                yield stream_success(output_line, ignore_exceptions) if raw else stream_success(_process(output_line), ignore_exceptions)
+            else:
+                raise ParseError('Not vmstat data')
+
+        except Exception as e:
+            yield stream_error(e, ignore_exceptions, line)

jc/parsers/xml.py

@@ -65,19 +65,13 @@ Examples:
       ...
     }
 """
-import sys
 import jc.utils
-# check if xml library is installed and fail gracefully if it is not
-try:
-    import xmltodict
-except Exception:
-    jc.utils.error_message('The xmltodict library is not installed.')
-    sys.exit(1)
+from jc.exceptions import LibraryNotInstalled
 
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.4'
+    version = '1.5'
     description = 'XML file parser'
     author = 'Kelly Brazil'
     author_email = 'kellyjonbrazil@gmail.com'
@@ -121,6 +115,12 @@ def parse(data, raw=False, quiet=False):
 
         Dictionary. Raw or processed structured data.
     """
+    # check if xml library is installed and fail gracefully if it is not
+    try:
+        import xmltodict
+    except Exception:
+        raise LibraryNotInstalled('The xmltodict library is not installed.')
+
     if not quiet:
         jc.utils.compatibility(__name__, info.compatible)
 

jc/parsers/yaml.py

@@ -79,19 +79,13 @@ Examples:
       }
     ]
 """
-import sys
 import jc.utils
-# check if yaml library is installed and fail gracefully if it is not
-try:
-    from ruamel.yaml import YAML
-except Exception:
-    jc.utils.error_message('The ruamel.yaml library is not installed.')
-    sys.exit(1)
+from jc.exceptions import LibraryNotInstalled
 
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.4'
+    version = '1.5'
     description = 'YAML file parser'
     author = 'Kelly Brazil'
     author_email = 'kellyjonbrazil@gmail.com'
@@ -135,6 +129,12 @@ def parse(data, raw=False, quiet=False):
 
         List of Dictionaries representing the YAML documents.
     """
+    # check if yaml library is installed and fail gracefully if it is not
+    try:
+        from ruamel.yaml import YAML
+    except Exception:
+        raise LibraryNotInstalled('The ruamel.yaml library is not installed.')
+
     if not quiet:
         jc.utils.compatibility(__name__, info.compatible)
 

jc/utils.py

@@ -2,41 +2,77 @@
 import sys
 import re
 import locale
+import shutil
 from datetime import datetime, timezone
+from textwrap import TextWrapper
 
 
-def warning_message(message):
+def warning_message(message_lines):
     """
-    Prints a warning message for non-fatal issues
+    Prints warning message for non-fatal issues. The first line is prepended with
+    'jc:  Warning - ' and subsequent lines are indented. Wraps text as needed based
+    on the terminal width.
 
     Parameters:
 
-        message:        (string) text of message
+        message:   (list) list of string lines
 
     Returns:
 
         None - just prints output to STDERR
     """
+    # this is for backwards compatibility with existing custom parsers
+    if isinstance(message_lines, str):
+        message_lines = [message_lines]
 
-    error_string = f'jc:  Warning - {message}'
-    print(error_string, file=sys.stderr)
+    columns = shutil.get_terminal_size().columns
+
+    first_wrapper = TextWrapper(width=columns, subsequent_indent=' ' * 15)
+    next_wrapper = TextWrapper(width=columns, initial_indent=' ' * 15,
+                               subsequent_indent=' ' * 19)
+
+    first_line = message_lines.pop(0)
+    first_str = f'jc:  Warning - {first_line}'
+    first_str = first_wrapper.fill(first_str)
+    print(first_str, file=sys.stderr)
+
+    for line in message_lines:
+        if line == '':
+            continue
+        message = next_wrapper.fill(line)
+        print(message, file=sys.stderr)
 
 
-def error_message(message):
+def error_message(message_lines):
     """
-    Prints an error message for fatal issues
+    Prints an error message for fatal issues. The first line is prepended with
+    'jc:  Error - ' and subsequent lines are indented. Wraps text as needed based
+    on the terminal width.
 
     Parameters:
 
-        message:        (string) text of message
+        message:   (list) list of string lines
 
     Returns:
 
         None - just prints output to STDERR
     """
+    columns = shutil.get_terminal_size().columns
 
-    error_string = f'jc:  Error - {message}'
-    print(error_string, file=sys.stderr)
+    first_wrapper = TextWrapper(width=columns, subsequent_indent=' ' * 13)
+    next_wrapper = TextWrapper(width=columns, initial_indent=' ' * 13,
+                               subsequent_indent=' ' * 17)
+
+    first_line = message_lines.pop(0)
+    first_str = f'jc:  Error - {first_line}'
+    first_str = first_wrapper.fill(first_str)
+    print(first_str, file=sys.stderr)
+
+    for line in message_lines:
+        if line == '':
+            continue
+        message = next_wrapper.fill(line)
+        print(message, file=sys.stderr)
 
 
 def compatibility(mod_name, compatible):
@@ -64,8 +100,8 @@ def compatibility(mod_name, compatible):
     if not platform_found:
         mod = mod_name.split('.')[-1]
         compat_list = ', '.join(compatible)
-        warning_message(f'{mod} parser not compatible with your OS ({sys.platform}).\n'
-                        f'               Compatible platforms: {compat_list}')
+        warning_message([f'{mod} parser not compatible with your OS ({sys.platform}).',
+                         f'Compatible platforms: {compat_list}'])
 
 
 def has_data(data):
@@ -80,7 +116,7 @@ def has_data(data):
 
         Boolean      True if input string (data) contains non-whitespace characters, otherwise False
     """
-    return True if data and not data.isspace() else False
+    return bool(data and not data.isspace())
 
 
 def convert_to_int(value):
@@ -154,7 +190,7 @@ def convert_to_bool(value):
     #   if float converts, then bool the result
     #   if float does not convert then look for truthy string and bool True
     #   else False
-    truthy = ['y', 'yes', 'true']
+    truthy = ['y', 'yes', 'true', '*']
 
     if isinstance(value, (int, float)):
         return bool(value)
@@ -168,11 +204,37 @@ def convert_to_bool(value):
             pass
 
         if value:
-            return True if value.lower() in truthy else False
+            return value.lower() in truthy
 
     return False
 
 
+def stream_success(output_line, ignore_exceptions):
+    """Add `_jc_meta` object to output line if `ignore_exceptions=True`"""
+    if ignore_exceptions:
+        output_line.update({'_jc_meta': {'success': True}})
+
+    return output_line
+
+
+def stream_error(e, ignore_exceptions, line):
+    """Reraise the stream exception with annotation or print an error `_jc_meta`
+       field if `ignore_exceptions=True`
+    """
+    if not ignore_exceptions:
+        e.args = (str(e) + '... Use the ignore_exceptions option (-qq) to ignore streaming parser errors.',)
+        raise e
+
+    return {
+        '_jc_meta':
+            {
+                'success': False,
+                'error': f'{e.__class__.__name__}: {e}',
+                'line': line.strip()
+            }
+    }
+
+
 class timestamp:
     """
     Input a date-time text string of several formats and convert to a naive or timezone-aware epoch timestamp in UTC
@@ -184,7 +246,7 @@ class timestamp:
     Attributes:
 
         string              (str)   the input datetime string
-        format              (int)   the format rule that was used to decode the datetime string
+        format              (int)   the format rule that was used to decode the datetime string. None if conversion fails
         naive               (int)   timestamp based on locally configured timezone. None if conversion fails
         utc                 (int)   aware timestamp only if UTC timezone detected in datetime string. None if conversion fails
     """
@@ -245,10 +307,8 @@ class timestamp:
         if 'UTC' in data:
             utc_tz = True
             if 'UTC+' in data or 'UTC-' in data:
-                if 'UTC+0000' in data or 'UTC-0000' in data:
-                    utc_tz = True
-                else:
-                    utc_tz = False
+                utc_tz = bool('UTC+0000' in data or 'UTC-0000' in data)
+
         elif '+0000' in data or '-0000' in data:
             utc_tz = True
 
@@ -267,6 +327,8 @@ class timestamp:
             {'id': 7000, 'format': '%a %b %d %H:%M:%S %Z %Y', 'locale': None},  # C locale format (found in date cli): Wed Mar 24 11:11:30 UTC 2021
             {'id': 7100, 'format': '%b %d %H:%M:%S %Y', 'locale': None},  # C locale format (found in stat cli output - osx): # Mar 29 11:49:05 2021
             {'id': 7200, 'format': '%Y-%m-%d %H:%M:%S.%f %z', 'locale': None},  # C locale format (found in stat cli output - linux): 2019-08-13 18:13:43.555604315 -0400
+            {'id': 7250, 'format': '%Y-%m-%d %H:%M:%S', 'locale': None},  # C locale format with non-UTC tz (found in modified vmstat cli output): # 2021-09-16 20:32:28 PDT
+            {'id': 7255, 'format': '%Y-%m-%d %H:%M:%S %Z', 'locale': None},  # C locale format (found in modified vmstat cli output): # 2021-09-16 20:32:28 UTC
             {'id': 7300, 'format': '%a %Y-%m-%d %H:%M:%S %Z', 'locale': None},  # C locale format (found in timedatectl cli output): # Wed 2020-03-11 00:53:21 UTC
             # attempt locale changes last
             {'id': 8000, 'format': '%a %d %b %Y %H:%M:%S %Z', 'locale': ''},  # current locale format (found in upower cli output): # mar. 23 mars 2021 23:12:11 UTC

man/jc.1

@@ -0,0 +1,619 @@
+.TH jc 1 2021-11-18 1.17.2 "JSON CLI output utility"
+.SH NAME
+jc \- JSONifies the output of many CLI tools and file-types
+.SH SYNOPSIS
+COMMAND | jc PARSER [OPTIONS]
+
+or "Magic" syntax:
+
+jc [OPTIONS] COMMAND
+
+.SH DESCRIPTION
+jc JSONifies the output of many CLI tools and file-types for easier parsing in scripts. jc accepts piped input from \fBSTDIN\fP and outputs a JSON representation of the previous command's output to \fBSTDOUT\fP. Alternatively, the "Magic" syntax can be used by prepending jc to the command to be converted. Options can be passed to jc immediately before the command is given. (Note: "Magic" syntax does not support shell builtins or command aliases)
+
+.SH OPTIONS
+.B
+Parsers:
+.RS
+
+
+.TP
+.B
+\fB--acpi\fP
+`acpi` command parser
+
+.TP
+.B
+\fB--airport\fP
+`airport -I` command parser
+
+.TP
+.B
+\fB--airport-s\fP
+`airport -s` command parser
+
+.TP
+.B
+\fB--arp\fP
+`arp` command parser
+
+.TP
+.B
+\fB--blkid\fP
+`blkid` command parser
+
+.TP
+.B
+\fB--cksum\fP
+`cksum` and `sum` command parser
+
+.TP
+.B
+\fB--crontab\fP
+`crontab` command and file parser
+
+.TP
+.B
+\fB--crontab-u\fP
+`crontab` file parser with user support
+
+.TP
+.B
+\fB--csv\fP
+CSV file parser
+
+.TP
+.B
+\fB--csv-s\fP
+CSV file streaming parser
+
+.TP
+.B
+\fB--date\fP
+`date` command parser
+
+.TP
+.B
+\fB--df\fP
+`df` command parser
+
+.TP
+.B
+\fB--dig\fP
+`dig` command parser
+
+.TP
+.B
+\fB--dir\fP
+`dir` command parser
+
+.TP
+.B
+\fB--dmidecode\fP
+`dmidecode` command parser
+
+.TP
+.B
+\fB--dpkg-l\fP
+`dpkg -l` command parser
+
+.TP
+.B
+\fB--du\fP
+`du` command parser
+
+.TP
+.B
+\fB--env\fP
+`env` command parser
+
+.TP
+.B
+\fB--file\fP
+`file` command parser
+
+.TP
+.B
+\fB--finger\fP
+`finger` command parser
+
+.TP
+.B
+\fB--free\fP
+`free` command parser
+
+.TP
+.B
+\fB--fstab\fP
+`/etc/fstab` file parser
+
+.TP
+.B
+\fB--group\fP
+`/etc/group` file parser
+
+.TP
+.B
+\fB--gshadow\fP
+`/etc/gshadow` file parser
+
+.TP
+.B
+\fB--hash\fP
+`hash` command parser
+
+.TP
+.B
+\fB--hashsum\fP
+hashsum command parser (`md5sum`, `shasum`, etc.)
+
+.TP
+.B
+\fB--hciconfig\fP
+`hciconfig` command parser
+
+.TP
+.B
+\fB--history\fP
+`history` command parser
+
+.TP
+.B
+\fB--hosts\fP
+`/etc/hosts` file parser
+
+.TP
+.B
+\fB--id\fP
+`id` command parser
+
+.TP
+.B
+\fB--ifconfig\fP
+`ifconfig` command parser
+
+.TP
+.B
+\fB--ini\fP
+INI file parser
+
+.TP
+.B
+\fB--iptables\fP
+`iptables` command parser
+
+.TP
+.B
+\fB--iw-scan\fP
+`iw dev [device] scan` command parser
+
+.TP
+.B
+\fB--jobs\fP
+`jobs` command parser
+
+.TP
+.B
+\fB--kv\fP
+Key/Value file parser
+
+.TP
+.B
+\fB--last\fP
+`last` and `lastb` command parser
+
+.TP
+.B
+\fB--ls\fP
+`ls` command parser
+
+.TP
+.B
+\fB--ls-s\fP
+`ls` command streaming parser
+
+.TP
+.B
+\fB--lsblk\fP
+`lsblk` command parser
+
+.TP
+.B
+\fB--lsmod\fP
+`lsmod` command parser
+
+.TP
+.B
+\fB--lsof\fP
+`lsof` command parser
+
+.TP
+.B
+\fB--lsusb\fP
+`lsusb` command parser
+
+.TP
+.B
+\fB--mount\fP
+`mount` command parser
+
+.TP
+.B
+\fB--netstat\fP
+`netstat` command parser
+
+.TP
+.B
+\fB--ntpq\fP
+`ntpq -p` command parser
+
+.TP
+.B
+\fB--passwd\fP
+`/etc/passwd` file parser
+
+.TP
+.B
+\fB--ping\fP
+`ping` and `ping6` command parser
+
+.TP
+.B
+\fB--ping-s\fP
+`ping` and `ping6` command streaming parser
+
+.TP
+.B
+\fB--pip-list\fP
+`pip list` command parser
+
+.TP
+.B
+\fB--pip-show\fP
+`pip show` command parser
+
+.TP
+.B
+\fB--ps\fP
+`ps` command parser
+
+.TP
+.B
+\fB--route\fP
+`route` command parser
+
+.TP
+.B
+\fB--rpm-qi\fP
+`rpm -qi` command parser
+
+.TP
+.B
+\fB--sfdisk\fP
+`sfdisk` command parser
+
+.TP
+.B
+\fB--shadow\fP
+`/etc/shadow` file parser
+
+.TP
+.B
+\fB--ss\fP
+`ss` command parser
+
+.TP
+.B
+\fB--stat\fP
+`stat` command parser
+
+.TP
+.B
+\fB--sysctl\fP
+`sysctl` command parser
+
+.TP
+.B
+\fB--systemctl\fP
+`systemctl` command parser
+
+.TP
+.B
+\fB--systemctl-lj\fP
+`systemctl list-jobs` command parser
+
+.TP
+.B
+\fB--systemctl-ls\fP
+`systemctl list-sockets` command parser
+
+.TP
+.B
+\fB--systemctl-luf\fP
+`systemctl list-unit-files` command parser
+
+.TP
+.B
+\fB--systeminfo\fP
+`systeminfo` command parser
+
+.TP
+.B
+\fB--time\fP
+`/usr/bin/time` command parser
+
+.TP
+.B
+\fB--timedatectl\fP
+`timedatectl status` command parser
+
+.TP
+.B
+\fB--tracepath\fP
+`tracepath` and `tracepath6` command parser
+
+.TP
+.B
+\fB--traceroute\fP
+`traceroute` and `traceroute6` command parser
+
+.TP
+.B
+\fB--ufw\fP
+`ufw status` command parser
+
+.TP
+.B
+\fB--ufw-appinfo\fP
+`ufw app info [application]` command parser
+
+.TP
+.B
+\fB--uname\fP
+`uname -a` command parser
+
+.TP
+.B
+\fB--upower\fP
+`upower` command parser
+
+.TP
+.B
+\fB--uptime\fP
+`uptime` command parser
+
+.TP
+.B
+\fB--vmstat\fP
+`vmstat` command parser
+
+.TP
+.B
+\fB--vmstat-s\fP
+`vmstat` command streaming parser
+
+.TP
+.B
+\fB--w\fP
+`w` command parser
+
+.TP
+.B
+\fB--wc\fP
+`wc` command parser
+
+.TP
+.B
+\fB--who\fP
+`who` command parser
+
+.TP
+.B
+\fB--xml\fP
+XML file parser
+
+.TP
+.B
+\fB--yaml\fP
+YAML file parser
+
+
+.RE
+.PP
+.B
+Options:
+.RS
+
+.TP
+.B
+\fB-a\fP
+about jc (JSON output)
+.TP
+.B
+\fB-d\fP
+debug - show traceback (use \fB-dd\fP for verbose traceback)
+.TP
+.B
+\fB-h\fP
+help (\fB-h --parser_name\fP for parser documentation)
+.TP
+.B
+\fB-m\fP
+monochrome output
+.TP
+.B
+\fB-p\fP
+pretty print output
+.TP
+.B
+\fB-q\fP
+quiet - suppress warnings (use \fB-qq\fP to ignore streaming parser errors)
+.TP
+.B
+\fB-r\fP
+raw JSON output
+.TP
+.B
+\fB-u\fP
+unbuffer output (useful for slow streaming data with streaming parsers)
+.TP
+.B
+\fB-v\fP
+version information
+
+.SH EXIT CODES
+Any fatal errors within jc will generate an exit code of \fB100\fP, otherwise the exit code will be \fB0\fP. When using the "Magic" syntax (e.g. \fBjc ifconfig eth0\fP), jc will store the exit code of the program being parsed and add it to the  jc exit code. This way it is easier to determine if an error was from the parsed program or jc.
+
+Consider the following examples using `ifconfig`:
+
+.RS
+ifconfig exit code = \fB0\fP, jc exit code = \fB0\fP, combined exit code = \fB0\fP (no errors)
+
+ifconfig exit code = \fB1\fP, jc exit code = \fB0\fP, combined exit code = \fB1\fP (error in ifconfig)
+
+ifconfig exit code = \fB0\fP, jc exit code = \fB100\fP, combined exit code = \fB100\fP (error in jc)
+
+ifconfig exit code = \fB1\fP, jc exit code = \fB100\fP, combined exit code = \fB101\fP (error in both ifconfig and jc)
+.RE
+
+.SH ENVIRONMENT
+You can specify custom colors via the \fBJC_COLORS\fP environment variable. The \fBJC_COLORS\fP environment variable takes four comma separated string values in the following format:
+
+JC_COLORS=<keyname_color>,<keyword_color>,<number_color>,<string_color>
+
+Where colors are: \fBblack\fP, \fBred\fP, \fBgreen\fP, \fByellow\fP, \fBblue\fP, \fBmagenta\fP, \fBcyan\fP, \fBgray\fP, \fBbrightblack\fP, \fBbrightred\fP, \fBbrightgreen\fP, \fBbrightyellow\fP, \fBbrightblue\fP, \fBbrightmagenta\fP, \fBbrightcyan\fP, \fBwhite\fP, or \fBdefault\fP
+
+For example, to set to the default colors:
+
+.RS
+JC_COLORS=blue,brightblack,magenta,green
+
+or
+
+JC_COLORS=default,default,default,default
+.RE
+
+
+.SH STREAMING PARSERS
+Most parsers load all of the data from \fBSTDIN\fP, parse it, then output the entire JSON document serially. There are some streaming parsers (e.g. \fBls-s\fP and \fBping-s\fP) that immediately start processing and outputing the data line-by-line as JSON Lines (aka NDJSON) while it is being received from \fBSTDIN\fP. This can significantly reduce the amount of memory required to parse large amounts of command output (e.g. \fBls -lR /\fP) and can sometimes process the data more quickly. Streaming parsers have slightly different behavior than standard parsers as outlined below.
+
+.RS
+Note: Streaming parsers cannot be used with the "magic" syntax
+.RE
+
+\fBIgnoring Errors\fP
+
+You may want to ignore parsing errors when using streaming parsers since these may be used in long-lived processing pipelines and errors can break the pipe. To ignore parsing errors, use the \fB-qq\fP cli option. This will add a \fB_jc_meta\fP object to the JSON output with a \fBsuccess\fP attribute. If \fBsuccess\fP is \fBtrue\fP, then there were no issues parsing the line. If \fBsuccess\fP is \fBfalse\fP, then a parsing issue was found and \fBerror\fP and \fBline\fP fields will be added to include a short error description and the contents of the unparsable line, respectively:
+
+.RS
+Successfully parsed line with \fB-qq\fP option:
+.RS
+{
+
+  "command_data": "data",
+  
+  "_jc_meta": {
+  
+    "success": true
+    
+  }
+  
+}
+.RE
+
+Unsuccessfully parsed line with \fB-qq\fP option:
+.RS
+{
+
+  "_jc_meta": {
+  
+    "success": false,
+    
+    "error": "error message",
+    
+    "line": "original line data"
+    
+  }
+  
+}
+.RE
+
+.RE
+\fBUnbuffering Output\fP
+
+Most operating systems will buffer output that is being piped from process to process. The buffer is usually around 4KB. When viewing the output in the terminal the OS buffer is not engaged so output is immediately displayed on the screen. When piping multiple processes together, though, it may seem as if the output is hanging when the input data is very slow (e.g. \fBping\fP):
+
+.RS
+$ ping 1.1.1.1 | jc --ping-s | jq
+
+<slow output>
+.RE
+
+This is because the OS engages the 4KB buffer between \fBjc\fP and \fBjq\fP in this example. To display the data on the terminal in realtime, you can disable the buffer with the \fB-u\fP (unbuffer) cli option:
+
+.RS
+$ ping 1.1.1.1 | jc --ping-s -u | jq
+
+{"type":"reply","pattern":null,"timestamp":null,"bytes":"64","response_ip":"1.1.1.1","icmp_seq":"1","ttl":"128","time_ms":"24.6","duplicate":false}
+
+{"type":"reply","pattern":null,"timestamp":null,"bytes":"64","response_ip":"1.1.1.1","icmp_seq":"2","ttl":"128","time_ms":"26.8","duplicate":false}
+
+etc...
+
+Note: Unbuffered output can be slower for large data streams.
+.RE
+
+.SH CUSTOM PARSERS
+Custom local parser plugins may be placed in a \fBjc/jcparsers\fP folder in your local "App data directory":
+
+.RS
+- Linux/unix: \fB$HOME/.local/share/jc/jcparsers\fP
+
+- macOS: \fB$HOME/Library/Application Support/jc/jcparsers\fP
+
+- Windows: \fB$LOCALAPPDATA\\jc\\jc\\jcparsers\fP
+.RE
+
+Local parser plugins are standard python module files. Use the \fBjc/parsers/foo.py\fP parser as a template and simply place a \fB.py\fP file in the \fBjcparsers\fP subfolder.
+
+Local plugin filenames must be valid python module names, therefore must consist entirely of alphanumerics and start with a letter. Local plugins may override default plugins.
+
+Note: The application data directory follows the XDG Base Directory Specification
+
+.SH CAVEATS
+\fBLocale:\fP For best results set the \fBLANG\fP locale environment variable to \fBC\fP or \fBen_US.UTF-8\fP. For example, either by setting directly on the command-line:
+
+\fB$ LANG=C date | jc --date\fP
+
+or by exporting to the environment before running commands:
+
+\fB$ export LANG=C\fP
+
+\fBTimezones:\fP Some parsers have calculated epoch timestamp fields added to the output. Unless a timestamp field name has a \fB_utc\fP suffix it is considered naive. (i.e. based on the local timezone of the system the \fBjc\fP parser was run on).
+
+If a UTC timezone can be detected in the text of the command output, the timestamp will be timezone aware and have a \fB_utc\fP suffix on the key name. (e.g. \fBepoch_utc\fP) No other timezones are supported for aware timestamps.
+
+.SH EXAMPLES
+Standard Syntax:
+.RS
+$ dig www.google.com | jc \fB--dig\fP \fB-p\fP
+.RE
+
+Magic Syntax:
+.RS
+$ jc \fB-p\fP dig www.google.com
+.RE
+
+For parser documentation:
+.RS
+$ jc \fB-h\fP \fB--dig\fP
+.RE
+.SH AUTHOR
+Kelly Brazil (kellyjonbrazil@gmail.com)
+
+https://github.com/kellyjonbrazil/jc
+
+.SH COPYRIGHT
+Copyright (c) 2019-2021 Kelly Brazil
+
+License:  MIT License

mangen.py

@@ -15,11 +15,3 @@ output = template.render(today=date.today(),
 
 with open('man/jc.1', 'w') as f:
     f.write(output)
-
-with open('man/jc.1', 'rb') as f_in:
-    with gzip.open('man/jc.1.gz', 'wb') as f_out:
-        shutil.copyfileobj(f_in, f_out)
-
-shutil.copyfile('man/jc.1.gz', 'jc/man/jc.1.gz')
-
-os.remove('man/jc.1')

setup.py

@@ -5,7 +5,7 @@ with open('README.md', 'r') as f:
 
 setuptools.setup(
     name='jc',
-    version='1.15.4',
+    version='1.17.2',
     author='Kelly Brazil',
     author_email='kellyjonbrazil@gmail.com',
     description='Converts the output of popular command-line tools and file-types to JSON.',

templates/manpage_template

@@ -4,12 +4,12 @@ jc \- JSONifies the output of many CLI tools and file-types
 .SH SYNOPSIS
 COMMAND | jc PARSER [OPTIONS]
 
-or magic syntax:
+or "Magic" syntax:
 
 jc [OPTIONS] COMMAND
 
 .SH DESCRIPTION
-jc JSONifies the output of many CLI tools and file-types for easier parsing in scripts. jc accepts piped input from \fBSTDIN\fP and outputs a JSON representation of the previous command's output to \fBSTDOUT\fP. Alternatively, the "magic" syntax can be used by prepending jc to the command to be converted. Options can be passed to jc immediately before the command is given. (Note: command aliases are not supported).
+jc JSONifies the output of many CLI tools and file-types for easier parsing in scripts. jc accepts piped input from \fBSTDIN\fP and outputs a JSON representation of the previous command's output to \fBSTDOUT\fP. Alternatively, the "Magic" syntax can be used by prepending jc to the command to be converted. Options can be passed to jc immediately before the command is given. (Note: "Magic" syntax does not support shell builtins or command aliases)
 
 .SH OPTIONS
 .B
@@ -36,7 +36,7 @@ about jc (JSON output)
 .TP
 .B
 \fB-d\fP
-debug - show traceback (\fB-dd\fP for verbose traceback)
+debug - show traceback (use \fB-dd\fP for verbose traceback)
 .TP
 .B
 \fB-h\fP
@@ -52,18 +52,22 @@ pretty print output
 .TP
 .B
 \fB-q\fP
-quiet - suppress warnings
+quiet - suppress warnings (use \fB-qq\fP to ignore streaming parser errors)
 .TP
 .B
 \fB-r\fP
 raw JSON output
 .TP
 .B
+\fB-u\fP
+unbuffer output (useful for slow streaming data with streaming parsers)
+.TP
+.B
 \fB-v\fP
 version information
 
 .SH EXIT CODES
-Any fatal errors within jc will generate an exit code of \fB100\fP, otherwise the exit code will be \fB0\fP. When using the "magic" syntax (e.g. \fBjc ifconfig eth0\fP), jc will store the exit code of the program being parsed and add it to the  jc exit code. This way it is easier to determine if an error was from the parsed program or jc.
+Any fatal errors within jc will generate an exit code of \fB100\fP, otherwise the exit code will be \fB0\fP. When using the "Magic" syntax (e.g. \fBjc ifconfig eth0\fP), jc will store the exit code of the program being parsed and add it to the  jc exit code. This way it is easier to determine if an error was from the parsed program or jc.
 
 Consider the following examples using `ifconfig`:
 
@@ -94,6 +98,76 @@ or
 JC_COLORS=default,default,default,default
 .RE
 
+
+.SH STREAMING PARSERS
+Most parsers load all of the data from \fBSTDIN\fP, parse it, then output the entire JSON document serially. There are some streaming parsers (e.g. \fBls-s\fP and \fBping-s\fP) that immediately start processing and outputing the data line-by-line as JSON Lines (aka NDJSON) while it is being received from \fBSTDIN\fP. This can significantly reduce the amount of memory required to parse large amounts of command output (e.g. \fBls -lR /\fP) and can sometimes process the data more quickly. Streaming parsers have slightly different behavior than standard parsers as outlined below.
+
+.RS
+Note: Streaming parsers cannot be used with the "magic" syntax
+.RE
+
+\fBIgnoring Errors\fP
+
+You may want to ignore parsing errors when using streaming parsers since these may be used in long-lived processing pipelines and errors can break the pipe. To ignore parsing errors, use the \fB-qq\fP cli option. This will add a \fB_jc_meta\fP object to the JSON output with a \fBsuccess\fP attribute. If \fBsuccess\fP is \fBtrue\fP, then there were no issues parsing the line. If \fBsuccess\fP is \fBfalse\fP, then a parsing issue was found and \fBerror\fP and \fBline\fP fields will be added to include a short error description and the contents of the unparsable line, respectively:
+
+.RS
+Successfully parsed line with \fB-qq\fP option:
+.RS
+{
+
+  "command_data": "data",
+  
+  "_jc_meta": {
+  
+    "success": true
+    
+  }
+  
+}
+.RE
+
+Unsuccessfully parsed line with \fB-qq\fP option:
+.RS
+{
+
+  "_jc_meta": {
+  
+    "success": false,
+    
+    "error": "error message",
+    
+    "line": "original line data"
+    
+  }
+  
+}
+.RE
+
+.RE
+\fBUnbuffering Output\fP
+
+Most operating systems will buffer output that is being piped from process to process. The buffer is usually around 4KB. When viewing the output in the terminal the OS buffer is not engaged so output is immediately displayed on the screen. When piping multiple processes together, though, it may seem as if the output is hanging when the input data is very slow (e.g. \fBping\fP):
+
+.RS
+$ ping 1.1.1.1 | jc --ping-s | jq
+
+<slow output>
+.RE
+
+This is because the OS engages the 4KB buffer between \fBjc\fP and \fBjq\fP in this example. To display the data on the terminal in realtime, you can disable the buffer with the \fB-u\fP (unbuffer) cli option:
+
+.RS
+$ ping 1.1.1.1 | jc --ping-s -u | jq
+
+{"type":"reply","pattern":null,"timestamp":null,"bytes":"64","response_ip":"1.1.1.1","icmp_seq":"1","ttl":"128","time_ms":"24.6","duplicate":false}
+
+{"type":"reply","pattern":null,"timestamp":null,"bytes":"64","response_ip":"1.1.1.1","icmp_seq":"2","ttl":"128","time_ms":"26.8","duplicate":false}
+
+etc...
+
+Note: Unbuffered output can be slower for large data streams.
+.RE
+
 .SH CUSTOM PARSERS
 Custom local parser plugins may be placed in a \fBjc/jcparsers\fP folder in your local "App data directory":
 
@@ -112,7 +186,7 @@ Local plugin filenames must be valid python module names, therefore must consist
 Note: The application data directory follows the XDG Base Directory Specification
 
 .SH CAVEATS
-\fBLocale:\fP For best results set the \fBLANG\fP locale environment variable to \fBC\fP. For example, either by setting directly on the command-line:
+\fBLocale:\fP For best results set the \fBLANG\fP locale environment variable to \fBC\fP or \fBen_US.UTF-8\fP. For example, either by setting directly on the command-line:
 
 \fB$ LANG=C date | jc --date\fP
 

templates/readme_template

@@ -1,9 +1,9 @@
 ![Tests](https://github.com/kellyjonbrazil/jc/workflows/Tests/badge.svg?branch=master)
 ![Pypi](https://img.shields.io/pypi/v/jc.svg)
 
-> Try the new `jc` [web demo](https://jc-web-demo.herokuapp.com/)!
+> Try the `jc` [web demo](https://jc-web-demo.herokuapp.com/)
 
-> JC is [now available](https://galaxy.ansible.com/community/general) as an Ansible filter plugin in the `community.general` collection! See this [blog post](https://blog.kellybrazil.com/2020/08/30/parsing-command-output-in-ansible-with-jc/) for an example.
+> JC is [now available](https://galaxy.ansible.com/community/general) as an Ansible filter plugin in the `community.general` collection. See this [blog post](https://blog.kellybrazil.com/2020/08/30/parsing-command-output-in-ansible-with-jc/) for an example.
 
 # JC
 JSON CLI output utility
@@ -19,7 +19,7 @@ dig example.com | jc --dig
 39049,"data":"93.184.216.34"}],"query_time":49,"server":"2600:1700:bab0:d40::1#53(2600:1700:bab0:d40::1)","when":
 "Fri Apr 16 16:09:00 PDT 2021","rcvd":56,"when_epoch":1618614540,"when_epoch_utc":null}]
 ```
-This allows further command-line processing of output with tools like `jq` by piping commands:
+This allows further command-line processing of output with tools like `jq` or [`jello`](https://github.com/kellyjonbrazil/jello) by piping commands:
 ```bash
 $ dig example.com | jc --dig | jq -r '.[].answer[].data'
 93.184.216.34
@@ -31,28 +31,13 @@ $ jc dig example.com | jq -r '.[].answer[].data'
 ```
 The `jc` parsers can also be used as python modules. In this case the output will be a python dictionary, or list of dictionaries, instead of JSON:
 ```python
+>>> import subprocess
 >>> import jc.parsers.dig
 >>> 
->>> data = '''; <<>> DiG 9.10.6 <<>> example.com
-... ;; global options: +cmd
-... ;; Got answer:
-... ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64612
-... ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
-... 
-... ;; OPT PSEUDOSECTION:
-... ; EDNS: version: 0, flags:; udp: 4096
-... ;; QUESTION SECTION:
-... ;example.com.           IN  A
-... 
-... ;; ANSWER SECTION:
-... example.com.        29658   IN  A   93.184.216.34
-... 
-... ;; Query time: 52 msec
-... ;; SERVER: 2600:1700:bab0:d40::1#53(2600:1700:bab0:d40::1)
-... ;; WHEN: Fri Apr 16 16:13:00 PDT 2021
-... ;; MSG SIZE  rcvd: 56'''
+>>> cmd_output = subprocess.check_output(['dig', 'example.com'], text=True)
+>>> data = jc.parsers.dig.parse(cmd_output)
 >>>
->>> jc.parsers.dig.parse(data)
+>>> data
 [{'id': 64612, 'opcode': 'QUERY', 'status': 'NOERROR', 'flags': ['qr', 'rd', 'ra'], 'query_num': 1, 'answer_num':
 1, 'authority_num': 0, 'additional_num': 1, 'opt_pseudosection': {'edns': {'version': 0, 'flags': [], 'udp':
 4096}}, 'question': {'name': 'example.com.', 'class': 'IN', 'type': 'A'}, 'answer': [{'name': 'example.com.',
@@ -60,7 +45,7 @@ The `jc` parsers can also be used as python modules. In this case the output wil
 '2600:1700:bab0:d40::1#53(2600:1700:bab0:d40::1)', 'when': 'Fri Apr 16 16:13:00 PDT 2021', 'rcvd': 56,
 'when_epoch': 1618614780, 'when_epoch_utc': None}]
 ```
-Two representations of the data are possible. The default representation uses a strict schema per parser and converts known numbers to int/float JSON values. Certain known values of `None` are converted to JSON `null`, known boolean values are converted, and, in some cases, additional semantic context fields are added.
+Two representations of the data are available. The default representation uses a strict schema per parser and converts known numbers to int/float JSON values. Certain known values of `None` are converted to JSON `null`, known boolean values are converted, and, in some cases, additional semantic context fields are added.
 
 To access the raw, pre-processed JSON, use the `-r` cli option or the `raw=True` function parameter in `parse()`.
 
@@ -75,6 +60,8 @@ See also:
 - [libxo on FreeBSD](http://juniper.github.io/libxo/libxo-manual.html)
 - [powershell](https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/convertto-json?view=powershell-7)
 - [blog: linux apps should have a json flag](https://thomashunter.name/posts/2012-06-06-linux-cli-apps-should-have-a-json-flag)
+- [Hacker News discussion](https://news.ycombinator.com/item?id=28266193)
+- [Reddit discussion](https://www.reddit.com/r/programming/comments/pa4cbb/bringing_the_unix_philosophy_to_the_21st_century/)
 
 Use Cases:
 - [Bash scripting](https://blog.kellybrazil.com/2021/04/12/practical-json-at-the-command-line/)
@@ -83,7 +70,7 @@ Use Cases:
 - [Nornir command output parsing](https://blog.kellybrazil.com/2020/12/09/parsing-command-output-in-nornir-with-jc/)
 
 ## Installation
-There are several ways to get `jc`. You can install via `pip`; other OS package repositories like `apt-get`, `dnf`, `zypper`, `pacman`, `nix-env`, `guix`, `brew`, or `portsnap`; via DEB/RPM packaged binaries; or by downloading the correct binary for your architecture and running it anywhere on your filesystem.
+There are several ways to get `jc`. You can install via `pip`, OS package repositories, via DEB/RPM/MSI packaged binaries for linux and Windows, or by downloading the correct binary for your architecture and running it anywhere on your filesystem.
 
 ### Pip (macOS, linux, unix, Windows)
 ```bash
@@ -100,7 +87,7 @@ pip3 install jc
 | Arch linux            | `pacman -S jc`                                                                |
 | NixOS linux           | `nix-env -iA nixpkgs.jc` or `nix-env -iA nixos.jc`                            |
 | Guix System linux     | `guix install jc`                                                             |
-| MacOS                 | `brew install jc`                                                             |
+| macOS                 | `brew install jc`                                                             |
 | FreeBSD               | `portsnap fetch update && cd /usr/ports/textproc/py-jc && make install clean` |
 | Ansible filter plugin | `ansible-galaxy collection install community.general`                         |
 
@@ -124,11 +111,12 @@ The JSON output can be compact (default) or pretty formatted with the `-p` optio
 ### Options
 - `-a` about `jc`. Prints information about `jc` and the parsers (in JSON, of course!)
 - `-d` debug mode. Prints trace messages if parsing issues are encountered (use `-dd` for verbose debugging)
-- `-h` `jc` help. Use `jc -h --parser_name` for parser documentation
+- `-h` help. Use `jc -h --parser_name` for parser documentation
 - `-m` monochrome JSON output
 - `-p` pretty format the JSON output
-- `-q` quiet mode. Suppresses parser warning messages
+- `-q` quiet mode. Suppresses parser warning messages (use `-qq` to ignore streaming parser errors)
 - `-r` raw output. Provides a more literal JSON output, typically with string values and no additional semantic processing
+- `-u` unbuffer output
 - `-v` version information
 
 ### Exit Codes
@@ -160,6 +148,64 @@ or
 JC_COLORS=default,default,default,default
 ```
 
+### Streaming Parsers
+Most parsers load all of the data from STDIN, parse it, then output the entire JSON document serially. There are some streaming parsers (e.g. `ls-s` and `ping-s`) that immediately start processing and outputing the data line-by-line as [JSON Lines](https://jsonlines.org/) (aka [NDJSON](http://ndjson.org/)) while it is being received from STDIN. This can significantly reduce the amount of memory required to parse large amounts of command output (e.g. `ls -lR /`) and can sometimes process the data more quickly. Streaming parsers have slightly different behavior than standard parsers as outlined below.
+
+> Note: Streaming parsers cannot be used with the "magic" syntax
+
+#### Ignoring Errors
+
+You may want to ignore parsing errors when using streaming parsers since these may be used in long-lived processing pipelines and errors can break the pipe. To ignore parsing errors, use the `-qq` cli option or the `ignore_exceptions=True` argument with the `parse()` function. This will add a `_jc_meta` object to the JSON output with a `success` attribute. If `success` is `true`, then there were no issues parsing the line. If `success` is `false`, then a parsing issue was found and `error` and `line` fields will be added to include a short error description and the contents of the unparsable line, respectively:
+
+Successfully parsed line with `-qq` option:
+```json
+{
+  "command_data": "data",
+  "_jc_meta": {
+    "success": true
+  }
+}
+```
+Unsuccessfully parsed line with `-qq` option:
+```json
+{
+  "_jc_meta": {
+    "success": false,
+    "error": "error message",
+    "line": "original line data"
+  }
+}
+```
+
+#### Unbuffering Output
+
+Most operating systems will buffer output that is being piped from process to process. The buffer is usually around 4KB. When viewing the output in the terminal the OS buffer is not engaged so output is immediately displayed on the screen. When piping multiple processes together, though, it may seem as if the output is hanging when the input data is very slow (e.g. `ping`):
+```
+$ ping 1.1.1.1 | jc --ping-s | jq
+<slow output>
+```
+This is because the OS engages the 4KB buffer between `jc` and `jq` in this example. To display the data on the terminal in realtime, you can disable the buffer with the `-u` (unbuffer) cli option:
+```
+$ ping 1.1.1.1 | jc --ping-s -u | jq
+{"type":"reply","pattern":null,"timestamp":null,"bytes":"64","response_ip":"1.1.1.1","icmp_seq":"1","ttl":"128","time_ms":"24.6","duplicate":false}
+{"type":"reply","pattern":null,"timestamp":null,"bytes":"64","response_ip":"1.1.1.1","icmp_seq":"2","ttl":"128","time_ms":"26.8","duplicate":false}
+...
+```
+> Note: Unbuffered output can be slower for large data streams.
+
+#### Using Streaming Parsers as Python Modules
+
+Streaming parsers accept any iterable object and return a generator iterator object allowing lazy processing of the data. The input data should iterate on lines of string data. Examples of good input data are `sys.stdin` or `str.splitlines()`.
+
+To use the generator object in your code, simply loop through it or use the [next()](https://docs.python.org/3/library/functions.html#next) builtin function:
+```python
+import jc.parsers.ls_s
+
+result = jc.parsers.ls_s.parse(ls_command_output.splitlines())
+for item in result:
+    print(item["filename"])
+```
+
 ### Custom Parsers
 Custom local parser plugins may be placed in a `jc/jcparsers` folder in your local **"App data directory"**:
 
@@ -174,9 +220,10 @@ Local plugin filenames must be valid python module names, therefore must consist
 > Note: The application data directory follows the [XDG Base Directory Specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html)
 
 ### Caveats
-**Locale:**
 
-For best results set the `LANG` locale environment variable to `C`. For example, either by setting directly on the command-line:
+#### Locale
+
+For best results set the `LANG` locale environment variable to `C` or `en_US.UTF-8`. For example, either by setting directly on the command-line:
 ```
 $ LANG=C date | jc --date
 ```
@@ -185,7 +232,7 @@ or by exporting to the environment before running commands:
 $ export LANG=C
 ```
 
-**Timezones:**
+#### Timezones
 
 Some parsers have calculated epoch timestamp fields added to the output. Unless a timestamp field name has a `_utc` suffix it is considered naive. (i.e. based on the local timezone of the system the `jc` parser was run on).
 
@@ -194,11 +241,16 @@ If a UTC timezone can be detected in the text of the command output, the timesta
 ## Compatibility
 Some parsers like `dig`, `xml`, `csv`, etc. will work on any platform. Other parsers that convert platform-specific output will generate a warning message if they are run on an unsupported platform. To see all parser information, including compatibility, run `jc -ap`.
 
-You may still use a parser on an unsupported platform - for example, you may want to parse a file with linux `lsof` output on an macOS laptop. In that case you can suppress the warning message with the `-q` cli option or the `quiet=True` function parameter in `parse()`:
+You may still use a parser on an unsupported platform - for example, you may want to parse a file with linux `lsof` output on an macOS or Windows laptop. In that case you can suppress the warning message with the `-q` cli option or the `quiet=True` function parameter in `parse()`:
 
+macOS:
 ```bash
 cat lsof.out | jc --lsof -q
 ```
+or Windows:
+```bash
+type lsof.out | jc --lsof -q
+```
 
 Tested on:
 - Centos 7.7
@@ -210,9 +262,11 @@ Tested on:
 - NixOS
 - FreeBSD12
 - Windows 10
+- Windows 2016 Server
+- Windows 2019 Server
 
 ## Contributions
-Feel free to add/improve code or parsers! You can use the [`jc/parsers/foo.py`](https://github.com/kellyjonbrazil/jc/blob/master/jc/parsers/foo.py) parser as a template and submit your parser with a pull request.
+Feel free to add/improve code or parsers! You can use the [`jc/parsers/foo.py`](https://github.com/kellyjonbrazil/jc/blob/master/jc/parsers/foo.py) or [`jc/parsers/foo_s.py (streaming)`](https://github.com/kellyjonbrazil/jc/blob/master/jc/parsers/foo_s.py) parsers as a template and submit your parser with a pull request.
 
 Please see the [Contributing Guidelines](https://github.com/kellyjonbrazil/jc/blob/master/CONTRIBUTING.md) for more information.
 

tests/fixtures/alpine-linux-3.13/ping-hostname.json

@@ -0,0 +1 @@
+{"destination_ip":"142.250.125.102","data_bytes":56,"pattern":null,"destination":"google.com","packets_transmitted":4,"packets_received":4,"packet_loss_percent":0.0,"duplicates":0,"round_trip_ms_min":1.281,"round_trip_ms_avg":1.345,"round_trip_ms_max":1.408,"round_trip_ms_stddev":null,"responses":[{"type":"reply","bytes":64,"response_ip":"142.250.125.102","icmp_seq":0,"ttl":42,"time_ms":1.331,"duplicate":false},{"type":"reply","bytes":64,"response_ip":"142.250.125.102","icmp_seq":1,"ttl":42,"time_ms":1.281,"duplicate":false},{"type":"reply","bytes":64,"response_ip":"142.250.125.102","icmp_seq":2,"ttl":42,"time_ms":1.408,"duplicate":false},{"type":"reply","bytes":64,"response_ip":"142.250.125.102","icmp_seq":3,"ttl":42,"time_ms":1.36,"duplicate":false}]}

tests/fixtures/alpine-linux-3.13/ping-hostname.out

@@ -0,0 +1,9 @@
+PING google.com (142.250.125.102): 56 data bytes
+64 bytes from 142.250.125.102: seq=0 ttl=42 time=1.331 ms
+64 bytes from 142.250.125.102: seq=1 ttl=42 time=1.281 ms
+64 bytes from 142.250.125.102: seq=2 ttl=42 time=1.408 ms
+64 bytes from 142.250.125.102: seq=3 ttl=42 time=1.360 ms
+
+--- google.com ping statistics ---
+4 packets transmitted, 4 packets received, 0% packet loss
+round-trip min/avg/max = 1.281/1.345/1.408 ms

tests/fixtures/alpine-linux-3.13/ping-ip.json

@@ -0,0 +1 @@
+{"destination_ip":"8.8.8.8","data_bytes":56,"pattern":null,"destination":"8.8.8.8","packets_transmitted":1,"packets_received":1,"packet_loss_percent":0.0,"duplicates":0,"round_trip_ms_min":1.637,"round_trip_ms_avg":1.637,"round_trip_ms_max":1.637,"round_trip_ms_stddev":null,"responses":[{"type":"reply","bytes":64,"response_ip":"8.8.8.8","icmp_seq":0,"ttl":42,"time_ms":1.637,"duplicate":false}]}

tests/fixtures/alpine-linux-3.13/ping-ip.out

@@ -0,0 +1,6 @@
+PING 8.8.8.8 (8.8.8.8): 56 data bytes
+64 bytes from 8.8.8.8: seq=0 ttl=42 time=1.637 ms
+
+--- 8.8.8.8 ping statistics ---
+1 packets transmitted, 1 packets received, 0% packet loss
+round-trip min/avg/max = 1.637/1.637/1.637 ms

tests/fixtures/centos-7.7/ls-al-streaming.json

@@ -0,0 +1 @@
+[{"filename":".","flags":"dr-xr-xr-x.","links":17,"owner":"root","group":"root","size":224,"date":"Aug 15 10:56"},{"filename":"..","flags":"dr-xr-xr-x.","links":17,"owner":"root","group":"root","size":224,"date":"Aug 15 10:56"},{"filename":"bin","link_to":"usr/bin","flags":"lrwxrwxrwx.","links":1,"owner":"root","group":"root","size":7,"date":"Aug 15 10:53"},{"filename":"boot","flags":"dr-xr-xr-x.","links":5,"owner":"root","group":"root","size":4096,"date":"Oct 21 13:18"},{"filename":"dev","flags":"drwxr-xr-x.","links":20,"owner":"root","group":"root","size":3180,"date":"Oct 25 18:21"},{"filename":"etc","flags":"drwxr-xr-x.","links":78,"owner":"root","group":"root","size":8192,"date":"Oct 25 18:47"},{"filename":"home","flags":"drwxr-xr-x.","links":3,"owner":"root","group":"root","size":21,"date":"Aug 15 10:56"},{"filename":"lib","link_to":"usr/lib","flags":"lrwxrwxrwx.","links":1,"owner":"root","group":"root","size":7,"date":"Aug 15 10:53"},{"filename":"lib64","link_to":"usr/lib64","flags":"lrwxrwxrwx.","links":1,"owner":"root","group":"root","size":9,"date":"Aug 15 10:53"},{"filename":"media","flags":"drwxr-xr-x.","links":2,"owner":"root","group":"root","size":6,"date":"Apr 10 2018"},{"filename":"mnt","flags":"drwxr-xr-x.","links":2,"owner":"root","group":"root","size":6,"date":"Apr 10 2018"},{"filename":"opt","flags":"drwxr-xr-x.","links":2,"owner":"root","group":"root","size":6,"date":"Apr 10 2018"},{"filename":"proc","flags":"dr-xr-xr-x.","links":121,"owner":"root","group":"root","size":0,"date":"Oct 25 18:21"},{"filename":"root","flags":"dr-xr-x---.","links":3,"owner":"root","group":"root","size":170,"date":"Oct 15 11:11"},{"filename":"run","flags":"drwxr-xr-x.","links":26,"owner":"root","group":"root","size":800,"date":"Oct 25 18:47"},{"filename":"sbin","link_to":"usr/sbin","flags":"lrwxrwxrwx.","links":1,"owner":"root","group":"root","size":8,"date":"Aug 15 10:53"},{"filename":"srv","flags":"drwxr-xr-x.","links":2,"owner":"root","group":"root","size":6,"date":"Apr 10 2018"},{"filename":"sys","flags":"dr-xr-xr-x.","links":13,"owner":"root","group":"root","size":0,"date":"Oct 25 18:21"},{"filename":"tmp","flags":"drwxrwxrwt.","links":19,"owner":"root","group":"root","size":4096,"date":"Oct 26 10:14"},{"filename":"usr","flags":"drwxr-xr-x.","links":13,"owner":"root","group":"root","size":155,"date":"Aug 15 10:53"},{"filename":"var","flags":"drwxr-xr-x.","links":19,"owner":"root","group":"root","size":267,"date":"Aug 15 10:57"}]

tests/fixtures/centos-7.7/ls-alh-streaming.json

@@ -0,0 +1 @@
+[{"filename":".","flags":"dr-xr-xr-x.","links":17,"owner":"root","group":"root","size":224,"date":"Aug 15 10:56"},{"filename":"..","flags":"dr-xr-xr-x.","links":17,"owner":"root","group":"root","size":224,"date":"Aug 15 10:56"},{"filename":"bin","link_to":"usr/bin","flags":"lrwxrwxrwx.","links":1,"owner":"root","group":"root","size":7,"date":"Aug 15 10:53"},{"filename":"boot","flags":"dr-xr-xr-x.","links":5,"owner":"root","group":"root","size":4,"date":"Oct 21 13:18"},{"filename":"dev","flags":"drwxr-xr-x.","links":20,"owner":"root","group":"root","size":3,"date":"Oct 25 18:21"},{"filename":"etc","flags":"drwxr-xr-x.","links":78,"owner":"root","group":"root","size":8,"date":"Oct 25 18:47"},{"filename":"home","flags":"drwxr-xr-x.","links":3,"owner":"root","group":"root","size":21,"date":"Aug 15 10:56"},{"filename":"lib","link_to":"usr/lib","flags":"lrwxrwxrwx.","links":1,"owner":"root","group":"root","size":7,"date":"Aug 15 10:53"},{"filename":"lib64","link_to":"usr/lib64","flags":"lrwxrwxrwx.","links":1,"owner":"root","group":"root","size":9,"date":"Aug 15 10:53"},{"filename":"media","flags":"drwxr-xr-x.","links":2,"owner":"root","group":"root","size":6,"date":"Apr 10 2018"},{"filename":"mnt","flags":"drwxr-xr-x.","links":2,"owner":"root","group":"root","size":6,"date":"Apr 10 2018"},{"filename":"opt","flags":"drwxr-xr-x.","links":2,"owner":"root","group":"root","size":6,"date":"Apr 10 2018"},{"filename":"proc","flags":"dr-xr-xr-x.","links":121,"owner":"root","group":"root","size":0,"date":"Oct 25 18:21"},{"filename":"root","flags":"dr-xr-x---.","links":3,"owner":"root","group":"root","size":170,"date":"Oct 15 11:11"},{"filename":"run","flags":"drwxr-xr-x.","links":26,"owner":"root","group":"root","size":800,"date":"Oct 25 18:47"},{"filename":"sbin","link_to":"usr/sbin","flags":"lrwxrwxrwx.","links":1,"owner":"root","group":"root","size":8,"date":"Aug 15 10:53"},{"filename":"srv","flags":"drwxr-xr-x.","links":2,"owner":"root","group":"root","size":6,"date":"Apr 10 2018"},{"filename":"sys","flags":"dr-xr-xr-x.","links":13,"owner":"root","group":"root","size":0,"date":"Oct 25 18:21"},{"filename":"tmp","flags":"drwxrwxrwt.","links":19,"owner":"root","group":"root","size":4,"date":"Oct 26 10:14"},{"filename":"usr","flags":"drwxr-xr-x.","links":13,"owner":"root","group":"root","size":155,"date":"Aug 15 10:53"},{"filename":"var","flags":"drwxr-xr-x.","links":19,"owner":"root","group":"root","size":267,"date":"Aug 15 10:57"}]

tests/fixtures/centos-7.7/lsusb-v-single.json

@@ -0,0 +1 @@
+[{"bus":"003","device":"090","id":"1915:521a","description":"Nordic Semiconductor ASA nRF52 USB CDC BLE Demo","device_descriptor":{"bLength":{"value":"18"},"bDescriptorType":{"value":"1"},"bcdUSB":{"value":"2.00"},"bDeviceClass":{"value":"0"},"bDeviceSubClass":{"value":"0"},"bDeviceProtocol":{"value":"0"},"bMaxPacketSize0":{"value":"64"},"idVendor":{"value":"0x1915","description":"Nordic Semiconductor ASA"},"idProduct":{"value":"0x521a"},"bcdDevice":{"value":"1.00"},"iManufacturer":{"value":"1","description":"Nordic Semiconductor"},"iProduct":{"value":"2","description":"nRF52 USB CDC BLE Demo"},"iSerial":{"value":"3","description":"F42DE60BE261"},"bNumConfigurations":{"value":"1"},"configuration_descriptor":{"bLength":{"value":"9"},"bDescriptorType":{"value":"2"},"wTotalLength":{"value":"0x004b"},"bNumInterfaces":{"value":"2"},"bConfigurationValue":{"value":"1"},"iConfiguration":{"value":"4"},"bmAttributes":{"value":"0xc0","attributes":["Self Powered"]},"MaxPower":{"description":"100mA"},"interface_association":{"bLength":{"value":"8"},"bDescriptorType":{"value":"11"},"bFirstInterface":{"value":"0"},"bInterfaceCount":{"value":"2"},"bFunctionClass":{"value":"2","description":"Communications"},"bFunctionSubClass":{"value":"2","description":"Abstract (modem)"},"bFunctionProtocol":{"value":"1","description":"AT-commands (v.25ter)"},"iFunction":{"value":"0"}},"interface_descriptors":[{"bLength":{"value":"9"},"bDescriptorType":{"value":"4"},"bInterfaceNumber":{"value":"0"},"bAlternateSetting":{"value":"0"},"bNumEndpoints":{"value":"1"},"bInterfaceClass":{"value":"2","description":"Communications"},"bInterfaceSubClass":{"value":"2","description":"Abstract (modem)"},"bInterfaceProtocol":{"value":"1","description":"AT-commands (v.25ter)"},"iInterface":{"value":"0"},"cdc_header":{"bcdCDC":{"value":"1.10"}},"cdc_call_management":{"bmCapabilities":{"value":"0x03","attributes":["call management","use DataInterface"]},"bDataInterface":{"value":"1"}},"cdc_acm":{"bmCapabilities":{"value":"0x02","attributes":["line coding and serial state"]}},"cdc_union":{"bMasterInterface":{"value":"0"},"bSlaveInterface":{"value":"1"}},"endpoint_descriptors":[{"bLength":{"value":"7"},"bDescriptorType":{"value":"5"},"bEndpointAddress":{"value":"0x82","description":"EP 2 IN"},"bmAttributes":{"value":"3","attributes":["Transfer Type  Interrupt","Synch Type  None","Usage Type  Data"]},"wMaxPacketSize":{"value":"0x0040","description":"1x 64 bytes"},"bInterval":{"value":"16"}}]},{"bLength":{"value":"9"},"bDescriptorType":{"value":"4"},"bInterfaceNumber":{"value":"1"},"bAlternateSetting":{"value":"0"},"bNumEndpoints":{"value":"2"},"bInterfaceClass":{"value":"10","description":"CDC Data"},"bInterfaceSubClass":{"value":"0"},"bInterfaceProtocol":{"value":"0"},"iInterface":{"value":"0"},"endpoint_descriptors":[{"bLength":{"value":"7"},"bDescriptorType":{"value":"5"},"bEndpointAddress":{"value":"0x81","description":"EP 1 IN"},"bmAttributes":{"value":"2","attributes":["Transfer Type  Bulk","Synch Type  None","Usage Type  Data"]},"wMaxPacketSize":{"value":"0x0040","description":"1x 64 bytes"},"bInterval":{"value":"0"}},{"bLength":{"value":"7"},"bDescriptorType":{"value":"5"},"bEndpointAddress":{"value":"0x01","description":"EP 1 OUT"},"bmAttributes":{"value":"2","attributes":["Transfer Type  Bulk","Synch Type  None","Usage Type  Data"]},"wMaxPacketSize":{"value":"0x0040","description":"1x 64 bytes"},"bInterval":{"value":"0"}}]}]}}}]

tests/fixtures/centos-7.7/lsusb-v-single.out

@@ -0,0 +1,100 @@
+Bus 003 Device 090: ID 1915:521a Nordic Semiconductor ASA nRF52 USB CDC BLE Demo
+Couldn't open device, some information will be missing
+Device Descriptor:
+  bLength                18
+  bDescriptorType         1
+  bcdUSB               2.00
+  bDeviceClass            0 
+  bDeviceSubClass         0 
+  bDeviceProtocol         0 
+  bMaxPacketSize0        64
+  idVendor           0x1915 Nordic Semiconductor ASA
+  idProduct          0x521a 
+  bcdDevice            1.00
+  iManufacturer           1 Nordic Semiconductor
+  iProduct                2 nRF52 USB CDC BLE Demo
+  iSerial                 3 F42DE60BE261
+  bNumConfigurations      1
+  Configuration Descriptor:
+    bLength                 9
+    bDescriptorType         2
+    wTotalLength       0x004b
+    bNumInterfaces          2
+    bConfigurationValue     1
+    iConfiguration          4 
+    bmAttributes         0xc0
+      Self Powered
+    MaxPower              100mA
+    Interface Association:
+      bLength                 8
+      bDescriptorType        11
+      bFirstInterface         0
+      bInterfaceCount         2
+      bFunctionClass          2 Communications
+      bFunctionSubClass       2 Abstract (modem)
+      bFunctionProtocol       1 AT-commands (v.25ter)
+      iFunction               0 
+    Interface Descriptor:
+      bLength                 9
+      bDescriptorType         4
+      bInterfaceNumber        0
+      bAlternateSetting       0
+      bNumEndpoints           1
+      bInterfaceClass         2 Communications
+      bInterfaceSubClass      2 Abstract (modem)
+      bInterfaceProtocol      1 AT-commands (v.25ter)
+      iInterface              0 
+      CDC Header:
+        bcdCDC               1.10
+      CDC Call Management:
+        bmCapabilities       0x03
+          call management
+          use DataInterface
+        bDataInterface          1
+      CDC ACM:
+        bmCapabilities       0x02
+          line coding and serial state
+      CDC Union:
+        bMasterInterface        0
+        bSlaveInterface         1 
+      Endpoint Descriptor:
+        bLength                 7
+        bDescriptorType         5
+        bEndpointAddress     0x82  EP 2 IN
+        bmAttributes            3
+          Transfer Type            Interrupt
+          Synch Type               None
+          Usage Type               Data
+        wMaxPacketSize     0x0040  1x 64 bytes
+        bInterval              16
+    Interface Descriptor:
+      bLength                 9
+      bDescriptorType         4
+      bInterfaceNumber        1
+      bAlternateSetting       0
+      bNumEndpoints           2
+      bInterfaceClass        10 CDC Data
+      bInterfaceSubClass      0 
+      bInterfaceProtocol      0 
+      iInterface              0 
+      Endpoint Descriptor:
+        bLength                 7
+        bDescriptorType         5
+        bEndpointAddress     0x81  EP 1 IN
+        bmAttributes            2
+          Transfer Type            Bulk
+          Synch Type               None
+          Usage Type               Data
+        wMaxPacketSize     0x0040  1x 64 bytes
+        bInterval               0
+      Endpoint Descriptor:
+        bLength                 7
+        bDescriptorType         5
+        bEndpointAddress     0x01  EP 1 OUT
+        bmAttributes            2
+          Transfer Type            Bulk
+          Synch Type               None
+          Usage Type               Data
+        wMaxPacketSize     0x0040  1x 64 bytes
+        bInterval               0
+

tests/fixtures/centos-7.7/lsusb-v.out

@@ -0,0 +1,509 @@
+
+Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
+Device Descriptor:
+  bLength                18
+  bDescriptorType         1
+  bcdUSB               2.00
+  bDeviceClass            9 Hub
+  bDeviceSubClass         0 Unused
+  bDeviceProtocol         0 Full speed (or root) hub
+  bMaxPacketSize0        64
+  idVendor           0x1d6b Linux Foundation
+  idProduct          0x0002 2.0 root hub
+  bcdDevice            3.10
+  iManufacturer           3 Linux 3.10.0-1062.1.2.el7.x86_64 ehci_hcd
+  iProduct                2 EHCI Host Controller
+  iSerial                 1 0000:02:02.0
+  bNumConfigurations      1
+  Configuration Descriptor:
+    bLength                 9
+    bDescriptorType         2
+    wTotalLength           25
+    bNumInterfaces          1
+    bConfigurationValue     1
+    iConfiguration          0 
+    bmAttributes         0xe0
+      Self Powered
+      Remote Wakeup
+    MaxPower                0mA
+    Interface Descriptor:
+      bLength                 9
+      bDescriptorType         4
+      bInterfaceNumber        0
+      bAlternateSetting       0
+      bNumEndpoints           1
+      bInterfaceClass         9 Hub
+      bInterfaceSubClass      0 Unused
+      bInterfaceProtocol      0 Full speed (or root) hub
+      iInterface              0 
+      Endpoint Descriptor:
+        bLength                 7
+        bDescriptorType         5
+        bEndpointAddress     0x81  EP 1 IN
+        bmAttributes            3
+          Transfer Type            Interrupt
+          Synch Type               None
+          Usage Type               Data
+        wMaxPacketSize     0x0004  1x 4 bytes
+        bInterval              12
+Hub Descriptor:
+  bLength               9
+  bDescriptorType      41
+  nNbrPorts             6
+  wHubCharacteristic 0x000a
+    No power switching (usb 1.0)
+    Per-port overcurrent protection
+  bPwrOn2PwrGood       10 * 2 milli seconds
+  bHubContrCurrent      0 milli Ampere
+  DeviceRemovable    0x00
+  PortPwrCtrlMask    0xff
+ Hub Port Status:
+   Port 1: 0000.0100 power
+   Port 2: 0000.0100 power
+   Port 3: 0000.0100 power
+   Port 4: 0000.0100 power
+   Port 5: 0000.0100 power
+   Port 6: 0000.0100 power
+Device Status:     0x0001
+  Self Powered
+
+Bus 002 Device 004: ID 0e0f:0008 VMware, Inc. 
+Device Descriptor:
+  bLength                18
+  bDescriptorType         1
+  bcdUSB               2.00
+  bDeviceClass          224 Wireless
+  bDeviceSubClass         1 Radio Frequency
+  bDeviceProtocol         1 Bluetooth
+  bMaxPacketSize0        64
+  idVendor           0x0e0f VMware, Inc.
+  idProduct          0x0008 
+  bcdDevice            1.00
+  iManufacturer           1 VMware
+  iProduct                2 Virtual Bluetooth Adapter
+  iSerial                 3 000650268328
+  bNumConfigurations      1
+  Configuration Descriptor:
+    bLength                 9
+    bDescriptorType         2
+    wTotalLength          177
+    bNumInterfaces          2
+    bConfigurationValue     1
+    iConfiguration          0 
+    bmAttributes         0xc0
+      Self Powered
+    MaxPower                0mA
+    Interface Descriptor:
+      bLength                 9
+      bDescriptorType         4
+      bInterfaceNumber        0
+      bAlternateSetting       0
+      bNumEndpoints           3
+      bInterfaceClass       224 Wireless
+      bInterfaceSubClass      1 Radio Frequency
+      bInterfaceProtocol      1 Bluetooth
+      iInterface              0 
+      Endpoint Descriptor:
+        bLength                 7
+        bDescriptorType         5
+        bEndpointAddress     0x81  EP 1 IN
+        bmAttributes            3
+          Transfer Type            Interrupt
+          Synch Type               None
+          Usage Type               Data
+        wMaxPacketSize     0x0010  1x 16 bytes
+        bInterval               1
+      Endpoint Descriptor:
+        bLength                 7
+        bDescriptorType         5
+        bEndpointAddress     0x02  EP 2 OUT
+        bmAttributes            2
+          Transfer Type            Bulk
+          Synch Type               None
+          Usage Type               Data
+        wMaxPacketSize     0x0040  1x 64 bytes
+        bInterval               0
+      Endpoint Descriptor:
+        bLength                 7
+        bDescriptorType         5
+        bEndpointAddress     0x82  EP 2 IN
+        bmAttributes            2
+          Transfer Type            Bulk
+          Synch Type               None
+          Usage Type               Data
+        wMaxPacketSize     0x0040  1x 64 bytes
+        bInterval               0
+    Interface Descriptor:
+      bLength                 9
+      bDescriptorType         4
+      bInterfaceNumber        1
+      bAlternateSetting       0
+      bNumEndpoints           2
+      bInterfaceClass       224 Wireless
+      bInterfaceSubClass      1 Radio Frequency
+      bInterfaceProtocol      1 Bluetooth
+      iInterface              0 
+      Endpoint Descriptor:
+        bLength                 7
+        bDescriptorType         5
+        bEndpointAddress     0x03  EP 3 OUT
+        bmAttributes            1
+          Transfer Type            Isochronous
+          Synch Type               None
+          Usage Type               Data
+        wMaxPacketSize     0x0000  1x 0 bytes
+        bInterval               1
+      Endpoint Descriptor:
+        bLength                 7
+        bDescriptorType         5
+        bEndpointAddress     0x83  EP 3 IN
+        bmAttributes            1
+          Transfer Type            Isochronous
+          Synch Type               None
+          Usage Type               Data
+        wMaxPacketSize     0x0000  1x 0 bytes
+        bInterval               1
+    Interface Descriptor:
+      bLength                 9
+      bDescriptorType         4
+      bInterfaceNumber        1
+      bAlternateSetting       1
+      bNumEndpoints           2
+      bInterfaceClass       224 Wireless
+      bInterfaceSubClass      1 Radio Frequency
+      bInterfaceProtocol      1 Bluetooth
+      iInterface              0 
+      Endpoint Descriptor:
+        bLength                 7
+        bDescriptorType         5
+        bEndpointAddress     0x03  EP 3 OUT
+        bmAttributes            1
+          Transfer Type            Isochronous
+          Synch Type               None
+          Usage Type               Data
+        wMaxPacketSize     0x0009  1x 9 bytes
+        bInterval               1
+      Endpoint Descriptor:
+        bLength                 7
+        bDescriptorType         5
+        bEndpointAddress     0x83  EP 3 IN
+        bmAttributes            1
+          Transfer Type            Isochronous
+          Synch Type               None
+          Usage Type               Data
+        wMaxPacketSize     0x0009  1x 9 bytes
+        bInterval               1
+    Interface Descriptor:
+      bLength                 9
+      bDescriptorType         4
+      bInterfaceNumber        1
+      bAlternateSetting       2
+      bNumEndpoints           2
+      bInterfaceClass       224 Wireless
+      bInterfaceSubClass      1 Radio Frequency
+      bInterfaceProtocol      1 Bluetooth
+      iInterface              0 
+      Endpoint Descriptor:
+        bLength                 7
+        bDescriptorType         5
+        bEndpointAddress     0x03  EP 3 OUT
+        bmAttributes            1
+          Transfer Type            Isochronous
+          Synch Type               None
+          Usage Type               Data
+        wMaxPacketSize     0x0011  1x 17 bytes
+        bInterval               1
+      Endpoint Descriptor:
+        bLength                 7
+        bDescriptorType         5
+        bEndpointAddress     0x83  EP 3 IN
+        bmAttributes            1
+          Transfer Type            Isochronous
+          Synch Type               None
+          Usage Type               Data
+        wMaxPacketSize     0x0011  1x 17 bytes
+        bInterval               1
+    Interface Descriptor:
+      bLength                 9
+      bDescriptorType         4
+      bInterfaceNumber        1
+      bAlternateSetting       3
+      bNumEndpoints           2
+      bInterfaceClass       224 Wireless
+      bInterfaceSubClass      1 Radio Frequency
+      bInterfaceProtocol      1 Bluetooth
+      iInterface              0 
+      Endpoint Descriptor:
+        bLength                 7
+        bDescriptorType         5
+        bEndpointAddress     0x03  EP 3 OUT
+        bmAttributes            1
+          Transfer Type            Isochronous
+          Synch Type               None
+          Usage Type               Data
+        wMaxPacketSize     0x0019  1x 25 bytes
+        bInterval               1
+      Endpoint Descriptor:
+        bLength                 7
+        bDescriptorType         5
+        bEndpointAddress     0x83  EP 3 IN
+        bmAttributes            1
+          Transfer Type            Isochronous
+          Synch Type               None
+          Usage Type               Data
+        wMaxPacketSize     0x0019  1x 25 bytes
+        bInterval               1
+    Interface Descriptor:
+      bLength                 9
+      bDescriptorType         4
+      bInterfaceNumber        1
+      bAlternateSetting       4
+      bNumEndpoints           2
+      bInterfaceClass       224 Wireless
+      bInterfaceSubClass      1 Radio Frequency
+      bInterfaceProtocol      1 Bluetooth
+      iInterface              0 
+      Endpoint Descriptor:
+        bLength                 7
+        bDescriptorType         5
+        bEndpointAddress     0x03  EP 3 OUT
+        bmAttributes            1
+          Transfer Type            Isochronous
+          Synch Type               None
+          Usage Type               Data
+        wMaxPacketSize     0x0021  1x 33 bytes
+        bInterval               1
+      Endpoint Descriptor:
+        bLength                 7
+        bDescriptorType         5
+        bEndpointAddress     0x83  EP 3 IN
+        bmAttributes            1
+          Transfer Type            Isochronous
+          Synch Type               None
+          Usage Type               Data
+        wMaxPacketSize     0x0021  1x 33 bytes
+        bInterval               1
+    Interface Descriptor:
+      bLength                 9
+      bDescriptorType         4
+      bInterfaceNumber        1
+      bAlternateSetting       5
+      bNumEndpoints           2
+      bInterfaceClass       224 Wireless
+      bInterfaceSubClass      1 Radio Frequency
+      bInterfaceProtocol      1 Bluetooth
+      iInterface              0 
+      Endpoint Descriptor:
+        bLength                 7
+        bDescriptorType         5
+        bEndpointAddress     0x03  EP 3 OUT
+        bmAttributes            1
+          Transfer Type            Isochronous
+          Synch Type               None
+          Usage Type               Data
+        wMaxPacketSize     0x0031  1x 49 bytes
+        bInterval               1
+      Endpoint Descriptor:
+        bLength                 7
+        bDescriptorType         5
+        bEndpointAddress     0x83  EP 3 IN
+        bmAttributes            1
+          Transfer Type            Isochronous
+          Synch Type               None
+          Usage Type               Data
+        wMaxPacketSize     0x0031  1x 49 bytes
+        bInterval               1
+Device Status:     0x0001
+  Self Powered
+
+Bus 002 Device 003: ID 0e0f:0002 VMware, Inc. Virtual USB Hub
+Device Descriptor:
+  bLength                18
+  bDescriptorType         1
+  bcdUSB               1.10
+  bDeviceClass            9 Hub
+  bDeviceSubClass         0 Unused
+  bDeviceProtocol         0 Full speed (or root) hub
+  bMaxPacketSize0         8
+  idVendor           0x0e0f VMware, Inc.
+  idProduct          0x0002 Virtual USB Hub
+  bcdDevice            1.00
+  iManufacturer           1 VMware, Inc.
+  iProduct                2 VMware Virtual USB Hub
+  iSerial                 0 
+  bNumConfigurations      1
+  Configuration Descriptor:
+    bLength                 9
+    bDescriptorType         2
+    wTotalLength           25
+    bNumInterfaces          1
+    bConfigurationValue     1
+    iConfiguration          1 VMware, Inc.
+    bmAttributes         0xe0
+      Self Powered
+      Remote Wakeup
+    MaxPower                0mA
+    Interface Descriptor:
+      bLength                 9
+      bDescriptorType         4
+      bInterfaceNumber        0
+      bAlternateSetting       0
+      bNumEndpoints           1
+      bInterfaceClass         9 Hub
+      bInterfaceSubClass      0 Unused
+      bInterfaceProtocol      0 Full speed (or root) hub
+      iInterface              1 VMware, Inc.
+      Endpoint Descriptor:
+        bLength                 7
+        bDescriptorType         5
+        bEndpointAddress     0x81  EP 1 IN
+        bmAttributes            3
+          Transfer Type            Interrupt
+          Synch Type               None
+          Usage Type               Data
+        wMaxPacketSize     0x0001  1x 1 bytes
+        bInterval             255
+Hub Descriptor:
+  bLength               9
+  bDescriptorType      41
+  nNbrPorts             7
+  wHubCharacteristic 0x0009
+    Per-port power switching
+    Per-port overcurrent protection
+  bPwrOn2PwrGood       50 * 2 milli seconds
+  bHubContrCurrent    100 milli Ampere
+  DeviceRemovable    0x00
+  PortPwrCtrlMask    0xfe
+ Hub Port Status:
+   Port 1: 0000.0103 power enable connect
+   Port 2: 0000.0100 power
+   Port 3: 0000.0100 power
+   Port 4: 0000.0100 power
+   Port 5: 0000.0100 power
+   Port 6: 0000.0100 power
+   Port 7: 0000.0100 power
+Device Status:     0x2909
+  Self Powered
+
+Bus 002 Device 002: ID 0e0f:0003 VMware, Inc. Virtual Mouse
+Device Descriptor:
+  bLength                18
+  bDescriptorType         1
+  bcdUSB               1.10
+  bDeviceClass            0 (Defined at Interface level)
+  bDeviceSubClass         0 
+  bDeviceProtocol         0 
+  bMaxPacketSize0         8
+  idVendor           0x0e0f VMware, Inc.
+  idProduct          0x0003 Virtual Mouse
+  bcdDevice            1.03
+  iManufacturer           1 VMware
+  iProduct                2 VMware Virtual USB Mouse
+  iSerial                 0 
+  bNumConfigurations      1
+  Configuration Descriptor:
+    bLength                 9
+    bDescriptorType         2
+    wTotalLength           34
+    bNumInterfaces          1
+    bConfigurationValue     1
+    iConfiguration          1 VMware
+    bmAttributes         0xc0
+      Self Powered
+    MaxPower                0mA
+    Interface Descriptor:
+      bLength                 9
+      bDescriptorType         4
+      bInterfaceNumber        0
+      bAlternateSetting       0
+      bNumEndpoints           1
+      bInterfaceClass         3 Human Interface Device
+      bInterfaceSubClass      1 Boot Interface Subclass
+      bInterfaceProtocol      2 Mouse
+      iInterface              1 VMware
+        HID Device Descriptor:
+          bLength                 9
+          bDescriptorType        33
+          bcdHID               1.10
+          bCountryCode            0 Not supported
+          bNumDescriptors         1
+          bDescriptorType        34 Report
+          wDescriptorLength      46
+         Report Descriptors: 
+           ** UNAVAILABLE **
+      Endpoint Descriptor:
+        bLength                 7
+        bDescriptorType         5
+        bEndpointAddress     0x81  EP 1 IN
+        bmAttributes            3
+          Transfer Type            Interrupt
+          Synch Type               None
+          Usage Type               Data
+        wMaxPacketSize     0x0008  1x 8 bytes
+        bInterval               1
+Device Status:     0x0001
+  Self Powered
+
+Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
+Device Descriptor:
+  bLength                18
+  bDescriptorType         1
+  bcdUSB               1.10
+  bDeviceClass            9 Hub
+  bDeviceSubClass         0 Unused
+  bDeviceProtocol         0 Full speed (or root) hub
+  bMaxPacketSize0        64
+  idVendor           0x1d6b Linux Foundation
+  idProduct          0x0001 1.1 root hub
+  bcdDevice            3.10
+  iManufacturer           3 Linux 3.10.0-1062.1.2.el7.x86_64 uhci_hcd
+  iProduct                2 UHCI Host Controller
+  iSerial                 1 0000:02:00.0
+  bNumConfigurations      1
+  Configuration Descriptor:
+    bLength                 9
+    bDescriptorType         2
+    wTotalLength           25
+    bNumInterfaces          1
+    bConfigurationValue     1
+    iConfiguration          0 
+    bmAttributes         0xe0
+      Self Powered
+      Remote Wakeup
+    MaxPower                0mA
+    Interface Descriptor:
+      bLength                 9
+      bDescriptorType         4
+      bInterfaceNumber        0
+      bAlternateSetting       0
+      bNumEndpoints           1
+      bInterfaceClass         9 Hub
+      bInterfaceSubClass      0 Unused
+      bInterfaceProtocol      0 Full speed (or root) hub
+      iInterface              0 
+      Endpoint Descriptor:
+        bLength                 7
+        bDescriptorType         5
+        bEndpointAddress     0x81  EP 1 IN
+        bmAttributes            3
+          Transfer Type            Interrupt
+          Synch Type               None
+          Usage Type               Data
+        wMaxPacketSize     0x0002  1x 2 bytes
+        bInterval             255
+Hub Descriptor:
+  bLength               9
+  bDescriptorType      41
+  nNbrPorts             2
+  wHubCharacteristic 0x000a
+    No power switching (usb 1.0)
+    Per-port overcurrent protection
+  bPwrOn2PwrGood        1 * 2 milli seconds
+  bHubContrCurrent      0 milli Ampere
+  DeviceRemovable    0x00
+  PortPwrCtrlMask    0xff
+ Hub Port Status:
+   Port 1: 0000.0103 power enable connect
+   Port 2: 0000.0103 power enable connect
+Device Status:     0x0001
+  Self Powered

tests/fixtures/centos-7.7/lsusb.json

@@ -0,0 +1 @@
+[{"bus":"001","device":"001","id":"1d6b:0002","description":"Linux Foundation 2.0 root hub"},{"bus":"002","device":"004","id":"0e0f:0008","description":"VMware, Inc."},{"bus":"002","device":"003","id":"0e0f:0002","description":"VMware, Inc. Virtual USB Hub"},{"bus":"002","device":"002","id":"0e0f:0003","description":"VMware, Inc. Virtual Mouse"},{"bus":"002","device":"001","id":"1d6b:0001","description":"Linux Foundation 1.1 root hub"}]

tests/fixtures/centos-7.7/lsusb.out

@@ -0,0 +1,5 @@
+Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
+Bus 002 Device 004: ID 0e0f:0008 VMware, Inc. 
+Bus 002 Device 003: ID 0e0f:0002 VMware, Inc. Virtual USB Hub
+Bus 002 Device 002: ID 0e0f:0003 VMware, Inc. Virtual Mouse
+Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub

tests/fixtures/centos-7.7/ping-hostname-O-D-p-s-streaming.json

@@ -0,0 +1 @@
+[{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978465.914536,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":1,"ttl":59,"time_ms":31.4,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978465.993009,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":2,"ttl":59,"time_ms":30.3,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978467.010196,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":3,"ttl":59,"time_ms":32.0,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978468.033743,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":4,"ttl":59,"time_ms":38.8,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978469.051227,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":5,"ttl":59,"time_ms":38.0,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978470.048764,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":6,"ttl":59,"time_ms":29.9,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978471.051945,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":7,"ttl":59,"time_ms":28.9,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978472.064206,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":8,"ttl":59,"time_ms":37.4,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978473.062587,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":9,"ttl":59,"time_ms":31.5,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978474.074343,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":10,"ttl":59,"time_ms":38.3,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978475.079703,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":11,"ttl":59,"time_ms":38.8,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978476.076383,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":12,"ttl":59,"time_ms":30.7,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978477.084119,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":13,"ttl":59,"time_ms":30.7,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978478.092207,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":14,"ttl":59,"time_ms":31.6,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978479.104358,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":15,"ttl":59,"time_ms":37.7,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978480.106907,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":16,"ttl":59,"time_ms":37.5,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978481.11558,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":17,"ttl":59,"time_ms":37.3,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978482.119872,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":18,"ttl":59,"time_ms":33.8,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978483.131901,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":19,"ttl":59,"time_ms":37.0,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978484.141117,"response_bytes":1408,"response_ip":"151.101.189.67","icmp_seq":20,"ttl":59,"time_ms":36.9,"duplicate":false},{"type":"summary","destination_ip":"151.101.189.67","sent_bytes":1400,"pattern":"0xabcd","packets_transmitted":20,"packets_received":20,"packet_loss_percent":0.0,"duplicates":0,"time_ms":19146.0,"round_trip_ms_min":28.96,"round_trip_ms_avg":34.468,"round_trip_ms_max":38.892,"round_trip_ms_stddev":3.497}]

tests/fixtures/centos-7.7/ping-hostname-O-p-streaming.json

@@ -0,0 +1 @@
+[{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":1,"ttl":59,"time_ms":24.4,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":2,"ttl":59,"time_ms":23.3,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":3,"ttl":59,"time_ms":32.6,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":4,"ttl":59,"time_ms":32.6,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":5,"ttl":59,"time_ms":26.9,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":6,"ttl":59,"time_ms":24.1,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":7,"ttl":59,"time_ms":24.6,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":8,"ttl":59,"time_ms":33.9,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":9,"ttl":59,"time_ms":32.7,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":10,"ttl":59,"time_ms":31.2,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":11,"ttl":59,"time_ms":25.7,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":12,"ttl":59,"time_ms":33.8,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":13,"ttl":59,"time_ms":23.7,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":14,"ttl":59,"time_ms":23.9,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":15,"ttl":59,"time_ms":33.6,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":16,"ttl":59,"time_ms":24.5,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":17,"ttl":59,"time_ms":30.1,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":18,"ttl":59,"time_ms":24.1,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":19,"ttl":59,"time_ms":32.2,"duplicate":false},{"type":"reply","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"151.101.129.67","icmp_seq":20,"ttl":59,"time_ms":31.0,"duplicate":false},{"type":"summary","destination_ip":"151.101.129.67","sent_bytes":56,"pattern":"0xabcd","packets_transmitted":20,"packets_received":20,"packet_loss_percent":0.0,"duplicates":0,"time_ms":19233.0,"round_trip_ms_min":23.359,"round_trip_ms_avg":28.495,"round_trip_ms_max":33.979,"round_trip_ms_stddev":4.081}]

tests/fixtures/centos-7.7/ping-hostname-O-streaming.json

@@ -0,0 +1 @@
+[{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":1,"ttl":59,"time_ms":29.6,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":2,"ttl":59,"time_ms":30.1,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":3,"ttl":59,"time_ms":35.5,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":4,"ttl":59,"time_ms":35.5,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":5,"ttl":59,"time_ms":34.9,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":6,"ttl":59,"time_ms":29.9,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":7,"ttl":59,"time_ms":27.6,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":8,"ttl":59,"time_ms":28.6,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":9,"ttl":59,"time_ms":35.2,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":10,"ttl":59,"time_ms":34.4,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":11,"ttl":59,"time_ms":35.9,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":12,"ttl":59,"time_ms":35.8,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":13,"ttl":59,"time_ms":34.4,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":14,"ttl":59,"time_ms":35.5,"duplicate":false},{"type":"timeout","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"icmp_seq":15},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":16,"ttl":59,"time_ms":36.6,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":17,"ttl":59,"time_ms":34.6,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":18,"ttl":59,"time_ms":34.6,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":19,"ttl":59,"time_ms":36.7,"duplicate":false},{"type":"reply","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"151.101.189.67","icmp_seq":20,"ttl":59,"time_ms":34.3,"duplicate":false},{"type":"summary","destination_ip":"151.101.189.67","sent_bytes":56,"pattern":null,"packets_transmitted":20,"packets_received":19,"packet_loss_percent":5.0,"duplicates":0,"time_ms":19125.0,"round_trip_ms_min":27.656,"round_trip_ms_avg":33.717,"round_trip_ms_max":36.758,"round_trip_ms_stddev":2.814}]

tests/fixtures/centos-7.7/ping-ip-O-D-streaming.json

@@ -0,0 +1 @@
+[{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037214.261953,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":1,"ttl":64,"time_ms":0.041,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037215.264798,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":2,"ttl":64,"time_ms":0.048,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037216.272296,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":3,"ttl":64,"time_ms":0.047,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037217.275851,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":4,"ttl":64,"time_ms":0.062,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037218.284242,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":5,"ttl":64,"time_ms":0.045,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037219.283712,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":6,"ttl":64,"time_ms":0.043,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037220.290949,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":7,"ttl":64,"time_ms":0.046,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037221.295962,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":8,"ttl":64,"time_ms":0.044,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037222.30702,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":9,"ttl":64,"time_ms":0.048,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037223.313919,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":10,"ttl":64,"time_ms":0.081,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037224.313679,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":11,"ttl":64,"time_ms":0.043,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037225.320748,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":12,"ttl":64,"time_ms":0.044,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037226.324322,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":13,"ttl":64,"time_ms":0.045,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037227.325835,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":14,"ttl":64,"time_ms":0.046,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037228.327028,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":15,"ttl":64,"time_ms":0.046,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037229.329891,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":16,"ttl":64,"time_ms":0.052,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037230.333891,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":17,"ttl":64,"time_ms":0.044,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037231.338137,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":18,"ttl":64,"time_ms":0.046,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037232.340475,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":19,"ttl":64,"time_ms":0.048,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":1595037233.343058,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":20,"ttl":64,"time_ms":0.045,"duplicate":false},{"type":"summary","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"packets_transmitted":20,"packets_received":20,"packet_loss_percent":0.0,"duplicates":0,"time_ms":19081.0,"round_trip_ms_min":0.041,"round_trip_ms_avg":0.048,"round_trip_ms_max":0.081,"round_trip_ms_stddev":0.009}]

tests/fixtures/centos-7.7/ping-ip-O-streaming-ignore-exceptions.json

@@ -0,0 +1 @@
+[{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":1,"ttl":64,"time_ms":0.038,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":2,"ttl":64,"time_ms":0.043,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":3,"ttl":64,"time_ms":0.044,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":4,"ttl":64,"time_ms":0.052,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":5,"ttl":64,"time_ms":0.08,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":6,"ttl":64,"time_ms":0.043,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":7,"ttl":64,"time_ms":0.047,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":8,"ttl":64,"time_ms":0.04,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":9,"ttl":64,"time_ms":0.052,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":10,"ttl":64,"time_ms":0.044,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":11,"ttl":64,"time_ms":0.043,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":12,"ttl":64,"time_ms":0.043,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":13,"ttl":64,"time_ms":0.05,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":14,"ttl":64,"time_ms":0.045,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":15,"ttl":64,"time_ms":0.062,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":16,"ttl":64,"time_ms":0.046,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":17,"ttl":64,"time_ms":0.046,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":18,"ttl":64,"time_ms":0.045,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":19,"ttl":64,"time_ms":0.044,"duplicate":false,"_jc_meta":{"success":true}},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":20,"ttl":64,"time_ms":0.044,"duplicate":false,"_jc_meta":{"success":true}},{"type":"summary","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"packets_transmitted":20,"packets_received":20,"packet_loss_percent":0.0,"duplicates":0,"time_ms":19070.0,"round_trip_ms_min":0.038,"round_trip_ms_avg":0.047,"round_trip_ms_max":0.08,"round_trip_ms_stddev":0.011,"_jc_meta":{"success":true}}]

tests/fixtures/centos-7.7/ping-ip-O-streaming.json

@@ -0,0 +1 @@
+[{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":1,"ttl":64,"time_ms":0.038,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":2,"ttl":64,"time_ms":0.043,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":3,"ttl":64,"time_ms":0.044,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":4,"ttl":64,"time_ms":0.052,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":5,"ttl":64,"time_ms":0.08,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":6,"ttl":64,"time_ms":0.043,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":7,"ttl":64,"time_ms":0.047,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":8,"ttl":64,"time_ms":0.04,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":9,"ttl":64,"time_ms":0.052,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":10,"ttl":64,"time_ms":0.044,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":11,"ttl":64,"time_ms":0.043,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":12,"ttl":64,"time_ms":0.043,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":13,"ttl":64,"time_ms":0.05,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":14,"ttl":64,"time_ms":0.045,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":15,"ttl":64,"time_ms":0.062,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":16,"ttl":64,"time_ms":0.046,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":17,"ttl":64,"time_ms":0.046,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":18,"ttl":64,"time_ms":0.045,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":19,"ttl":64,"time_ms":0.044,"duplicate":false},{"type":"reply","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"127.0.0.1","icmp_seq":20,"ttl":64,"time_ms":0.044,"duplicate":false},{"type":"summary","destination_ip":"127.0.0.1","sent_bytes":56,"pattern":null,"packets_transmitted":20,"packets_received":20,"packet_loss_percent":0.0,"duplicates":0,"time_ms":19070.0,"round_trip_ms_min":0.038,"round_trip_ms_avg":0.047,"round_trip_ms_max":0.08,"round_trip_ms_stddev":0.011}]

tests/fixtures/centos-7.7/ping-ip-dup-streaming.json

@@ -0,0 +1 @@
+[{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.221","icmp_seq":1,"ttl":64,"time_ms":0.586,"duplicate":false},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.88","icmp_seq":1,"ttl":64,"time_ms":382.0,"duplicate":true},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.78","icmp_seq":1,"ttl":128,"time_ms":382.0,"duplicate":true},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.217","icmp_seq":1,"ttl":255,"time_ms":387.0,"duplicate":true},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.186","icmp_seq":1,"ttl":64,"time_ms":389.0,"duplicate":true},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.89","icmp_seq":1,"ttl":64,"time_ms":389.0,"duplicate":true},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.75","icmp_seq":1,"ttl":64,"time_ms":584.0,"duplicate":true},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.221","icmp_seq":2,"ttl":64,"time_ms":0.861,"duplicate":false},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.78","icmp_seq":2,"ttl":128,"time_ms":4.17,"duplicate":true},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.88","icmp_seq":2,"ttl":64,"time_ms":4.19,"duplicate":true},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.89","icmp_seq":2,"ttl":64,"time_ms":12.7,"duplicate":true},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.81","icmp_seq":1,"ttl":64,"time_ms":1029.0,"duplicate":true},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.72","icmp_seq":1,"ttl":64,"time_ms":1276.0,"duplicate":true},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.251","icmp_seq":1,"ttl":64,"time_ms":1276.0,"duplicate":true},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.251","icmp_seq":2,"ttl":64,"time_ms":262.0,"duplicate":true},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.72","icmp_seq":2,"ttl":64,"time_ms":263.0,"duplicate":true},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.246","icmp_seq":2,"ttl":255,"time_ms":263.0,"duplicate":true},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.217","icmp_seq":2,"ttl":255,"time_ms":919.0,"duplicate":true},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.186","icmp_seq":2,"ttl":64,"time_ms":919.0,"duplicate":true},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.75","icmp_seq":2,"ttl":64,"time_ms":919.0,"duplicate":true},{"type":"reply","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"192.168.1.81","icmp_seq":2,"ttl":64,"time_ms":919.0,"duplicate":true},{"type":"summary","destination_ip":"192.168.1.255","sent_bytes":56,"pattern":null,"packets_transmitted":2,"packets_received":2,"packet_loss_percent":0.0,"duplicates":19,"time_ms":1013.0,"round_trip_ms_min":0.586,"round_trip_ms_avg":504.26,"round_trip_ms_max":1276.448,"round_trip_ms_stddev":417.208}]

tests/fixtures/centos-7.7/ping6-hostname-O-D-p-s-streaming.json

@@ -0,0 +1 @@
+[{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978345.609669,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":1,"ttl":59,"time_ms":32.4,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978346.58542,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":2,"ttl":59,"time_ms":39.9,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978347.594128,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":3,"ttl":59,"time_ms":42.3,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978348.595221,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":4,"ttl":59,"time_ms":40.2,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978349.600372,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":5,"ttl":59,"time_ms":43.2,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978350.590676,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":6,"ttl":59,"time_ms":31.8,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978351.601527,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":7,"ttl":59,"time_ms":41.8,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978352.604195,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":8,"ttl":59,"time_ms":41.7,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978353.607212,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":9,"ttl":59,"time_ms":42.0,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978354.610771,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":10,"ttl":59,"time_ms":40.7,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978355.613729,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":11,"ttl":59,"time_ms":40.4,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978356.611887,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":12,"ttl":59,"time_ms":32.6,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978357.62481,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":13,"ttl":59,"time_ms":40.1,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978358.629185,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":14,"ttl":59,"time_ms":42.0,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978359.634854,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":15,"ttl":59,"time_ms":41.2,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978360.638344,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":16,"ttl":59,"time_ms":40.6,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978361.640968,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":17,"ttl":59,"time_ms":40.7,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978362.645739,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":18,"ttl":59,"time_ms":39.9,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978363.6467,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":19,"ttl":59,"time_ms":37.5,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","timestamp":1594978364.650853,"response_bytes":1408,"response_ip":"2a04:4e42:2d::323","icmp_seq":20,"ttl":59,"time_ms":33.6,"duplicate":false},{"type":"summary","destination_ip":"2a04:4e42:2d::323","sent_bytes":1400,"pattern":"0xabcd","packets_transmitted":20,"packets_received":20,"packet_loss_percent":0.0,"duplicates":0,"time_ms":19077.0,"round_trip_ms_min":31.845,"round_trip_ms_avg":39.274,"round_trip_ms_max":43.243,"round_trip_ms_stddev":3.522}]

tests/fixtures/centos-7.7/ping6-hostname-O-p-streaming.json

@@ -0,0 +1 @@
+[{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":1,"ttl":59,"time_ms":30.9,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":2,"ttl":59,"time_ms":39.0,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":3,"ttl":59,"time_ms":32.6,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":4,"ttl":59,"time_ms":38.4,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":5,"ttl":59,"time_ms":38.8,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":6,"ttl":59,"time_ms":42.6,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":7,"ttl":59,"time_ms":30.7,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":8,"ttl":59,"time_ms":39.4,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":9,"ttl":59,"time_ms":39.3,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":10,"ttl":59,"time_ms":38.9,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":11,"ttl":59,"time_ms":38.6,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":12,"ttl":59,"time_ms":38.2,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":13,"ttl":59,"time_ms":39.6,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":14,"ttl":59,"time_ms":37.4,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":15,"ttl":59,"time_ms":33.7,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":16,"ttl":59,"time_ms":39.4,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":17,"ttl":59,"time_ms":38.9,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":18,"ttl":59,"time_ms":41.3,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":19,"ttl":59,"time_ms":32.2,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:2d::323","icmp_seq":20,"ttl":59,"time_ms":38.4,"duplicate":false},{"type":"summary","destination_ip":"2a04:4e42:2d::323","sent_bytes":56,"pattern":"0xabcd","packets_transmitted":20,"packets_received":20,"packet_loss_percent":0.0,"duplicates":0,"time_ms":19164.0,"round_trip_ms_min":30.757,"round_trip_ms_avg":37.455,"round_trip_ms_max":42.652,"round_trip_ms_stddev":3.338}]

tests/fixtures/centos-7.7/ping6-ip-O-D-p-streaming.json

@@ -0,0 +1 @@
+[{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976827.240914,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":1,"ttl":59,"time_ms":28.7,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976828.25493,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":2,"ttl":59,"time_ms":37.2,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976829.252877,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":3,"ttl":59,"time_ms":29.7,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976830.262654,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":4,"ttl":59,"time_ms":37.7,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976831.265626,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":5,"ttl":59,"time_ms":34.8,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976832.269834,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":6,"ttl":59,"time_ms":35.6,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976833.268059,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":7,"ttl":59,"time_ms":28.4,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976834.274292,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":8,"ttl":59,"time_ms":28.1,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976835.287123,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":9,"ttl":59,"time_ms":34.9,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976836.287707,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":10,"ttl":59,"time_ms":34.4,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976837.290589,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":11,"ttl":59,"time_ms":35.2,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976838.293514,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":12,"ttl":59,"time_ms":35.4,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976839.290914,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":13,"ttl":59,"time_ms":29.8,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976840.292897,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":14,"ttl":59,"time_ms":28.5,"duplicate":false},{"type":"timeout","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976842.269238,"icmp_seq":15},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976842.30145,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":16,"ttl":59,"time_ms":31.8,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976843.312998,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":17,"ttl":59,"time_ms":39.8,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976844.314228,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":18,"ttl":59,"time_ms":35.7,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976845.315518,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":19,"ttl":59,"time_ms":35.1,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":1594976846.321706,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":20,"ttl":59,"time_ms":35.4,"duplicate":false},{"type":"summary","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","packets_transmitted":20,"packets_received":19,"packet_loss_percent":5.0,"duplicates":0,"time_ms":19074.0,"round_trip_ms_min":28.15,"round_trip_ms_avg":33.534,"round_trip_ms_max":39.843,"round_trip_ms_stddev":3.489}]

tests/fixtures/centos-7.7/ping6-ip-O-p-streaming.json

@@ -0,0 +1 @@
+[{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":1,"ttl":59,"time_ms":27.9,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":2,"ttl":59,"time_ms":28.4,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":3,"ttl":59,"time_ms":36.0,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":4,"ttl":59,"time_ms":28.5,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":5,"ttl":59,"time_ms":35.8,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":6,"ttl":59,"time_ms":34.4,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":7,"ttl":59,"time_ms":30.7,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":8,"ttl":59,"time_ms":28.5,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":9,"ttl":59,"time_ms":36.5,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":10,"ttl":59,"time_ms":36.3,"duplicate":false},{"type":"timeout","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"icmp_seq":11},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":12,"ttl":59,"time_ms":37.4,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":13,"ttl":59,"time_ms":30.7,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":14,"ttl":59,"time_ms":36.5,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":15,"ttl":59,"time_ms":35.4,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":16,"ttl":59,"time_ms":36.3,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":17,"ttl":59,"time_ms":37.5,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":18,"ttl":59,"time_ms":36.2,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":19,"ttl":59,"time_ms":27.0,"duplicate":false},{"type":"reply","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","timestamp":null,"response_bytes":64,"response_ip":"2a04:4e42:600::323","icmp_seq":20,"ttl":59,"time_ms":38.1,"duplicate":false},{"type":"summary","destination_ip":"2a04:4e42:600::323","sent_bytes":56,"pattern":"0xabcd","packets_transmitted":20,"packets_received":19,"packet_loss_percent":5.0,"duplicates":0,"time_ms":19067.0,"round_trip_ms_min":27.064,"round_trip_ms_avg":33.626,"round_trip_ms_max":38.146,"round_trip_ms_stddev":3.803}]

tests/fixtures/centos-7.7/ping6-ip-dup-streaming.json

@@ -0,0 +1 @@
+[{"type":"reply","destination_ip":"ff02::1%ens33","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"fe80::c48:5896:526d:81ba%ens33","icmp_seq":1,"ttl":64,"time_ms":0.245,"duplicate":false},{"type":"reply","destination_ip":"ff02::1%ens33","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"fe80::feae:34ff:fea1:3a80%ens33","icmp_seq":1,"ttl":64,"time_ms":3.65,"duplicate":true},{"type":"reply","destination_ip":"ff02::1%ens33","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"fe80::c48:5896:526d:81ba%ens33","icmp_seq":2,"ttl":64,"time_ms":0.329,"duplicate":false},{"type":"reply","destination_ip":"ff02::1%ens33","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"fe80::feae:34ff:fea1:3a80%ens33","icmp_seq":2,"ttl":64,"time_ms":11.7,"duplicate":true},{"type":"reply","destination_ip":"ff02::1%ens33","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"fe80::c48:5896:526d:81ba%ens33","icmp_seq":3,"ttl":64,"time_ms":0.592,"duplicate":false},{"type":"reply","destination_ip":"ff02::1%ens33","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"fe80::feae:34ff:fea1:3a80%ens33","icmp_seq":3,"ttl":64,"time_ms":12.3,"duplicate":true},{"type":"reply","destination_ip":"ff02::1%ens33","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"fe80::c48:5896:526d:81ba%ens33","icmp_seq":4,"ttl":64,"time_ms":0.51,"duplicate":false},{"type":"reply","destination_ip":"ff02::1%ens33","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"fe80::feae:34ff:fea1:3a80%ens33","icmp_seq":4,"ttl":64,"time_ms":12.5,"duplicate":true},{"type":"reply","destination_ip":"ff02::1%ens33","sent_bytes":56,"pattern":null,"timestamp":null,"response_bytes":64,"response_ip":"fe80::c48:5896:526d:81ba%ens33","icmp_seq":5,"ttl":64,"time_ms":0.538,"duplicate":false},{"type":"summary","destination_ip":"ff02::1%ens33","sent_bytes":56,"pattern":null,"packets_transmitted":5,"packets_received":5,"packet_loss_percent":0.0,"duplicates":4,"time_ms":4017.0,"round_trip_ms_min":0.245,"round_trip_ms_avg":4.726,"round_trip_ms_max":12.568,"round_trip_ms_stddev":5.395}]

tests/fixtures/centos-7.7/sfdisk-d-multi.json

@@ -0,0 +1 @@
+[{"disk":"/dev/sda","units":"sectors","partitions":[{"device":"/dev/sda1","start":2048,"size":2097152,"id":"83","boot":true},{"device":"/dev/sda2","start":2099200,"size":39843840,"id":"8e","boot":false},{"device":"/dev/sda3","start":0,"size":0,"id":"0","boot":false},{"device":"/dev/sda4","start":0,"size":0,"id":"0","boot":false}]},{"disk":"/dev/sdb","units":"sectors","partitions":[{"device":"/dev/sdb1","start":2048,"size":2097152,"id":"83","boot":true},{"device":"/dev/sdb2","start":2099200,"size":39843840,"id":"8e","boot":false},{"device":"/dev/sdb3","start":0,"size":0,"id":"0","boot":false},{"device":"/dev/sdb4","start":0,"size":0,"id":"0","boot":false}]}]

tests/fixtures/centos-7.7/sfdisk-d-multi.out

@@ -0,0 +1,16 @@
+# partition table of /dev/sda
+unit: sectors
+
+/dev/sda1 : start=     2048, size=  2097152, Id=83, bootable
+/dev/sda2 : start=  2099200, size= 39843840, Id=8e
+/dev/sda3 : start=        0, size=        0, Id= 0
+/dev/sda4 : start=        0, size=        0, Id= 0
+
+# partition table of /dev/sdb
+unit: sectors
+
+/dev/sdb1 : start=     2048, size=  2097152, Id=83, bootable
+/dev/sdb2 : start=  2099200, size= 39843840, Id=8e
+/dev/sdb3 : start=        0, size=        0, Id= 0
+/dev/sdb4 : start=        0, size=        0, Id= 0
+

tests/fixtures/centos-7.7/sfdisk-d.json

@@ -0,0 +1 @@
+[{"disk":"/dev/sda","units":"sectors","partitions":[{"device":"/dev/sda1","start":2048,"size":2097152,"id":"83","boot":true},{"device":"/dev/sda2","start":2099200,"size":39843840,"id":"8e","boot":false},{"device":"/dev/sda3","start":0,"size":0,"id":"0","boot":false},{"device":"/dev/sda4","start":0,"size":0,"id":"0","boot":false}]}]

tests/fixtures/centos-7.7/sfdisk-d.out

@@ -0,0 +1,7 @@
+# partition table of /dev/sda
+unit: sectors
+
+/dev/sda1 : start=     2048, size=  2097152, Id=83, bootable
+/dev/sda2 : start=  2099200, size= 39843840, Id=8e
+/dev/sda3 : start=        0, size=        0, Id= 0
+/dev/sda4 : start=        0, size=        0, Id= 0

tests/fixtures/centos-7.7/sfdisk-l-multi.json

@@ -0,0 +1 @@
+[{"disk":"/dev/sda","cylinders":2610,"heads":255,"sectors_per_track":63,"units":"cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0","partitions":[{"device":"/dev/sda1","boot":true,"start":0,"end":130,"cyls":131,"blocks":1048576,"id":"83","system":"Linux"},{"device":"/dev/sda2","boot":false,"start":130,"end":2610,"cyls":2481,"blocks":19921920,"id":"8e","system":"Linux LVM"},{"device":"/dev/sda3","boot":false,"start":0,"end":null,"cyls":0,"blocks":0,"id":"0","system":"Empty"},{"device":"/dev/sda4","boot":false,"start":0,"end":null,"cyls":0,"blocks":0,"id":"0","system":"Empty"}]},{"disk":"/dev/mapper/centos-root","cylinders":2218,"heads":255,"sectors_per_track":63},{"disk":"/dev/mapper/centos-swap","cylinders":261,"heads":255,"sectors_per_track":63,"units":"cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0","partitions":[{"device":"/dev/sda1","boot":true,"start":0,"end":130,"cyls":131,"blocks":1048576,"id":"83","system":"Linux"},{"device":"/dev/sda2","boot":false,"start":130,"end":2610,"cyls":2481,"blocks":19921920,"id":"8e","system":"Linux LVM"},{"device":"/dev/sda3","boot":false,"start":0,"end":null,"cyls":0,"blocks":0,"id":"0","system":"Empty"},{"device":"/dev/sda4","boot":false,"start":0,"end":null,"cyls":0,"blocks":0,"id":"0","system":"Empty"}]}]

tests/fixtures/centos-7.7/sfdisk-l-multi.out

@@ -0,0 +1,21 @@
+
+Disk /dev/sda: 2610 cylinders, 255 heads, 63 sectors/track
+Units: cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0
+
+   Device Boot Start     End   #cyls    #blocks   Id  System
+/dev/sda1   *      0+    130-    131-   1048576   83  Linux
+/dev/sda2        130+   2610-   2481-  19921920   8e  Linux LVM
+/dev/sda3          0       -       0          0    0  Empty
+/dev/sda4          0       -       0          0    0  Empty
+
+Disk /dev/mapper/centos-root: 2218 cylinders, 255 heads, 63 sectors/track
+
+Disk /dev/mapper/centos-swap: 261 cylinders, 255 heads, 63 sectors/track
+Units: cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0
+
+   Device Boot Start     End   #cyls    #blocks   Id  System
+/dev/sda1   *      0+    130-    131-   1048576   83  Linux
+/dev/sda2        130+   2610-   2481-  19921920   8e  Linux LVM
+/dev/sda3          0       -       0          0    0  Empty
+/dev/sda4          0       -       0          0    0  Empty
+

tests/fixtures/centos-7.7/sfdisk-l.json

@@ -0,0 +1 @@
+[{"disk":"/dev/sda","cylinders":2610,"heads":255,"sectors_per_track":63,"units":"cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0","partitions":[{"device":"/dev/sda1","boot":true,"start":0,"end":130,"cyls":131,"blocks":1048576,"id":"83","system":"Linux"},{"device":"/dev/sda2","boot":false,"start":130,"end":2610,"cyls":2481,"blocks":19921920,"id":"8e","system":"Linux LVM"},{"device":"/dev/sda3","boot":false,"start":0,"end":null,"cyls":0,"blocks":0,"id":"0","system":"Empty"},{"device":"/dev/sda4","boot":false,"start":0,"end":null,"cyls":0,"blocks":0,"id":"0","system":"Empty"}]},{"disk":"/dev/mapper/centos-root","cylinders":2218,"heads":255,"sectors_per_track":63},{"disk":"/dev/mapper/centos-swap","cylinders":261,"heads":255,"sectors_per_track":63}]

tests/fixtures/centos-7.7/sfdisk-l.out

@@ -0,0 +1,14 @@
+
+Disk /dev/sda: 2610 cylinders, 255 heads, 63 sectors/track
+Units: cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0
+
+   Device Boot Start     End   #cyls    #blocks   Id  System
+/dev/sda1   *      0+    130-    131-   1048576   83  Linux
+/dev/sda2        130+   2610-   2481-  19921920   8e  Linux LVM
+/dev/sda3          0       -       0          0    0  Empty
+/dev/sda4          0       -       0          0    0  Empty
+
+Disk /dev/mapper/centos-root: 2218 cylinders, 255 heads, 63 sectors/track
+
+Disk /dev/mapper/centos-swap: 261 cylinders, 255 heads, 63 sectors/track
+

tests/fixtures/centos-7.7/sfdisk-luB.json

@@ -0,0 +1 @@
+[{"disk":"/dev/sda","cylinders":2610,"heads":255,"sectors_per_track":63,"units":"blocks of 1024 bytes, counting from 0","partitions":[{"device":"/dev/sda1","boot":true,"start":1024,"end":1049599,"blocks":1048576,"id":"83","system":"Linux"},{"device":"/dev/sda2","boot":false,"start":1049600,"end":20971519,"blocks":19921920,"id":"8e","system":"Linux LVM"},{"device":"/dev/sda3","boot":false,"start":0,"end":null,"blocks":0,"id":"0","system":"Empty"},{"device":"/dev/sda4","boot":false,"start":0,"end":null,"blocks":0,"id":"0","system":"Empty"}]},{"disk":"/dev/mapper/centos-root","cylinders":2218,"heads":255,"sectors_per_track":63},{"disk":"/dev/mapper/centos-swap","cylinders":261,"heads":255,"sectors_per_track":63}]

tests/fixtures/centos-7.7/sfdisk-luB.out

@@ -0,0 +1,14 @@
+
+Disk /dev/sda: 2610 cylinders, 255 heads, 63 sectors/track
+Units: blocks of 1024 bytes, counting from 0
+
+   Device Boot   Start       End    #blocks   Id  System
+/dev/sda1   *     1024   1049599    1048576   83  Linux
+/dev/sda2      1049600  20971519   19921920   8e  Linux LVM
+/dev/sda3            0         -          0    0  Empty
+/dev/sda4            0         -          0    0  Empty
+
+Disk /dev/mapper/centos-root: 2218 cylinders, 255 heads, 63 sectors/track
+
+Disk /dev/mapper/centos-swap: 261 cylinders, 255 heads, 63 sectors/track
+

tests/fixtures/centos-7.7/sfdisk-luM.json

@@ -0,0 +1 @@
+[{"disk":"/dev/sda","cylinders":2610,"heads":255,"sectors_per_track":63,"units":"1MiB = 1024*1024 bytes, blocks of 1024 bytes, counting from 0","partitions":[{"device":"/dev/sda1","boot":true,"start":1,"end":1024,"mib":1024,"blocks":1048576,"id":"83","system":"Linux"},{"device":"/dev/sda2","boot":false,"start":1025,"end":20479,"mib":19455,"blocks":19921920,"id":"8e","system":"Linux LVM"},{"device":"/dev/sda3","boot":false,"start":0,"end":null,"mib":0,"blocks":0,"id":"0","system":"Empty"},{"device":"/dev/sda4","boot":false,"start":0,"end":null,"mib":0,"blocks":0,"id":"0","system":"Empty"}]},{"disk":"/dev/mapper/centos-root","cylinders":2218,"heads":255,"sectors_per_track":63},{"disk":"/dev/mapper/centos-swap","cylinders":261,"heads":255,"sectors_per_track":63}]

tests/fixtures/centos-7.7/sfdisk-luM.out

@@ -0,0 +1,14 @@
+
+Disk /dev/sda: 2610 cylinders, 255 heads, 63 sectors/track
+Units: 1MiB = 1024*1024 bytes, blocks of 1024 bytes, counting from 0
+
+   Device Boot Start   End    MiB    #blocks   Id  System
+/dev/sda1   *     1   1024   1024    1048576   83  Linux
+/dev/sda2      1025  20479  19455   19921920   8e  Linux LVM
+/dev/sda3         0      -      0          0    0  Empty
+/dev/sda4         0      -      0          0    0  Empty
+
+Disk /dev/mapper/centos-root: 2218 cylinders, 255 heads, 63 sectors/track
+
+Disk /dev/mapper/centos-swap: 261 cylinders, 255 heads, 63 sectors/track
+

tests/fixtures/centos-7.7/sfdisk-luS.json

@@ -0,0 +1 @@
+[{"disk":"/dev/sda","cylinders":2610,"heads":255,"sectors_per_track":63,"units":"sectors of 512 bytes, counting from 0","partitions":[{"device":"/dev/sda1","boot":true,"start":2048,"end":2099199,"sectors":2097152,"id":"83","system":"Linux"},{"device":"/dev/sda2","boot":false,"start":2099200,"end":41943039,"sectors":39843840,"id":"8e","system":"Linux LVM"},{"device":"/dev/sda3","boot":false,"start":0,"end":null,"sectors":0,"id":"0","system":"Empty"},{"device":"/dev/sda4","boot":false,"start":0,"end":null,"sectors":0,"id":"0","system":"Empty"}]},{"disk":"/dev/mapper/centos-root","cylinders":2218,"heads":255,"sectors_per_track":63},{"disk":"/dev/mapper/centos-swap","cylinders":261,"heads":255,"sectors_per_track":63}]

tests/fixtures/centos-7.7/sfdisk-luS.out

@@ -0,0 +1,14 @@
+
+Disk /dev/sda: 2610 cylinders, 255 heads, 63 sectors/track
+Units: sectors of 512 bytes, counting from 0
+
+   Device Boot    Start       End   #sectors  Id  System
+/dev/sda1   *      2048   2099199    2097152  83  Linux
+/dev/sda2       2099200  41943039   39843840  8e  Linux LVM
+/dev/sda3             0         -          0   0  Empty
+/dev/sda4             0         -          0   0  Empty
+
+Disk /dev/mapper/centos-root: 2218 cylinders, 255 heads, 63 sectors/track
+
+Disk /dev/mapper/centos-swap: 261 cylinders, 255 heads, 63 sectors/track
+

tests/fixtures/centos-7.7/uname.out

@@ -0,0 +1 @@
+Linux

tests/fixtures/centos-7.7/vmstat-a-streaming.json

@@ -0,0 +1 @@
+[{"runnable_procs":2,"uninterruptible_sleeping_procs":0,"virtual_mem_used":0,"free_mem":2794696,"buffer_mem":null,"cache_mem":null,"inactive_mem":295968,"active_mem":312736,"swap_in":0,"swap_out":0,"blocks_in":1,"blocks_out":3,"interrupts":29,"context_switches":57,"user_time":0,"system_time":0,"idle_time":99,"io_wait_time":0,"stolen_time":0,"timestamp":null,"timezone":null}]

tests/fixtures/centos-7.7/vmstat-a.json

@@ -0,0 +1 @@
+[{"runnable_procs":2,"uninterruptible_sleeping_procs":0,"virtual_mem_used":0,"free_mem":2794696,"buffer_mem":null,"cache_mem":null,"inactive_mem":295968,"active_mem":312736,"swap_in":0,"swap_out":0,"blocks_in":1,"blocks_out":3,"interrupts":29,"context_switches":57,"user_time":0,"system_time":0,"idle_time":99,"io_wait_time":0,"stolen_time":0,"timestamp":null,"timezone":null}]