Merge pull request #170 from kellyjonbrazil/dev

jc v1.17.0 from dev

CHANGELOG

@@ -1,5 +1,75 @@
 jc changelog
 
+20210923 v1.17.0
+- Note to Package Maintainers: please see note at 20210720 v1.16.0
+- Add wrapping of warning and error messages
+- Add vmstat parser tested on linux
+- Add support for streaming parsers
+- Add ls command streaming parser tested on linux, macOS, and freeBSD
+- Add ping command streaming parser tested on linux, macOS, and freeBSD
+- Add vmstat command streaming parser tested on linux
+- Add -u option to allow unbuffered output
+
+20210830 v1.16.2
+- Note to Package Maintainers: please see note at 20210720 v1.16.0
+- Update sfdisk parser to support the -F option and newer versions of sfdisk
+
+20210813 v1.16.1
+- Note to Package Maintainers: please see note at 20210720 v1.16.0
+- Fix issue with process substitution with the magic syntax
+- Fix issue with globs not including filenames with spaces with magic syntax
+- Fix stat parser to properly handle filenames with spaces on macOS/BSD
+
+20210720 v1.16.0
+- Note to Package Maintainers:
+    TL;DR: `/man/jc.1.gz` and `/jc/man/jc.1.gz` are deprecated and only `/man/jc.1` should be used.
+    
+    The Man page in the PyPi source packages will be moving from `/jc/man/jc.1.gz` to `/man/jc.1`
+    in version 1.17.0. For now the Man pages will be available in both locations, but be aware that
+    the Man page at `/jc/man/jc.1.gz` is now considered deprecated.
+
+    Also, starting in v1.17.0, the Man page will no longer be compressed in the source package,
+    therefore `/man/jc.1.gz` should also be considered deprecated and will no longer be available
+    after v1.17.0. Please use `/man/jc.1` and compress downstream if you would like.
+- Include CHANGELOG in source distribution
+- Fix Man page location in source packages
+- Add sfdisk command parser tested on linux
+- Update unit test files to change the timezone when needed (POSIX only)
+- Binaries and DEB/RPM/MSI packages now include Python 3.9.5 interpreter
+
+20210628 v1.15.6
+- Fix issue to only load local plugin parsers that have filenames that end in .py 
+
+20210520 v1.15.5
+- Fix issue where help and about information would not display if a 3rd party parser library was missing. (e.g. xmltodict)
+- Add more error message detail when encountering ParseError and LibraryNotFound exceptions
+
+20210517 v1.15.4
+- Update ping parser to support error responses in OSX and BSD
+- Update ping parser to be more resilient against parsing errors for unknown error types
+- Update dig parser to support `+noall +answer` use case
+- Update dig parser compatibility to all platforms
+- Fix colors in Windows terminals (cmd.exe and PowerShell)
+- Fix epoch calculations when UTC is referenced as "Coordinated Universal Time"
+- Add Windows time format for systeminfo output
+- Add exceptions module to standardize parser exceptions
+- JC no longer swallows exit codes when using the "magic" syntax. See the Exit Codes section of the README and man page for details
+
+20210426 v1.15.3
+- Add ufw status command parser tested on linux
+- Add ufw-appinfo command parser tested on linux
+- Fix deb package name to conform to standard
+- Add Caveats section to readme and manpage
+
+20210418 v1.15.2
+- Add systeminfo parser tested on Windows
+- Update dig parser to fix an issue with IPv6 addresses in the server field
+- Update dig parser to fix an issue when axfr entries contain a semicolon
+- Update dig parser to add support for Additional Section and Opt Pseudosection
+- Update dig parser to add query_size field
+- Use dig parser as the main example in readme, documentation, and man page
+- Standardize int, float, and boolean conversion rules with functions in jc.utils
+
 20210413 v1.15.1
 - New feature to show parser documentation interactively with -h --parser_name
   for example: $ jc -h --arp

CONTRIBUTING.md

@@ -15,11 +15,12 @@ Pull requests are the best way to propose changes to the codebase (we use [Githu
 
 1. Open an issue to discuss the new feature, bug fix, or parser before opening a pull request. For new parsers, it is important to agree upon a schema before developing the parser.
 2. Fork the repo and create your branch from `dev`, if available, otherwise `master`.
-3. If you've added code that should be tested, add tests. All new parsers should have several sample outputs and tests.
-4. Documentation is auto-generated from docstrings, so ensure they are clear and accurate.
-5. Ensure the test suite passes.
-6. Make sure your code lints.
-7. Issue that pull request!
+3. For new parsers: Use the [`jc/parsers/foo.py`](https://github.com/kellyjonbrazil/jc/blob/master/jc/parsers/foo.py) or [`jc/parsers/foo_s.py (streaming)`](https://github.com/kellyjonbrazil/jc/blob/master/jc/parsers/foo_s.py) parsers as a template to get started. You can even place a new parser python module file in the [parser plugin directory](https://github.com/kellyjonbrazil/jc#custom-parsers) to get started right away with just a standard `jc` installation.
+4. If you've added code that should be tested, add tests. All new parsers should have several sample outputs and tests.
+5. Documentation is auto-generated from docstrings, so ensure they are clear and accurate.
+6. Ensure the test suite passes. (Note: "**America/Los_Angeles**" timezone should be configured on the test system)
+7. Make sure your code lints.
+8. Issue that pull request!
 
 ## Parser Schema Guidelines
 - Try to keep the schema as flat as possible - typically a list of flat dictionaries
@@ -31,36 +32,41 @@ This will make it easier to use tools like `jq` without requiring escaping of sp
 **Examples**
 
 Bad:
-```
+```json
 {
   "Interface 1": [
-    192.168.1.1,
-    172.16.1.1
+    "192.168.1.1",
+    "172.16.1.1"
   ],
   "Wifi Interface 1": [
-    10.1.1.1
+    "10.1.1.1"
   ]
 }
 ```
 Good:
-```
+```json
 [
   {
     "interface": "Interface 1",
     "ip_addresses": [
-      192.168.1.1,
-      172.16.1.1
+      "192.168.1.1",
+      "172.16.1.1"
     ]
   },
   {
     "interface": "Wifi Interface 1",
     "ip_addresses": [
-      10.1.1.1
+      "10.1.1.1"
     ]
   }
 ]
 ```
 
+## Tests
+It is essential to have good command output sample coverage and tests to keep the `jc` parser quality high.
+
+Many parsers include calculated timestamp fields using the `jc.utils.timestamp` class. Naive timestamps created with this class should be generated on a system configured with the "**America/Los_Angeles**" timezone on linux/macOS/unix and "**Pacific Standard Time**" timezone on Windows for tests to pass on the Github Actions CI tests. This timezone should be configured on your local system before running the tests locally, as well.
+
 ## Any contributions you make will be under the MIT Software License
 In short, when you submit code changes, your submissions are understood to be under the same [MIT License](http://choosealicense.com/licenses/mit/) that covers the project. Feel free to contact the maintainers if that's a concern.
 

EXAMPLES.md

@@ -548,7 +548,7 @@ dig cnn.com www.cnn.com @205.251.194.64 | jc --dig -p          # or:  jc -p dig
 ```json
 [
   {
-    "id": 52172,
+    "id": 10267,
     "opcode": "QUERY",
     "status": "NOERROR",
     "flags": [
@@ -560,6 +560,13 @@ dig cnn.com www.cnn.com @205.251.194.64 | jc --dig -p          # or:  jc -p dig
     "answer_num": 4,
     "authority_num": 0,
     "additional_num": 1,
+    "opt_pseudosection": {
+      "edns": {
+        "version": 0,
+        "flags": [],
+        "udp": 4096
+      }
+    },
     "question": {
       "name": "cnn.com.",
       "class": "IN",
@@ -570,40 +577,40 @@ dig cnn.com www.cnn.com @205.251.194.64 | jc --dig -p          # or:  jc -p dig
         "name": "cnn.com.",
         "class": "IN",
         "type": "A",
-        "ttl": 27,
+        "ttl": 17,
         "data": "151.101.65.67"
       },
       {
         "name": "cnn.com.",
         "class": "IN",
         "type": "A",
-        "ttl": 27,
+        "ttl": 17,
         "data": "151.101.129.67"
       },
       {
         "name": "cnn.com.",
         "class": "IN",
         "type": "A",
-        "ttl": 27,
+        "ttl": 17,
         "data": "151.101.1.67"
       },
       {
         "name": "cnn.com.",
         "class": "IN",
         "type": "A",
-        "ttl": 27,
+        "ttl": 17,
         "data": "151.101.193.67"
       }
     ],
-    "query_time": 38,
-    "server": "2600",
-    "when": "Tue Mar 30 20:07:59 PDT 2021",
+    "query_time": 51,
+    "server": "2600:1700:bab0:d40::1#53(2600:1700:bab0:d40::1)",
+    "when": "Fri Apr 16 16:24:32 PDT 2021",
     "rcvd": 100,
-    "when_epoch": 1617160079,
+    "when_epoch": 1618615472,
     "when_epoch_utc": null
   },
   {
-    "id": 36292,
+    "id": 56207,
     "opcode": "QUERY",
     "status": "NOERROR",
     "flags": [
@@ -615,6 +622,13 @@ dig cnn.com www.cnn.com @205.251.194.64 | jc --dig -p          # or:  jc -p dig
     "answer_num": 1,
     "authority_num": 4,
     "additional_num": 1,
+    "opt_pseudosection": {
+      "edns": {
+        "version": 0,
+        "flags": [],
+        "udp": 4096
+      }
+    },
     "question": {
       "name": "www.cnn.com.",
       "class": "IN",
@@ -659,11 +673,11 @@ dig cnn.com www.cnn.com @205.251.194.64 | jc --dig -p          # or:  jc -p dig
         "data": "ns-576.awsdns-08.net."
       }
     ],
-    "query_time": 27,
+    "query_time": 22,
     "server": "205.251.194.64#53(205.251.194.64)",
-    "when": "Tue Mar 30 20:07:59 PDT 2021",
+    "when": "Fri Apr 16 16:24:32 PDT 2021",
     "rcvd": 212,
-    "when_epoch": 1617160079,
+    "when_epoch": 1618615472,
     "when_epoch_utc": null
   }
 ]
@@ -674,7 +688,7 @@ dig -x 1.1.1.1 | jc --dig -p          # or:  jc -p dig -x 1.1.1.1
 ```json
 [
   {
-    "id": 22191,
+    "id": 20785,
     "opcode": "QUERY",
     "status": "NOERROR",
     "flags": [
@@ -686,6 +700,13 @@ dig -x 1.1.1.1 | jc --dig -p          # or:  jc -p dig -x 1.1.1.1
     "answer_num": 1,
     "authority_num": 0,
     "additional_num": 1,
+    "opt_pseudosection": {
+      "edns": {
+        "version": 0,
+        "flags": [],
+        "udp": 4096
+      }
+    },
     "question": {
       "name": "1.1.1.1.in-addr.arpa.",
       "class": "IN",
@@ -700,18 +721,18 @@ dig -x 1.1.1.1 | jc --dig -p          # or:  jc -p dig -x 1.1.1.1
         "data": "one.one.one.one."
       }
     ],
-    "query_time": 44,
-    "server": "2600",
-    "when": "Tue Mar 30 20:10:34 PDT 2021",
+    "query_time": 40,
+    "server": "2600:1700:bab0:d40::1#53(2600:1700:bab0:d40::1)",
+    "when": "Sat Apr 17 14:50:50 PDT 2021",
     "rcvd": 78,
-    "when_epoch": 1617160234,
+    "when_epoch": 1618696250,
     "when_epoch_utc": null
   }
 ]
 ```
 ### dir
 ```bash
-dir | jc --dir -p          # or:  jc -p dir
+dir | jc --dir -p
 ```
 ```json
 [
@@ -1650,7 +1671,7 @@ iw dev wlan0 scan | jc --iw-scan -p          # or:  jc -p iw dev wlan0 scan
 ```
 ### jobs
 ```bash
-jobs -l | jc --jobs -p          # or:  jc -p jobs
+jobs -l | jc --jobs -p
 ```
 ```json
 [
@@ -2535,6 +2556,75 @@ rpm_qia | jc --rpm_qi -p          # or:  jc -p rpm -qia
   }
 ]
 ```
+### sfdisk
+```bash
+sfdisk -l | jc --sfdisk -p          # or jc -p sfdisk -l
+```
+```json
+[
+  {
+    "disk": "/dev/sda",
+    "cylinders": 2610,
+    "heads": 255,
+    "sectors_per_track": 63,
+    "units": "cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0",
+    "partitions": [
+      {
+        "device": "/dev/sda1",
+        "boot": true,
+        "start": 0,
+        "end": 130,
+        "cyls": 131,
+        "blocks": 1048576,
+        "id": "83",
+        "system": "Linux"
+      },
+      {
+        "device": "/dev/sda2",
+        "boot": false,
+        "start": 130,
+        "end": 2610,
+        "cyls": 2481,
+        "blocks": 19921920,
+        "id": "8e",
+        "system": "Linux LVM"
+      },
+      {
+        "device": "/dev/sda3",
+        "boot": false,
+        "start": 0,
+        "end": null,
+        "cyls": 0,
+        "blocks": 0,
+        "id": "0",
+        "system": "Empty"
+      },
+      {
+        "device": "/dev/sda4",
+        "boot": false,
+        "start": 0,
+        "end": null,
+        "cyls": 0,
+        "blocks": 0,
+        "id": "0",
+        "system": "Empty"
+      }
+    ]
+  },
+  {
+    "disk": "/dev/mapper/centos-root",
+    "cylinders": 2218,
+    "heads": 255,
+    "sectors_per_track": 63
+  },
+  {
+    "disk": "/dev/mapper/centos-swap",
+    "cylinders": 261,
+    "heads": 255,
+    "sectors_per_track": 63
+  }
+]
+```
 ### /etc/shadow file
 ```bash
 cat /etc/shadow | jc --shadow -p
@@ -2869,6 +2959,72 @@ systemctl list-unit-files | jc --systemctl-luf -p          # or:  jc -p systemct
   }
 ]
 ```
+### systeminfo
+```bash
+systeminfo | jc --systeminfo -p          # or:  jc -p systeminfo
+```
+```json
+{
+  "host_name": "TESTLAPTOP",
+  "os_name": "Microsoft Windows 10 Enterprise",
+  "os_version": "10.0.17134 N/A Build 17134",
+  "os_manufacturer": "Microsoft Corporation",
+  "os_configuration": "Member Workstation",
+  "os_build_type": "Multiprocessor Free",
+  "registered_owner": "Test, Inc.",
+  "registered_organization": "Test, Inc.",
+  "product_id": "11111-11111-11111-AA111",
+  "original_install_date": "3/26/2019, 3:51:30 PM",
+  "system_boot_time": "3/30/2021, 6:13:59 AM",
+  "system_manufacturer": "Dell Inc.",
+  "system_model": "Precision 5530",
+  "system_type": "x64-based PC",
+  "processors": [
+    "Intel64 Family 6 Model 158 Stepping 10 GenuineIntel ~2592 Mhz"
+  ],
+  "bios_version": "Dell Inc. 1.16.2, 4/21/2020",
+  "windows_directory": "C:\\WINDOWS",
+  "system_directory": "C:\\WINDOWS\\system32",
+  "boot_device": "\\Device\\HarddiskVolume2",
+  "system_locale": "en-us;English (United States)",
+  "input_locale": "en-us;English (United States)",
+  "time_zone": "(UTC+00:00) UTC",
+  "total_physical_memory_mb": 32503,
+  "available_physical_memory_mb": 19743,
+  "virtual_memory_max_size_mb": 37367,
+  "virtual_memory_available_mb": 22266,
+  "virtual_memory_in_use_mb": 15101,
+  "page_file_locations": "C:\\pagefile.sys",
+  "domain": "test.com",
+  "logon_server": "\\\\TESTDC01",
+  "hotfixs": [
+    "KB2693643",
+    "KB4601054"
+  ],
+  "network_cards": [
+    {
+      "name": "Intel(R) Wireless-AC 9260 160MHz",
+      "connection_name": "Wi-Fi",
+      "status": null,
+      "dhcp_enabled": true,
+      "dhcp_server": "192.168.2.1",
+      "ip_addresses": [
+        "192.168.2.219"
+      ]
+    }
+  ],
+  "hyperv_requirements": {
+    "vm_monitor_mode_extensions": true,
+    "virtualization_enabled_in_firmware": true,
+    "second_level_address_translation": false,
+    "data_execution_prevention_available": true
+  },
+  "original_install_date_epoch": 1553640690,
+  "original_install_date_epoch_utc": 1553615490,
+  "system_boot_time_epoch": 1617110039,
+  "system_boot_time_epoch_utc": 1617084839
+}
+```
 ### /usr/bin/time
 ```bash
 /usr/bin/time --verbose -o timefile.out sleep 2.5; cat timefile.out | jc --time -p
@@ -3039,6 +3195,115 @@ traceroute -m 3 8.8.8.8 | jc --traceroute -p          # or:  jc -p traceroute -m
   ]
 }
 ```
+### ufw status
+```bash
+ufw status verbose  | jc --ufw -p          # or jc -p ufw status verbose
+```
+```json
+{
+  "status": "active",
+  "logging": "on",
+  "logging_level": "low",
+  "default": "deny (incoming), allow (outgoing), disabled (routed)",
+  "new_profiles": "skip",
+  "rules": [
+    {
+      "action": "ALLOW",
+      "action_direction": "IN",
+      "index": null,
+      "network_protocol": "ipv4",
+      "to_interface": "any",
+      "to_transport": "any",
+      "to_service": null,
+      "to_ports": [
+        22
+      ],
+      "to_ip": "0.0.0.0",
+      "to_ip_prefix": 0,
+      "comment": null,
+      "from_ip": "0.0.0.0",
+      "from_ip_prefix": 0,
+      "from_interface": "any",
+      "from_transport": "any",
+      "from_port_ranges": [
+        {
+          "start": 0,
+          "end": 65535
+        }
+      ],
+      "from_service": null
+    },
+    {
+      "action": "ALLOW",
+      "action_direction": "IN",
+      "index": null,
+      "network_protocol": "ipv4",
+      "to_interface": "any",
+      "to_transport": "tcp",
+      "to_service": null,
+      "to_ports": [
+        80,
+        443
+      ],
+      "to_ip": "0.0.0.0",
+      "to_ip_prefix": 0,
+      "comment": null,
+      "from_ip": "0.0.0.0",
+      "from_ip_prefix": 0,
+      "from_interface": "any",
+      "from_transport": "any",
+      "from_port_ranges": [
+        {
+          "start": 0,
+          "end": 65535
+        }
+      ],
+      "from_service": null
+    }
+  ]
+}
+```
+### ufw app info [application]
+```bash
+ufw app info MSN | jc --ufw-appinfo -p          # or:  jc -p ufw app info MSN
+```
+```json
+[
+  {
+    "profile": "MSN",
+    "title": "MSN Chat",
+    "description": "MSN chat protocol (with file transfer and voice)",
+    "tcp_list": [
+      1863,
+      6901
+    ],
+    "udp_list": [
+      1863,
+      6901
+    ],
+    "tcp_ranges": [
+      {
+        "start": 6891,
+        "end": 6900
+      }
+    ],
+    "normalized_tcp_list": [
+      1863,
+      6901
+    ],
+    "normalized_tcp_ranges": [
+      {
+        "start": 6891,
+        "end": 6900
+      }
+    ],
+    "normalized_udp_list": [
+      1863,
+      6901
+    ]
+  }
+]
+```
 ### uname -a
 ```bash
 uname -a | jc --uname -p          # or:  jc -p uname -a

MANIFEST.in

@@ -1 +1,3 @@
 include jc/man/jc.1.gz
+include man/jc.1
+include CHANGELOG

README.md

@@ -1,7 +1,7 @@
 ![Tests](https://github.com/kellyjonbrazil/jc/workflows/Tests/badge.svg?branch=master)
 ![Pypi](https://img.shields.io/pypi/v/jc.svg)
 
-> Try the new `jc` [web demo](https://jc-web-demo.herokuapp.com/)!
+> Try the `jc` [web demo](https://jc-web-demo.herokuapp.com/)
 
 > JC is [now available](https://galaxy.ansible.com/community/general) as an Ansible filter plugin in the `community.general` collection! See this [blog post](https://blog.kellybrazil.com/2020/08/30/parsing-command-output-in-ansible-with-jc/) for an example.
 
@@ -9,85 +9,68 @@
 JSON CLI output utility
 
 `jc` JSONifies the output of many CLI tools and file-types for easier parsing in scripts. See the [**Parsers**](#parsers) section for supported commands and file-types.
-
-This allows further command-line processing of output with tools like `jq` by piping commands:
 ```bash
-ls -l /usr/bin | jc --ls | jq '.[] | select(.size > 50000000)'
+dig example.com | jc --dig
 ```
 ```json
-{
-  "filename": "docker",
-  "flags": "-rwxr-xr-x",
-  "links": 1,
-  "owner": "root",
-  "group": "root",
-  "size": 68677120,
-  "date": "Aug 14 19:41"
-}
+[{"id":38052,"opcode":"QUERY","status":"NOERROR","flags":["qr","rd","ra"],"query_num":1,"answer_num":1,
+"authority_num":0,"additional_num":1,"opt_pseudosection":{"edns":{"version":0,"flags":[],"udp":4096}},"question":
+{"name":"example.com.","class":"IN","type":"A"},"answer":[{"name":"example.com.","class":"IN","type":"A","ttl":
+39049,"data":"93.184.216.34"}],"query_time":49,"server":"2600:1700:bab0:d40::1#53(2600:1700:bab0:d40::1)","when":
+"Fri Apr 16 16:09:00 PDT 2021","rcvd":56,"when_epoch":1618614540,"when_epoch_utc":null}]
+```
+This allows further command-line processing of output with tools like `jq` or [`jello`](https://github.com/kellyjonbrazil/jello) by piping commands:
+```bash
+$ dig example.com | jc --dig | jq -r '.[].answer[].data'
+93.184.216.34
 ```
 or using the alternative "magic" syntax:
 ```bash
-jc ls -l /usr/bin | jq '.[] | select(.size > 50000000)'
-```
-```json
-{
-  "filename": "docker",
-  "flags": "-rwxr-xr-x",
-  "links": 1,
-  "owner": "root",
-  "group": "root",
-  "size": 68677120,
-  "date": "Aug 14 19:41"
-}
+$ jc dig example.com | jq -r '.[].answer[].data'
+93.184.216.34
 ```
 The `jc` parsers can also be used as python modules. In this case the output will be a python dictionary, or list of dictionaries, instead of JSON:
 ```python
->>> import jc.parsers.ls
+>>> import subprocess
+>>> import jc.parsers.dig
 >>> 
->>> data='''-rwxr-xr-x  1 root  wheel    23648 May  3 22:26 cat
-... -rwxr-xr-x  1 root  wheel    30016 May  3 22:26 chmod
-... -rwxr-xr-x  1 root  wheel    29024 May  3 22:26 cp
-... -rwxr-xr-x  1 root  wheel   375824 May  3 22:26 csh
-... -rwxr-xr-x  1 root  wheel    28608 May  3 22:26 date
-... -rwxr-xr-x  1 root  wheel    32000 May  3 22:26 dd
-... -rwxr-xr-x  1 root  wheel    23392 May  3 22:26 df
-... -rwxr-xr-x  1 root  wheel    18128 May  3 22:26 echo'''
+>>> cmd_output = subprocess.check_output(['dig', 'example.com'], text=True)
+>>> data = jc.parsers.dig.parse(cmd_output)
 >>>
->>> jc.parsers.ls.parse(data)
-[{'filename': 'cat', 'flags': '-rwxr-xr-x', 'links': 1, 'owner': 'root', 'group': 'wheel', 'size': 23648,
-'date': 'May 3 22:26'}, {'filename': 'chmod', 'flags': '-rwxr-xr-x', 'links': 1, 'owner': 'root',
-'group': 'wheel', 'size': 30016, 'date': 'May 3 22:26'}, {'filename': 'cp', 'flags': '-rwxr-xr-x',
-'links': 1, 'owner': 'root', 'group': 'wheel', 'size': 29024, 'date': 'May 3 22:26'}, {'filename': 'csh',
-'flags': '-rwxr-xr-x', 'links': 1, 'owner': 'root', 'group': 'wheel', 'size': 375824, 'date': 'May 3
-22:26'}, {'filename': 'date', 'flags': '-rwxr-xr-x', 'links': 1, 'owner': 'root', 'group': 'wheel',
-'size': 28608, 'date': 'May 3 22:26'}, {'filename': 'dd', 'flags': '-rwxr-xr-x', 'links': 1, 'owner':
-'root', 'group': 'wheel', 'size': 32000, 'date': 'May 3 22:26'}, {'filename': 'df', 'flags':
-'-rwxr-xr-x', 'links': 1, 'owner': 'root', 'group': 'wheel', 'size': 23392, 'date': 'May 3 22:26'},
-{'filename': 'echo', 'flags': '-rwxr-xr-x', 'links': 1, 'owner': 'root', 'group': 'wheel', 'size': 18128,
-'date': 'May 3 22:26'}]
+>>> data
+[{'id': 64612, 'opcode': 'QUERY', 'status': 'NOERROR', 'flags': ['qr', 'rd', 'ra'], 'query_num': 1, 'answer_num':
+1, 'authority_num': 0, 'additional_num': 1, 'opt_pseudosection': {'edns': {'version': 0, 'flags': [], 'udp':
+4096}}, 'question': {'name': 'example.com.', 'class': 'IN', 'type': 'A'}, 'answer': [{'name': 'example.com.',
+'class': 'IN', 'type': 'A', 'ttl': 29658, 'data': '93.184.216.34'}], 'query_time': 52, 'server':
+'2600:1700:bab0:d40::1#53(2600:1700:bab0:d40::1)', 'when': 'Fri Apr 16 16:13:00 PDT 2021', 'rcvd': 56,
+'when_epoch': 1618614780, 'when_epoch_utc': None}]
 ```
-Two representations of the data are possible. The default representation uses a strict schema per parser and converts known numbers to int/float JSON values. Certain known values of `None` are converted to JSON `null`, known boolean values are converted, and, in some cases, additional semantic context fields are added.
-
-> Note: Some parsers have calculated epoch timestamp fields added to the output. Unless a timestamp field name has a `_utc` suffix it is considered naive. (i.e. based on the local timezone of the system the `jc` parser was run on). 
->
-> If a UTC timezone can be detected in the text of the command output, the timestamp will be timezone aware and have a `_utc` suffix on the key name. (e.g. `epoch_utc`) No other timezones are supported for aware timestamps.
+Two representations of the data are available. The default representation uses a strict schema per parser and converts known numbers to int/float JSON values. Certain known values of `None` are converted to JSON `null`, known boolean values are converted, and, in some cases, additional semantic context fields are added.
 
 To access the raw, pre-processed JSON, use the `-r` cli option or the `raw=True` function parameter in `parse()`.
 
-Schemas for each parser can be found at the documentation link beside each parser below.
+Schemas for each parser can be found at the documentation link beside each [**Parser**](#parsers) below.
 
 Release notes can be found [here](https://blog.kellybrazil.com/category/jc-news/).
 
 ## Why Would Anyone Do This!?
-For more information on the motivations for this project, please see my [blog post](https://blog.kellybrazil.com/2019/11/26/bringing-the-unix-philosophy-to-the-21st-century/).
+For more information on the motivations for this project, please see my blog post on [Bringing the Unix Philosophy to the 21st Century](https://blog.kellybrazil.com/2019/11/26/bringing-the-unix-philosophy-to-the-21st-century/).
 
 See also:
 - [libxo on FreeBSD](http://juniper.github.io/libxo/libxo-manual.html)
 - [powershell](https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/convertto-json?view=powershell-7)
 - [blog: linux apps should have a json flag](https://thomashunter.name/posts/2012-06-06-linux-cli-apps-should-have-a-json-flag)
+- [Hacker News discussion](https://news.ycombinator.com/item?id=28266193)
+- [Reddit discussion](https://www.reddit.com/r/programming/comments/pa4cbb/bringing_the_unix_philosophy_to_the_21st_century/)
+
+Use Cases:
+- [Bash scripting](https://blog.kellybrazil.com/2021/04/12/practical-json-at-the-command-line/)
+- [Ansible command output parsing](https://blog.kellybrazil.com/2020/08/30/parsing-command-output-in-ansible-with-jc/)
+- [Saltstack command output parsing](https://blog.kellybrazil.com/2020/09/15/parsing-command-output-in-saltstack-with-jc/)
+- [Nornir command output parsing](https://blog.kellybrazil.com/2020/12/09/parsing-command-output-in-nornir-with-jc/)
 
 ## Installation
-There are several ways to get `jc`. You can install via `pip`; other OS package repositories like `apt-get`, `dnf`, `zypper`, `pacman`, `nix-env`, `guix`, `brew`, or `portsnap`; via DEB/RPM packages; or by downloading the correct binary for your architecture and running it anywhere on your filesystem.
+There are several ways to get `jc`. You can install via `pip`, OS package repositories, via DEB/RPM/MSI packaged binaries for linux and Windows, or by downloading the correct binary for your architecture and running it anywhere on your filesystem.
 
 ### Pip (macOS, linux, unix, Windows)
 ```bash
@@ -104,11 +87,11 @@ pip3 install jc
 | Arch linux            | `pacman -S jc`                                                                |
 | NixOS linux           | `nix-env -iA nixpkgs.jc` or `nix-env -iA nixos.jc`                            |
 | Guix System linux     | `guix install jc`                                                             |
-| MacOS                 | `brew install jc`                                                             |
+| macOS                 | `brew install jc`                                                             |
 | FreeBSD               | `portsnap fetch update && cd /usr/ports/textproc/py-jc && make install clean` |
 | Ansible filter plugin | `ansible-galaxy collection install community.general`                         |
 
-> For more packages and binaries, see https://kellyjonbrazil.github.io/jc-packaging/.
+> For more packages and binaries, see the [jc packaging](https://kellyjonbrazil.github.io/jc-packaging/) site.
 
 ## Usage
 `jc` accepts piped input from `STDIN` and outputs a JSON representation of the previous command's output to `STDOUT`.
@@ -121,8 +104,6 @@ jc [OPTIONS] COMMAND
 ```
 The JSON output can be compact (default) or pretty formatted with the `-p` option.
 
-> Note: For best results set the `LANG` locale environment variable to `C`. For example, either by setting directly on the command-line: `$ LANG=C date | jc --date`, or by exporting to the environment before running commands: `$ export LANG=C`.
-
 ### Parsers
 
 - `--acpi` enables the `acpi` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/acpi))
@@ -162,6 +143,7 @@ The JSON output can be compact (default) or pretty formatted with the `-p` optio
 - `--kv` enables the Key/Value file parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/kv))
 - `--last` enables the `last` and `lastb` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/last))
 - `--ls` enables the `ls` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ls))
+- `--ls-s` enables the `ls` command streaming parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ls_s))
 - `--lsblk` enables the `lsblk` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/lsblk))
 - `--lsmod` enables the `lsmod` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/lsmod))
 - `--lsof` enables the `lsof` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/lsof))
@@ -170,11 +152,13 @@ The JSON output can be compact (default) or pretty formatted with the `-p` optio
 - `--ntpq` enables the `ntpq -p` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ntpq))
 - `--passwd` enables the `/etc/passwd` file parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/passwd))
 - `--ping` enables the `ping` and `ping6` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ping))
+- `--ping-s` enables the `ping` and `ping6` command streaming parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ping_s))
 - `--pip-list` enables the `pip list` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/pip_list))
 - `--pip-show` enables the `pip show` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/pip_show))
 - `--ps` enables the `ps` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ps))
 - `--route` enables the `route` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/route))
 - `--rpm-qi` enables the `rpm -qi` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/rpm_qi))
+- `--sfdisk` enables the `sfdisk` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/sfdisk))
 - `--shadow` enables the `/etc/shadow` file parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/shadow))
 - `--ss` enables the `ss` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ss))
 - `--stat` enables the `stat` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/stat))
@@ -183,13 +167,18 @@ The JSON output can be compact (default) or pretty formatted with the `-p` optio
 - `--systemctl-lj` enables the `systemctl list-jobs` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/systemctl_lj))
 - `--systemctl-ls` enables the `systemctl list-sockets` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/systemctl_ls))
 - `--systemctl-luf` enables the `systemctl list-unit-files` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/systemctl_luf))
+- `--systeminfo` enables the `systeminfo` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/systeminfo))
 - `--time` enables the `/usr/bin/time` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/time))
 - `--timedatectl` enables the `timedatectl status` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/timedatectl))
 - `--tracepath` enables the `tracepath` and `tracepath6` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/tracepath))
 - `--traceroute` enables the `traceroute` and `traceroute6` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/traceroute))
+- `--ufw` enables the `ufw status` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ufw))
+- `--ufw-appinfo` enables the `ufw app info [application]` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ufw_appinfo))
 - `--uname` enables the `uname -a` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/uname))
 - `--upower` enables the `upower` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/upower))
 - `--uptime` enables the `uptime` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/uptime))
+- `--vmstat` enables the `vmstat` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/vmstat))
+- `--vmstat-s` enables the `vmstat` command streaming parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/vmstat_s))
 - `--w` enables the `w` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/w))
 - `--wc` enables the `wc` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/wc))
 - `--who` enables the `who` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/who))
@@ -199,13 +188,27 @@ The JSON output can be compact (default) or pretty formatted with the `-p` optio
 ### Options
 - `-a` about `jc`. Prints information about `jc` and the parsers (in JSON, of course!)
 - `-d` debug mode. Prints trace messages if parsing issues are encountered (use `-dd` for verbose debugging)
-- `-h` `jc` help. Use `jc -h --parser_name` for parser documentation
+- `-h` help. Use `jc -h --parser_name` for parser documentation
 - `-m` monochrome JSON output
 - `-p` pretty format the JSON output
-- `-q` quiet mode. Suppresses parser warning messages
+- `-q` quiet mode. Suppresses parser warning messages (use `-qq` to ignore streaming parser errors)
 - `-r` raw output. Provides a more literal JSON output, typically with string values and no additional semantic processing
+- `-u` unbuffer output
 - `-v` version information
 
+### Exit Codes
+Any fatal errors within `jc` will generate an exit code of `100`, otherwise the exit code will be `0`. When using the "magic" syntax (e.g. `jc ifconfig eth0`), `jc` will store the exit code of the program being parsed and add it to the `jc` exit code. This way it is easier to determine if an error was from the parsed program or `jc`.
+
+Consider the following examples using `ifconfig`:
+
+| `ifconfig` exit code | `jc` exit code | Combined exit code | Interpretation                     | 
+|----------------------|----------------|--------------------|------------------------------------|
+| `0`                  | `0`            | `0`                | No errors                          |
+| `1`                  | `0`            | `1`                | Error in  `ifconfig`               |
+| `0`                  | `100`          | `100`              | Error in  `jc`                     |
+| `1`                  | `100`          | `101`              | Error in  both `ifconfig` and `jc` |
+
+
 ### Setting Custom Colors via Environment Variable
 You can specify custom colors via the `JC_COLORS` environment variable. The `JC_COLORS` environment variable takes four comma separated string values in the following format:
 ```bash
@@ -222,6 +225,64 @@ or
 JC_COLORS=default,default,default,default
 ```
 
+### Streaming Parsers
+Most parsers load all of the data from STDIN, parse it, then output the entire JSON document serially. There are some streaming parsers (e.g. `ls-s` and `ping-s`) that immediately start processing and outputing the data line-by-line as [JSON Lines](https://jsonlines.org/) (aka [NDJSON](http://ndjson.org/)) while it is being received from STDIN. This can significantly reduce the amount of memory required to parse large amounts of command output (e.g. `ls -lR /`) and can sometimes process the data more quickly. Streaming parsers have slightly different behavior than standard parsers as outlined below.
+
+> Note: Streaming parsers cannot be used with the "magic" syntax
+
+#### Ignoring Errors
+
+You may want to ignore parsing errors when using streaming parsers since these may be used in long-lived processing pipelines and errors can break the pipe. To ignore parsing errors, use the `-qq` cli option or the `ignore_exceptions=True` argument with the `parse()` function. This will add a `_jc_meta` object to the JSON output with a `success` attribute. If `success` is `true`, then there were no issues parsing the line. If `success` is `false`, then a parsing issue was found and `error` and `line` fields will be added to include a short error description and the contents of the unparsable line, respectively:
+
+Successfully parsed line with `-qq` option:
+```json
+{
+  "command_data": "data",
+  "_jc_meta": {
+    "success": true
+  }
+}
+```
+Unsuccessfully parsed line with `-qq` option:
+```json
+{
+  "_jc_meta": {
+    "success": false,
+    "error": "error message",
+    "line": "original line data"
+  }
+}
+```
+
+#### Unbuffering Output
+
+Most operating systems will buffer output that is being piped from process to process. The buffer is usually around 4KB. When viewing the output in the terminal the OS buffer is not engaged so output is immediately displayed on the screen. When piping multiple processes together, though, it may seem as if the output is hanging when the input data is very slow (e.g. `ping`):
+```
+$ ping 1.1.1.1 | jc --ping-s | jq
+<slow output>
+```
+This is because the OS engages the 4KB buffer between `jc` and `jq` in this example. To display the data on the terminal in realtime, you can disable the buffer with the `-u` (unbuffer) cli option:
+```
+$ ping 1.1.1.1 | jc --ping-s -u | jq
+{"type":"reply","pattern":null,"timestamp":null,"bytes":"64","response_ip":"1.1.1.1","icmp_seq":"1","ttl":"128","time_ms":"24.6","duplicate":false}
+{"type":"reply","pattern":null,"timestamp":null,"bytes":"64","response_ip":"1.1.1.1","icmp_seq":"2","ttl":"128","time_ms":"26.8","duplicate":false}
+...
+```
+> Note: Unbuffered output can be slower for large data streams.
+
+#### Using Streaming Parsers as Python Modules
+
+Streaming parsers accept any iterable object and return a generator iterator object allowing lazy processing of the data. The input data should iterate on lines of string data. Examples of good input data are `sys.stdin` or `str.splitlines()`.
+
+To use the generator object in your code, simply loop through it or use the [next()](https://docs.python.org/3/library/functions.html#next) builtin function:
+```python
+import jc.parsers.ls_s
+
+result = jc.parsers.ls_s.parse(ls_command_output.splitlines())
+for item in result:
+    print(item["filename"])
+```
+
 ### Custom Parsers
 Custom local parser plugins may be placed in a `jc/jcparsers` folder in your local **"App data directory"**:
 
@@ -231,18 +292,42 @@ Custom local parser plugins may be placed in a `jc/jcparsers` folder in your loc
 
 Local parser plugins are standard python module files. Use the [`jc/parsers/foo.py`](https://github.com/kellyjonbrazil/jc/blob/master/jc/parsers/foo.py) parser as a template and simply place a `.py` file in the `jcparsers` subfolder.
 
-Local plugin filenames must be valid python module names, therefore must consist entirely of alphanumerics and start with a letter. Local plugins may override default plugins.
+Local plugin filenames must be valid python module names, therefore must consist entirely of alphanumerics and start with a letter. Local plugins may override default parsers.
 
 > Note: The application data directory follows the [XDG Base Directory Specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html)
 
+### Caveats
+
+#### Locale
+
+For best results set the `LANG` locale environment variable to `C` or `en_US.UTF-8`. For example, either by setting directly on the command-line:
+```
+$ LANG=C date | jc --date
+```
+or by exporting to the environment before running commands:
+```
+$ export LANG=C
+```
+
+#### Timezones
+
+Some parsers have calculated epoch timestamp fields added to the output. Unless a timestamp field name has a `_utc` suffix it is considered naive. (i.e. based on the local timezone of the system the `jc` parser was run on).
+
+If a UTC timezone can be detected in the text of the command output, the timestamp will be timezone aware and have a `_utc` suffix on the key name. (e.g. `epoch_utc`) No other timezones are supported for aware timestamps.
+
 ## Compatibility
-Some parsers like `ls`, `ps`, `dig`, etc. will work on any platform. Other parsers that are platform-specific will generate a warning message if they are used on an unsupported platform. To see all parser information, including compatibility, run `jc -ap`.
+Some parsers like `dig`, `xml`, `csv`, etc. will work on any platform. Other parsers that convert platform-specific output will generate a warning message if they are run on an unsupported platform. To see all parser information, including compatibility, run `jc -ap`.
 
-You may still use a parser on an unsupported platform - for example, you may want to parse a file with linux `lsof` output on an macOS laptop. In that case you can suppress the warning message with the `-q` cli option or the `quiet=True` function parameter in `parse()`:
+You may still use a parser on an unsupported platform - for example, you may want to parse a file with linux `lsof` output on an macOS or Windows laptop. In that case you can suppress the warning message with the `-q` cli option or the `quiet=True` function parameter in `parse()`:
 
+macOS:
 ```bash
 cat lsof.out | jc --lsof -q
 ```
+or Windows:
+```bash
+type lsof.out | jc --lsof -q
+```
 
 Tested on:
 - Centos 7.7
@@ -254,9 +339,11 @@ Tested on:
 - NixOS
 - FreeBSD12
 - Windows 10
+- Windows 2016 Server
+- Windows 2019 Server
 
 ## Contributions
-Feel free to add/improve code or parsers! You can use the [`jc/parsers/foo.py`](https://github.com/kellyjonbrazil/jc/blob/master/jc/parsers/foo.py) parser as a template and submit your parser with a pull request.
+Feel free to add/improve code or parsers! You can use the [`jc/parsers/foo.py`](https://github.com/kellyjonbrazil/jc/blob/master/jc/parsers/foo.py) or [`jc/parsers/foo_s.py (streaming)`](https://github.com/kellyjonbrazil/jc/blob/master/jc/parsers/foo_s.py) parsers as a template and submit your parser with a pull request.
 
 Please see the [Contributing Guidelines](https://github.com/kellyjonbrazil/jc/blob/master/CONTRIBUTING.md) for more information.
 
@@ -352,48 +439,6 @@ cat homes.csv | jc --csv -p
   }
 ]
 ```
-### dig
-```bash
-dig cnn.com @205.251.194.64 | jc --dig -p          # or:  jc -p dig cnn.com @205.251.194.64
-```
-```json
-[
-  {
-    "id": 52172,
-    "opcode": "QUERY",
-    "status": "NOERROR",
-    "flags": [
-      "qr",
-      "rd",
-      "ra"
-    ],
-    "query_num": 1,
-    "answer_num": 1,
-    "authority_num": 0,
-    "additional_num": 1,
-    "question": {
-      "name": "cnn.com.",
-      "class": "IN",
-      "type": "A"
-    },
-    "answer": [
-      {
-        "name": "cnn.com.",
-        "class": "IN",
-        "type": "A",
-        "ttl": 27,
-        "data": "151.101.65.67"
-      }
-    ],
-    "query_time": 38,
-    "server": "2600",
-    "when": "Tue Mar 30 20:07:59 PDT 2021",
-    "rcvd": 100,
-    "when_epoch": 1617160079,
-    "when_epoch_utc": null
-  }
-]
-```
 ### /etc/hosts file
 ```bash
 cat /etc/hosts | jc --hosts -p

docs/parsers/acpi.md

@@ -252,4 +252,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux
 
-Version 1.1 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.2 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/airport.md

@@ -105,4 +105,4 @@ Returns:
 ## Parser Information
 Compatibility:  darwin
 
-Version 1.2 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.3 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/airport_s.md

@@ -133,4 +133,4 @@ Returns:
 ## Parser Information
 Compatibility:  darwin
 
-Version 1.3 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.4 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/blkid.md

@@ -145,4 +145,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux
 
-Version 1.3 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.4 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/cksum.md

@@ -79,4 +79,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux, darwin, cygwin, aix, freebsd
 
-Version 1.1 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.2 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/df.md

@@ -122,4 +122,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux, darwin, freebsd
 
-Version 1.6 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.7 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/dig.md

@@ -3,6 +3,10 @@
 # jc.parsers.dig
 jc - JSON CLI output utility `dig` command output parser
 
+Options supported:
+- `+noall +answer` options are supported in cases where only the answer information is desired.
+- `+axfr` option is supported on its own
+
 The `when_epoch` calculated timestamp field is naive (i.e. based on the local time of the system the parser is run on)
 
 The `when_epoch_utc` calculated timestamp field is timezone-aware and is only available if the timezone field is UTC.
@@ -24,64 +28,84 @@ Schema:
 
     [
       {
-        "id":             integer,
-        "opcode":         string,
-        "status":         string,
+        "id":                   integer,
+        "opcode":               string,
+        "status":               string,
         "flags": [
-                          string
+                                string
         ],
-        "query_num":      integer,
-        "answer_num":     integer,
-        "authority_num":  integer,
-        "additional_num": integer,
+        "query_num":            integer,
+        "answer_num":           integer,
+        "authority_num":        integer,
+        "additional_num":       integer,
         "axfr": [
           {
-            "name":       string,
-            "class":      string,
-            "type":       string,
-            "ttl":        integer,
-            "data":       string
+            "name":             string,
+            "class":            string,
+            "type":             string,
+            "ttl":              integer,
+            "data":             string
           }
         ],
+        "opt_pseudosection": {
+          "edns": {
+            "version":          integer,
+            "flags": [
+                                string
+            ],
+            "udp":              integer
+          },
+          "cookie":             string
+        },
         "question": {
-          "name":         string,
-          "class":        string,
-          "type":         string
+          "name":               string,
+          "class":              string,
+          "type":               string
         },
         "answer": [
           {
-            "name":       string,
-            "class":      string,
-            "type":       string,
-            "ttl":        integer,
-            "data":       string
+            "name":             string,
+            "class":            string,
+            "type":             string,
+            "ttl":              integer,
+            "data":             string
+          }
+        ],
+        "additional": [
+          {
+            "name":             string,
+            "class":            string,
+            "type":             string,
+            "ttl":              integer,
+            "data":             string
           }
         ],
         "authority": [
           {
-            "name":       string,
-            "class":      string,
-            "type":       string,
-            "ttl":        integer,
-            "data":       string
+            "name":             string,
+            "class":            string,
+            "type":             string,
+            "ttl":              integer,
+            "data":             string
           }
         ],
-        "query_time":     integer,   # in msec
-        "server":         string,
-        "when":           string,
-        "when_epoch":     integer,   # naive timestamp if when field is parsable, else null
-        "when_epoch_utc": integer,   # timezone aware timestamp availabe for UTC, else null
-        "rcvd":           integer
-        "size":           string
+        "query_size":           integer,
+        "query_time":           integer,   # in msec
+        "server":               string,
+        "when":                 string,
+        "when_epoch":           integer,   # naive timestamp if when field is parsable, else null
+        "when_epoch_utc":       integer,   # timezone aware timestamp availabe for UTC, else null
+        "rcvd":                 integer
+        "size":                 string
       }
     ]
 
 Examples:
 
-    $ dig cnn.com www.cnn.com @205.251.194.64 | jc --dig -p
+    $ dig example.com | jc --dig -p
     [
       {
-        "id": 52172,
+        "id": 2951,
         "opcode": "QUERY",
         "status": "NOERROR",
         "flags": [
@@ -90,113 +114,35 @@ Examples:
           "ra"
         ],
         "query_num": 1,
-        "answer_num": 4,
+        "answer_num": 1,
         "authority_num": 0,
         "additional_num": 1,
+        "opt_pseudosection": {
+          "edns": {
+            "version": 0,
+            "flags": [],
+            "udp": 4096
+          }
+        },
         "question": {
-          "name": "cnn.com.",
+          "name": "example.com.",
           "class": "IN",
           "type": "A"
         },
         "answer": [
           {
-            "name": "cnn.com.",
+            "name": "example.com.",
             "class": "IN",
             "type": "A",
-            "ttl": 27,
-            "data": "151.101.65.67"
-          },
-          {
-            "name": "cnn.com.",
-            "class": "IN",
-            "type": "A",
-            "ttl": 27,
-            "data": "151.101.129.67"
-          },
-          {
-            "name": "cnn.com.",
-            "class": "IN",
-            "type": "A",
-            "ttl": 27,
-            "data": "151.101.1.67"
-          },
-          {
-            "name": "cnn.com.",
-            "class": "IN",
-            "type": "A",
-            "ttl": 27,
-            "data": "151.101.193.67"
+            "ttl": 39302,
+            "data": "93.184.216.34"
           }
         ],
-        "query_time": 38,
-        "server": "2600",
-        "when": "Tue Mar 30 20:07:59 PDT 2021",
-        "rcvd": 100,
-        "when_epoch": 1617160079,
-        "when_epoch_utc": null
-      },
-      {
-        "id": 36292,
-        "opcode": "QUERY",
-        "status": "NOERROR",
-        "flags": [
-          "qr",
-          "aa",
-          "rd"
-        ],
-        "query_num": 1,
-        "answer_num": 1,
-        "authority_num": 4,
-        "additional_num": 1,
-        "question": {
-          "name": "www.cnn.com.",
-          "class": "IN",
-          "type": "A"
-        },
-        "answer": [
-          {
-            "name": "www.cnn.com.",
-            "class": "IN",
-            "type": "CNAME",
-            "ttl": 300,
-            "data": "turner-tls.map.fastly.net."
-          }
-        ],
-        "authority": [
-          {
-            "name": "cnn.com.",
-            "class": "IN",
-            "type": "NS",
-            "ttl": 3600,
-            "data": "ns-1086.awsdns-07.org."
-          },
-          {
-            "name": "cnn.com.",
-            "class": "IN",
-            "type": "NS",
-            "ttl": 3600,
-            "data": "ns-1630.awsdns-11.co.uk."
-          },
-          {
-            "name": "cnn.com.",
-            "class": "IN",
-            "type": "NS",
-            "ttl": 3600,
-            "data": "ns-47.awsdns-05.com."
-          },
-          {
-            "name": "cnn.com.",
-            "class": "IN",
-            "type": "NS",
-            "ttl": 3600,
-            "data": "ns-576.awsdns-08.net."
-          }
-        ],
-        "query_time": 27,
-        "server": "205.251.194.64#53(205.251.194.64)",
-        "when": "Tue Mar 30 20:07:59 PDT 2021",
-        "rcvd": 212,
-        "when_epoch": 1617160079,
+        "query_time": 49,
+        "server": "2600:1700:bab0:d40::1#53(2600:1700:bab0:d40::1)",
+        "when": "Fri Apr 16 16:05:10 PDT 2021",
+        "rcvd": 56,
+        "when_epoch": 1618614310,
         "when_epoch_utc": null
       }
     ]
@@ -204,7 +150,7 @@ Examples:
     $ dig cnn.com www.cnn.com @205.251.194.64 | jc --dig -p -r
     [
       {
-        "id": "23843",
+        "id": "46052",
         "opcode": "QUERY",
         "status": "NOERROR",
         "flags": [
@@ -213,117 +159,41 @@ Examples:
           "ra"
         ],
         "query_num": "1",
-        "answer_num": "4",
+        "answer_num": "1",
         "authority_num": "0",
         "additional_num": "1",
+        "opt_pseudosection": {
+          "edns": {
+            "version": "0",
+            "flags": [],
+            "udp": "4096"
+          }
+        },
         "question": {
-          "name": "cnn.com.",
+          "name": "example.com.",
           "class": "IN",
           "type": "A"
         },
         "answer": [
           {
-            "name": "cnn.com.",
+            "name": "example.com.",
             "class": "IN",
             "type": "A",
-            "ttl": "30",
-            "data": "151.101.193.67"
-          },
-          {
-            "name": "cnn.com.",
-            "class": "IN",
-            "type": "A",
-            "ttl": "30",
-            "data": "151.101.1.67"
-          },
-          {
-            "name": "cnn.com.",
-            "class": "IN",
-            "type": "A",
-            "ttl": "30",
-            "data": "151.101.65.67"
-          },
-          {
-            "name": "cnn.com.",
-            "class": "IN",
-            "type": "A",
-            "ttl": "30",
-            "data": "151.101.129.67"
+            "ttl": "40426",
+            "data": "93.184.216.34"
           }
         ],
-        "query_time": "24 msec",
-        "server": "192.168.1.254#53(192.168.1.254)",
-        "when": "Tue Nov 12 07:16:19 PST 2019",
-        "rcvd": "100"
-      },
-      {
-        "id": "8266",
-        "opcode": "QUERY",
-        "status": "NOERROR",
-        "flags": [
-          "qr",
-          "aa",
-          "rd"
-        ],
-        "query_num": "1",
-        "answer_num": "1",
-        "authority_num": "4",
-        "additional_num": "1",
-        "question": {
-          "name": "www.cnn.com.",
-          "class": "IN",
-          "type": "A"
-        },
-        "answer": [
-          {
-            "name": "www.cnn.com.",
-            "class": "IN",
-            "type": "CNAME",
-            "ttl": "300",
-            "data": "turner-tls.map.fastly.net."
-          }
-        ],
-        "authority": [
-          {
-            "name": "cnn.com.",
-            "class": "IN",
-            "type": "NS",
-            "ttl": "3600",
-            "data": "ns-1086.awsdns-07.org."
-          },
-          {
-            "name": "cnn.com.",
-            "class": "IN",
-            "type": "NS",
-            "ttl": "3600",
-            "data": "ns-1630.awsdns-11.co.uk."
-          },
-          {
-            "name": "cnn.com.",
-            "class": "IN",
-            "type": "NS",
-            "ttl": "3600",
-            "data": "ns-47.awsdns-05.com."
-          },
-          {
-            "name": "cnn.com.",
-            "class": "IN",
-            "type": "NS",
-            "ttl": "3600",
-            "data": "ns-576.awsdns-08.net."
-          }
-        ],
-        "query_time": "26 msec",
-        "server": "205.251.194.64#53(205.251.194.64)",
-        "when": "Tue Nov 12 07:16:19 PST 2019",
-        "rcvd": "212"
+        "query_time": "48 msec",
+        "server": "2600:1700:bab0:d40::1#53(2600:1700:bab0:d40::1)",
+        "when": "Fri Apr 16 16:06:12 PDT 2021",
+        "rcvd": "56"
       }
     ]
 
     $ dig -x 1.1.1.1 | jc --dig -p
     [
       {
-        "id": 22191,
+        "id": 20785,
         "opcode": "QUERY",
         "status": "NOERROR",
         "flags": [
@@ -335,6 +205,13 @@ Examples:
         "answer_num": 1,
         "authority_num": 0,
         "additional_num": 1,
+        "opt_pseudosection": {
+          "edns": {
+            "version": 0,
+            "flags": [],
+            "udp": 4096
+          }
+        },
         "question": {
           "name": "1.1.1.1.in-addr.arpa.",
           "class": "IN",
@@ -349,11 +226,11 @@ Examples:
             "data": "one.one.one.one."
           }
         ],
-        "query_time": 44,
-        "server": "2600",
-        "when": "Tue Mar 30 20:10:34 PDT 2021",
+        "query_time": 40,
+        "server": "2600:1700:bab0:d40::1#53(2600:1700:bab0:d40::1)",
+        "when": "Sat Apr 17 14:50:50 PDT 2021",
         "rcvd": 78,
-        "when_epoch": 1617160234,
+        "when_epoch": 1618696250,
         "when_epoch_utc": null
       }
     ]
@@ -361,7 +238,7 @@ Examples:
     $ dig -x 1.1.1.1 | jc --dig -p -r
     [
       {
-        "id": "50986",
+        "id": "32644",
         "opcode": "QUERY",
         "status": "NOERROR",
         "flags": [
@@ -373,6 +250,13 @@ Examples:
         "answer_num": "1",
         "authority_num": "0",
         "additional_num": "1",
+        "opt_pseudosection": {
+          "edns": {
+            "version": "0",
+            "flags": [],
+            "udp": "4096"
+          }
+        },
         "question": {
           "name": "1.1.1.1.in-addr.arpa.",
           "class": "IN",
@@ -387,13 +271,49 @@ Examples:
             "data": "one.one.one.one."
           }
         ],
-        "query_time": "38 msec",
-        "server": "2600",
-        "when": "Tue Nov 12 07:17:19 PST 2019",
+        "query_time": "52 msec",
+        "server": "2600:1700:bab0:d40::1#53(2600:1700:bab0:d40::1)",
+        "when": "Sat Apr 17 14:51:46 PDT 2021",
         "rcvd": "78"
       }
     ]
 
+    $ dig +noall +answer cnn.com | jc --dig -p
+    [
+      {
+        "answer": [
+          {
+            "name": "cnn.com.",
+            "class": "IN",
+            "type": "A",
+            "ttl": 60,
+            "data": "151.101.193.67"
+          },
+          {
+            "name": "cnn.com.",
+            "class": "IN",
+            "type": "A",
+            "ttl": 60,
+            "data": "151.101.65.67"
+          },
+          {
+            "name": "cnn.com.",
+            "class": "IN",
+            "type": "A",
+            "ttl": 60,
+            "data": "151.101.1.67"
+          },
+          {
+            "name": "cnn.com.",
+            "class": "IN",
+            "type": "A",
+            "ttl": 60,
+            "data": "151.101.129.67"
+          }
+        ]
+      }
+    ]
+
 
 ## info
 ```python
@@ -419,6 +339,6 @@ Returns:
     List of Dictionaries. Raw or processed structured data.
 
 ## Parser Information
-Compatibility:  linux, aix, freebsd, darwin
+Compatibility:  linux, aix, freebsd, darwin, win32, cygwin
 
-Version 1.7 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 2.1 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/dir.md

@@ -9,16 +9,14 @@ Options supported:
 - `/C, /-C`
 - `/S`
 
+The "Magic" syntax is not supported since the `dir` command is a shell builtin.
+
 The `epoch` calculated timestamp field is naive (i.e. based on the local time of the system the parser is run on)
 
 Usage (cli):
 
     C:> dir | jc --dir
 
-    or
-
-    C:> jc dir
-
 Usage (module):
 
     import jc.parsers.dir
@@ -145,4 +143,4 @@ Returns:
 ## Parser Information
 Compatibility:  win32
 
-Version 1.1 by Rasheed Elsaleh (rasheed@rebelliondefense.com)
+Version 1.3 by Rasheed Elsaleh (rasheed@rebelliondefense.com)

docs/parsers/dmidecode.md

@@ -150,4 +150,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux
 
-Version 1.2 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.3 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/du.md

@@ -112,4 +112,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux, darwin, aix, freebsd
 
-Version 1.3 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.4 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/free.md

@@ -97,4 +97,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux
 
-Version 1.3 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.4 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/fstab.md

@@ -110,4 +110,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux, freebsd
 
-Version 1.4 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.5 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/group.md

@@ -134,4 +134,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux, darwin, aix, freebsd
 
-Version 1.2 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.3 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/hash.md

@@ -62,4 +62,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux, darwin, cygwin, aix, freebsd
 
-Version 1.1 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.2 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/hciconfig.md

@@ -342,4 +342,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux
 
-Version 1.1 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.2 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/history.md

@@ -5,6 +5,8 @@ jc - JSON CLI output utility `history` command output parser
 
 This parser will output a list of dictionaries each containing `line` and `command` keys. If you would like a simple dictionary output, then use the `-r` command-line option or the `raw=True` argument in the `parse()` function.
 
+The "Magic" syntax is not supported since the `history` command is a shell builtin.
+
 Usage (cli):
 
     $ history | jc --history
@@ -83,4 +85,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux, darwin, cygwin, aix, freebsd
 
-Version 1.4 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.5 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/id.md

@@ -130,4 +130,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux, darwin, aix, freebsd
 
-Version 1.2 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.3 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/ifconfig.md

@@ -211,4 +211,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux, aix, freebsd, darwin
 
-Version 1.9 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.10 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/iptables.md

@@ -188,4 +188,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux
 
-Version 1.5 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.6 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/jobs.md

@@ -5,14 +5,12 @@ jc - JSON CLI output utility `jobs` command output parser
 
 Also supports the `-l` option.
 
+The "Magic" syntax is not supported since the `jobs` command is a shell builtin.
+
 Usage (cli):
 
     $ jobs | jc --jobs
 
-    or
-
-    $ jc jobs
-
 Usage (module):
 
     import jc.parsers.jobs
@@ -119,4 +117,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux, darwin, cygwin, aix, freebsd
 
-Version 1.3 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.4 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/kv.md

@@ -38,7 +38,7 @@ Examples:
     # but can be preserved with the -r argument
     occupation:"Engineer"
 
-    $ cat keyvalue.txt | jc --ini -p
+    $ cat keyvalue.txt | jc --kv -p
     {
       "name": "John Doe",
       "address": "555 California Drive",

docs/parsers/last.md

@@ -128,4 +128,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux, darwin, aix, freebsd
 
-Version 1.6 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.7 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/ls.md

@@ -6,7 +6,6 @@ jc - JSON CLI output utility `ls` and `vdir` command output parser
 Options supported:
 - `lbaR1`
 - `--time-style=full-iso`
-- `-h`: File sizes will be available in text form with `-r` but larger file sizes with human readable suffixes will be converted to `Null` in the default view since the parser attempts to convert this field to an integer.
 
 Note: The `-1`, `-l`, or `-b` option of `ls` should be used to correctly parse filenames that include newline characters. Since `ls` does not encode newlines in filenames when outputting to a pipe it will cause `jc` to see multiple files instead of a single file if `-1`, `-l`, or `-b` is not used. Alternatively, `vdir` can be used, which is the same as running `ls -lb`.
 
@@ -132,4 +131,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux, darwin, cygwin, aix, freebsd
 
-Version 1.8 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.9 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/ls_s.md

@@ -0,0 +1,92 @@
+[Home](https://kellyjonbrazil.github.io/jc/)
+
+# jc.parsers.ls_s
+jc - JSON CLI output utility `ls` and `vdir` command output streaming parser
+
+This streaming parser requires the `-l` option to be used on `ls`. If there are newline characters in the filename, then make sure to use the `-b` option on `ls`.
+
+The `jc` `-q` option can be used to ignore parsing errors. (e.g. filenames with newline characters, but `-b` was not used)
+
+The `epoch` calculated timestamp field is naive (i.e. based on the local time of the system the parser is run on)
+
+The `epoch_utc` calculated timestamp field is timezone-aware and is only available if the timezone field is UTC.
+
+Usage (cli):
+
+    $ ls | jc --ls-s
+
+Usage (module):
+
+    import jc.parsers.ls_s
+    result = jc.parsers.ls_s.parse(ls_command_output.splitlines())    # result is an iterable object
+    for item in result:
+        # do something
+
+Schema:
+
+    {
+      "filename":       string,
+      "flags":          string,
+      "links":          integer,
+      "parent":         string,
+      "owner":          string,
+      "group":          string,
+      "size":           integer,
+      "date":           string,
+      "epoch":          integer,     # naive timestamp if date field exists and can be converted
+      "epoch_utc":      integer,     # timezone aware timestamp if date field is in UTC and can be converted
+      "_jc_meta":                    # This object only exists if using -qq or ignore_exceptions=True
+        {
+          "success":    booean,      # true if successfully parsed, false if error
+          "error":      string,      # exists if "success" is false
+          "line":       string       # exists if "success" is false
+        }
+    }
+
+Examples:
+
+    $ ls -l /usr/bin | jc --ls-s
+    {"filename":"2to3-","flags":"-rwxr-xr-x","links":4,"owner":"root","group":"wheel","size":925,"date":"Feb 22 2019"}
+    {"filename":"2to3-2.7","link_to":"../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/2to3-2.7","flags":"lrwxr-xr-x","links":1,"owner":"root","group":"wheel","size":74,"date":"May 4 2019"}
+    {"filename":"AssetCacheLocatorUtil","flags":"-rwxr-xr-x","links":1,"owner":"root","group":"wheel","size":55152,"date":"May 3 2019"}
+    ...
+
+    $ ls -l /usr/bin | jc --ls-s -r
+    {"filename":"2to3-","flags":"-rwxr-xr-x","links":"4","owner":"root","group":"wheel","size":"925","date":"Feb 22 2019"}
+    {"filename":"2to3-2.7","link_to":"../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/2to3-2.7","flags":"lrwxr-xr-x","links":"1","owner":"root","group":"wheel","size":"74","date":"May 4 2019"}
+    {"filename":"AssetCacheLocatorUtil","flags":"-rwxr-xr-x","links":"1","owner":"root","group":"wheel","size":"55152","date":"May 3 2019"}
+    ...
+
+
+## info
+```python
+info()
+```
+Provides parser metadata (version, author, etc.)
+
+## parse
+```python
+parse(data, raw=False, quiet=False, ignore_exceptions=False)
+```
+
+Main text parsing generator function. Returns an iterator object.
+
+Parameters:
+
+    data:              (iterable)  line-based text data to parse (e.g. sys.stdin or str.splitlines())
+    raw:               (boolean)   output preprocessed JSON if True
+    quiet:             (boolean)   suppress warning messages if True
+    ignore_exceptions: (boolean)   ignore parsing exceptions if True
+
+Yields:
+
+    Dictionary. Raw or processed structured data.
+
+Returns:
+
+    Iterator object
+
+## Parser Information
+Compatibility:  linux, darwin, cygwin, aix, freebsd
+
+Version 0.5 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/lsblk.md

@@ -293,4 +293,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux
 
-Version 1.6 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.7 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/lsmod.md

@@ -150,4 +150,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux
 
-Version 1.4 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.5 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/lsof.md

@@ -144,4 +144,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux
 
-Version 1.3 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.4 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/netstat.md

@@ -379,4 +379,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux, darwin, freebsd
 
-Version 1.9 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.10 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/ntpq.md

@@ -231,4 +231,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux, freebsd
 
-Version 1.4 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.5 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/passwd.md

@@ -119,4 +119,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux, darwin, aix, freebsd
 
-Version 1.2 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.3 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/ping.md

@@ -38,14 +38,26 @@ Schema:
       "round_trip_ms_stddev":        float,
       "responses": [
         {
-          "type":                    string,        # ('reply' or 'timeout')
+          "type":                    string,        # 'reply', 'timeout', 'unparsable_line', etc. See `_error_type.type_map` for all options
+          "unparsed_line":           string,        # only if an 'unparsable_line' type
           "timestamp":               float,
           "bytes":                   integer,
           "response_ip":             string,
           "icmp_seq":                integer,
           "ttl":                     integer,
           "time_ms":                 float,
-          "duplicate":               boolean
+          "duplicate":               boolean,
+          "vr":                      integer,       # hex value converted to decimal
+          "hl":                      integer,       # hex value converted to decimal
+          "tos":                     integer,       # hex value converted to decimal
+          "len":                     integer,       # hex value converted to decimal
+          "id":                      integer,       # hex value converted to decimal
+          "flg":                     integer,       # hex value converted to decimal
+          "off":                     integer,       # hex value converted to decimal
+          "pro":                     integer,       # hex value converted to decimal
+          "cks":                     ingeger,       # hex value converted to decimal
+          "src":                     string,
+          "dst":                     string
         }
       ]
     }
@@ -97,7 +109,6 @@ Examples:
       ]
     }
 
-
     $ ping -c 3 -p ff cnn.com | jc --ping -p -r
     {
       "destination_ip": "151.101.129.67",
@@ -170,4 +181,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux, darwin, freebsd
 
-Version 1.3 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.5 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/ping_s.md

@@ -0,0 +1,99 @@
+[Home](https://kellyjonbrazil.github.io/jc/)
+
+# jc.parsers.ping_s
+jc - JSON CLI output utility `ping` command output streaming parser
+
+Supports `ping` and `ping6` output.
+
+Usage (cli):
+
+    $ ping | jc --ping-s
+
+> Note: When piping `jc` converted ping output to other processes it may appear the output is hanging due to the OS pipe buffers. This is because `ping` output is too small to quickly fill up the buffer. Use the `-u` option to unbuffer the `jc` output if you would like immediate output. See the [readme](https://github.com/kellyjonbrazil/jc/tree/streaming#streaming-parsers) for more information.
+
+Usage (module):
+
+    import jc.parsers.ping_s
+    result = jc.parsers.ping_s.parse(ping_command_output.splitlines())    # result is an iterable object
+    for item in result:
+        # do something
+
+Schema:
+
+    {
+      "type":                        string,        # 'reply', 'timeout', 'summary', etc. See `_error_type.type_map` for all options.
+      "source_ip":                   string,
+      "destination_ip":              string,
+      "sent_bytes":                  integer,
+      "pattern":                     string,        # (null if not set)
+      "destination":                 string,
+      "timestamp":                   float,
+      "response_bytes":              integer,
+      "response_ip":                 string,
+      "icmp_seq":                    integer,
+      "ttl":                         integer,
+      "time_ms":                     float,
+      "duplicate":                   boolean,
+      "packets_transmitted":         integer,
+      "packets_received":            integer,
+      "packet_loss_percent":         float,
+      "duplicates":                  integer,
+      "round_trip_ms_min":           float,
+      "round_trip_ms_avg":           float,
+      "round_trip_ms_max":           float,
+      "round_trip_ms_stddev":        float,
+      "_jc_meta":                                  # This object only exists if using -qq or ignore_exceptions=True
+        {
+          "success":                 booean,       # true if successfully parsed, false if error
+          "error":                   string,       # exists if "success" is false
+          "line":                    string        # exists if "success" is false
+        }
+    }
+
+Examples:
+
+    $ ping 1.1.1.1 | jc --ping-s
+    {"type":"reply","destination_ip":"1.1.1.1","sent_bytes":56,"pattern":null,"response_bytes":64,"response_ip":"1.1.1.1","icmp_seq":0,"ttl":56,"time_ms":23.703}
+    {"type":"reply","destination_ip":"1.1.1.1","sent_bytes":56,"pattern":null,"response_bytes":64,"response_ip":"1.1.1.1","icmp_seq":1,"ttl":56,"time_ms":22.862}
+    {"type":"reply","destination_ip":"1.1.1.1","sent_bytes":56,"pattern":null,"response_bytes":64,"response_ip":"1.1.1.1","icmp_seq":2,"ttl":56,"time_ms":22.82}
+    ...
+
+    $ ping 1.1.1.1 | jc --ping-s -r
+    {"type":"reply","destination_ip":"1.1.1.1","sent_bytes":"56","pattern":null,"response_bytes":"64","response_ip":"1.1.1.1","icmp_seq":"0","ttl":"56","time_ms":"23.054"}
+    {"type":"reply","destination_ip":"1.1.1.1","sent_bytes":"56","pattern":null,"response_bytes":"64","response_ip":"1.1.1.1","icmp_seq":"1","ttl":"56","time_ms":"24.739"}
+    {"type":"reply","destination_ip":"1.1.1.1","sent_bytes":"56","pattern":null,"response_bytes":"64","response_ip":"1.1.1.1","icmp_seq":"2","ttl":"56","time_ms":"23.232"}
+    ...
+
+
+## info
+```python
+info()
+```
+Provides parser metadata (version, author, etc.)
+
+## parse
+```python
+parse(data, raw=False, quiet=False, ignore_exceptions=False)
+```
+
+Main text parsing generator function. Returns an iterator object.
+
+Parameters:
+
+    data:              (iterable)  line-based text data to parse (e.g. sys.stdin or str.splitlines())
+    raw:               (boolean)   output preprocessed JSON if True
+    quiet:             (boolean)   suppress warning messages if True
+    ignore_exceptions: (boolean)   ignore parsing exceptions if True
+
+Yields:
+
+    Dictionary. Raw or processed structured data.
+
+Returns:
+
+    Iterator object
+
+## Parser Information
+Compatibility:  linux, darwin, freebsd
+
+Version 0.5 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/ps.md

@@ -231,4 +231,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux, darwin, cygwin, aix, freebsd
 
-Version 1.4 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.5 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/route.md

@@ -25,7 +25,7 @@ Schema:
         "genmask":         string,
         "flags":           string,
         "flags_pretty": [
-                           string,
+                           string
         ]
         "metric":          integer,
         "ref":             integer,
@@ -76,7 +76,6 @@ Examples:
       }
     ]
 
-
     $ route -ee | jc --route -p -r
     [
       {
@@ -108,7 +107,6 @@ Examples:
     ]
 
 
-
 ## info
 ```python
 info()
@@ -135,4 +133,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux
 
-Version 1.5 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.6 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/rpm_qi.md

@@ -181,4 +181,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux
 
-Version 1.2 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.3 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/sfdisk.md

@@ -0,0 +1,228 @@
+[Home](https://kellyjonbrazil.github.io/jc/)
+
+# jc.parsers.sfdisk
+jc - JSON CLI output utility `sfdisk` command output parser
+
+Supports the following `sfdisk` options:
+- `-l`
+- `-F`
+- `-d`   (deprecated - only for older versions of util-linux)
+- `-uM`  (deprecated - only for older versions of util-linux)
+- `-uC`  (deprecated - only for older versions of util-linux)
+- `-uS`  (deprecated - only for older versions of util-linux)
+- `-uB`  (deprecated - only for older versions of util-linux)
+
+Usage (cli):
+
+    # sfdisk -l | jc --sfdisk
+
+    or
+
+    # jc sfdisk -l
+
+Usage (module):
+
+    import jc.parsers.sfdisk
+    result = jc.parsers.sfdisk.parse(sfdisk_command_output)
+
+Schema:
+
+    [
+      {
+        "disk":                     string,
+        "disk_size":                string,
+        "free_disk_size":           string,
+        "bytes":                    integer,
+        "free_bytes":               integer,
+        "sectors":                  integer,
+        "free_sectors":             integer,
+        "cylinders":                integer,
+        "heads":                    integer,
+        "sectors_per_track":        integer,
+        "units":                    string,
+        "logical_sector_size":      integer,
+        "physical_sector_size":     integer,
+        "min_io_size":              integer,
+        "optimal_io_size":          integer,
+        "disk_label_type":          string,
+        "disk_identifier":          string,
+        "disk_model":               string,
+        "partitions": [
+          {
+            "device":               string,
+            "boot":                 boolean,
+            "start":                integer,
+            "end":                  integer,
+            "size":                 string,    # Note: will be integer when using deprecated -d sfdisk option
+            "cyls":                 integer,
+            "mib":                  integer,
+            "blocks":               integer,
+            "sectors":              integer,
+            "id":                   string,
+            "system":               string,
+            "type":                 string
+          }
+        ]
+      }
+    ]
+
+Examples:
+
+    # sfdisk -l | jc --sfdisk -p
+    [
+      {
+        "disk": "/dev/sda",
+        "cylinders": 2610,
+        "heads": 255,
+        "sectors_per_track": 63,
+        "units": "cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0",
+        "partitions": [
+          {
+            "device": "/dev/sda1",
+            "boot": true,
+            "start": 0,
+            "end": 130,
+            "cyls": 131,
+            "blocks": 1048576,
+            "id": "83",
+            "system": "Linux"
+          },
+          {
+            "device": "/dev/sda2",
+            "boot": false,
+            "start": 130,
+            "end": 2610,
+            "cyls": 2481,
+            "blocks": 19921920,
+            "id": "8e",
+            "system": "Linux LVM"
+          },
+          {
+            "device": "/dev/sda3",
+            "boot": false,
+            "start": 0,
+            "end": null,
+            "cyls": 0,
+            "blocks": 0,
+            "id": "0",
+            "system": "Empty"
+          },
+          {
+            "device": "/dev/sda4",
+            "boot": false,
+            "start": 0,
+            "end": null,
+            "cyls": 0,
+            "blocks": 0,
+            "id": "0",
+            "system": "Empty"
+          }
+        ]
+      },
+      {
+        "disk": "/dev/mapper/centos-root",
+        "cylinders": 2218,
+        "heads": 255,
+        "sectors_per_track": 63
+      },
+      {
+        "disk": "/dev/mapper/centos-swap",
+        "cylinders": 261,
+        "heads": 255,
+        "sectors_per_track": 63
+      }
+    ]
+
+    # sfdisk -l | jc --sfdisk -p -r
+    [
+      {
+        "disk": "/dev/sda",
+        "cylinders": "2610",
+        "heads": "255",
+        "sectors_per_track": "63",
+        "units": "cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0",
+        "partitions": [
+          {
+            "device": "/dev/sda1",
+            "boot": "*",
+            "start": "0+",
+            "end": "130-",
+            "cyls": "131-",
+            "blocks": "1048576",
+            "id": "83",
+            "system": "Linux"
+          },
+          {
+            "device": "/dev/sda2",
+            "boot": null,
+            "start": "130+",
+            "end": "2610-",
+            "cyls": "2481-",
+            "blocks": "19921920",
+            "id": "8e",
+            "system": "Linux LVM"
+          },
+          {
+            "device": "/dev/sda3",
+            "boot": null,
+            "start": "0",
+            "end": "-",
+            "cyls": "0",
+            "blocks": "0",
+            "id": "0",
+            "system": "Empty"
+          },
+          {
+            "device": "/dev/sda4",
+            "boot": null,
+            "start": "0",
+            "end": "-",
+            "cyls": "0",
+            "blocks": "0",
+            "id": "0",
+            "system": "Empty"
+          }
+        ]
+      },
+      {
+        "disk": "/dev/mapper/centos-root",
+        "cylinders": "2218",
+        "heads": "255",
+        "sectors_per_track": "63"
+      },
+      {
+        "disk": "/dev/mapper/centos-swap",
+        "cylinders": "261",
+        "heads": "255",
+        "sectors_per_track": "63"
+      }
+    ]
+
+
+## info
+```python
+info()
+```
+Provides parser metadata (version, author, etc.)
+
+## parse
+```python
+parse(data, raw=False, quiet=False)
+```
+
+Main text parsing function
+
+Parameters:
+
+    data:        (string)  text data to parse
+    raw:         (boolean) output preprocessed JSON if True
+    quiet:       (boolean) suppress warning messages if True
+
+Returns:
+
+    List of Dictionaries. Raw or processed structured data.
+
+## Parser Information
+Compatibility:  linux
+
+Version 1.1 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/shadow.md

@@ -126,4 +126,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux, darwin, aix, freebsd
 
-Version 1.2 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.3 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/ss.md

@@ -303,4 +303,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux
 
-Version 1.3 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.4 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/stat.md

@@ -193,4 +193,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux, darwin, freebsd
 
-Version 1.7 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.9 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/systemctl_lj.md

@@ -100,4 +100,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux
 
-Version 1.4 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.5 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/systeminfo.md

@@ -0,0 +1,232 @@
+[Home](https://kellyjonbrazil.github.io/jc/)
+
+# jc.parsers.systeminfo
+jc - JSON CLI output utility `systeminfo` command output parser
+
+Blank or missing elements are set to `null`.
+
+The `original_install_date_epoch` and `system_boot_time_epoch` calculated timestamp fields are naive (i.e. based on the local time of the system the parser is run on)
+
+The `original_install_date_epoch_utc` and `system_boot_time_epoch_utc` calculated timestamp fields are timezone-aware and are only available if the timezone field is UTC.
+
+Usage (cli):
+
+    $ systeminfo | jc --systeminfo
+
+Usage (module):
+
+    import jc.parsers.systeminfo
+    result = jc.parsers.systeminfo.parse(systeminfo_command_output)
+
+Schema:
+
+    {
+      "host_name":                                  string,
+      "os_name":                                    string,
+      "os_version":                                 string,
+      "os_manufacturer":                            string,
+      "os_configuration":                           string,
+      "os_build_type":                              string,
+      "registered_owner":                           string,
+      "registered_organization":                    string,
+      "product_id":                                 string,
+      "original_install_date":                      string,
+      "original_install_date_epoch":                integer,     # naive timestamp
+      "original_install_date_epoch_utc":            integer,     # timezone-aware timestamp
+      "system_boot_time":                           string,
+      "system_boot_time_epoch":                     integer,     # naive timestamp
+      "system_boot_time_epoch_utc":                 integer,     # timezone-aware timestamp
+      "system_manufacturer":                        string,
+      "system_model":                               string,
+      "system_type":                                string,
+      "processors": [
+                                                    string
+      ],
+      "bios_version":                               string,
+      "windows_directory":                          string,
+      "system_directory":                           string,
+      "boot_device":                                string,
+      "system_locale":                              string,
+      "input_locale":                               string,
+      "time_zone":                                  string,
+      "total_physical_memory_mb":                   string,
+      "available_physical_memory_mb":               integer,
+      "virtual_memory_max_size_mb":                 integer,
+      "virtual_memory_available_mb":                integer,
+      "virtual_memory_in_use_mb":                   integer,
+      "page_file_locations":                        string,
+      "domain":                                     string,
+      "logon_server":                               string,
+      "hotfixs": [
+                                                    string
+      ],
+      "network_cards": [
+        {
+          "name":                                   string,
+          "connection_name":                        string,
+          "status":                                 string,
+          "dhcp_enabled":                           boolean,
+          "dhcp_server":                            string,
+          "ip_addresses": [
+                                                    string
+          ]
+        }
+      ],
+      "hyperv_requirements": {
+        "vm_monitor_mode_extensions":               boolean,
+        "virtualization_enabled_in_firmware":       boolean,
+        "second_level_address_translation":         boolean,
+        "data_execution_prevention_available":      boolean
+      }
+    }
+
+Examples:
+
+    $ systeminfo | jc --systeminfo -p
+    {
+      "host_name": "TESTLAPTOP",
+      "os_name": "Microsoft Windows 10 Enterprise",
+      "os_version": "10.0.17134 N/A Build 17134",
+      "os_manufacturer": "Microsoft Corporation",
+      "os_configuration": "Member Workstation",
+      "os_build_type": "Multiprocessor Free",
+      "registered_owner": "Test, Inc.",
+      "registered_organization": "Test, Inc.",
+      "product_id": "11111-11111-11111-AA111",
+      "original_install_date": "3/26/2019, 3:51:30 PM",
+      "system_boot_time": "3/30/2021, 6:13:59 AM",
+      "system_manufacturer": "Dell Inc.",
+      "system_model": "Precision 5530",
+      "system_type": "x64-based PC",
+      "processors": [
+        "Intel64 Family 6 Model 158 Stepping 10 GenuineIntel ~2592 Mhz"
+      ],
+      "bios_version": "Dell Inc. 1.16.2, 4/21/2020",
+      "windows_directory": "C:\WINDOWS",
+      "system_directory": "C:\WINDOWS\system32",
+      "boot_device": "\Device\HarddiskVolume2",
+      "system_locale": "en-us;English (United States)",
+      "input_locale": "en-us;English (United States)",
+      "time_zone": "(UTC+00:00) UTC",
+      "total_physical_memory_mb": 32503,
+      "available_physical_memory_mb": 19743,
+      "virtual_memory_max_size_mb": 37367,
+      "virtual_memory_available_mb": 22266,
+      "virtual_memory_in_use_mb": 15101,
+      "page_file_locations": "C:\pagefile.sys",
+      "domain": "test.com",
+      "logon_server": "\\TESTDC01",
+      "hotfixs": [
+        "KB2693643",
+        "KB4601054"
+      ],
+      "network_cards": [
+        {
+          "name": "Intel(R) Wireless-AC 9260 160MHz",
+          "connection_name": "Wi-Fi",
+          "status": null,
+          "dhcp_enabled": true,
+          "dhcp_server": "192.168.2.1",
+          "ip_addresses": [
+            "192.168.2.219"
+          ]
+        }
+      ],
+      "hyperv_requirements": {
+        "vm_monitor_mode_extensions": true,
+        "virtualization_enabled_in_firmware": true,
+        "second_level_address_translation": false,
+        "data_execution_prevention_available": true
+      },
+      "original_install_date_epoch": 1553640690,
+      "original_install_date_epoch_utc": 1553615490,
+      "system_boot_time_epoch": 1617110039,
+      "system_boot_time_epoch_utc": 1617084839
+    }
+
+    $ systeminfo | jc --systeminfo -p -r
+    {
+      "host_name": "TESTLAPTOP",
+      "os_name": "Microsoft Windows 10 Enterprise",
+      "os_version": "10.0.17134 N/A Build 17134",
+      "os_manufacturer": "Microsoft Corporation",
+      "os_configuration": "Member Workstation",
+      "os_build_type": "Multiprocessor Free",
+      "registered_owner": "Test, Inc.",
+      "registered_organization": "Test, Inc.",
+      "product_id": "11111-11111-11111-AA111",
+      "original_install_date": "3/26/2019, 3:51:30 PM",
+      "system_boot_time": "3/30/2021, 6:13:59 AM",
+      "system_manufacturer": "Dell Inc.",
+      "system_model": "Precision 5530",
+      "system_type": "x64-based PC",
+      "processors": [
+        "Intel64 Family 6 Model 158 Stepping 10 GenuineIntel ~2592 Mhz"
+      ],
+      "bios_version": "Dell Inc. 1.16.2, 4/21/2020",
+      "windows_directory": "C:\WINDOWS",
+      "system_directory": "C:\WINDOWS\system32",
+      "boot_device": "\Device\HarddiskVolume2",
+      "system_locale": "en-us;English (United States)",
+      "input_locale": "en-us;English (United States)",
+      "time_zone": "(UTC+00:00) UTC",
+      "total_physical_memory_mb": "32,503 MB",
+      "available_physical_memory_mb": "19,743 MB",
+      "virtual_memory_max_size_mb": "37,367 MB",
+      "virtual_memory_available_mb": "22,266 MB",
+      "virtual_memory_in_use_mb": "15,101 MB",
+      "page_file_locations": "C:\pagefile.sys",
+      "domain": "test.com",
+      "logon_server": "\\TESTDC01",
+      "hotfixs": [
+        "KB2693643",
+        "KB4601054"
+      ],
+      "network_cards": [
+        {
+          "name": "Intel(R) Wireless-AC 9260 160MHz",
+          "connection_name": "Wi-Fi",
+          "status": "",
+          "dhcp_enabled": "Yes",
+          "dhcp_server": "192.168.2.1",
+          "ip_addresses": [
+            "192.168.2.219"
+          ]
+        }
+      ],
+      "hyperv_requirements": {
+        "vm_monitor_mode_extensions": "Yes",
+        "virtualization_enabled_in_firmware": "Yes",
+        "second_level_address_translation": "No",
+        "data_execution_prevention_available": "Yes"
+      }
+    }
+
+
+## info
+```python
+info()
+```
+Provides parser metadata (version, author, etc.)
+
+## parse
+```python
+parse(data, raw=False, quiet=False)
+```
+
+Main text parsing function
+
+Parameters:
+
+    data:        (string)  text data to parse
+    raw:         (boolean) output preprocessed JSON if True
+    quiet:       (boolean) suppress warning messages if True
+
+Returns:
+
+    List of Dictionaries. Raw or processed structured data.
+
+## Parser Information
+Compatibility:  win32
+
+Version 1.0 by Jon Smith (jon@rebelliondefense.com)

docs/parsers/time.md

@@ -148,4 +148,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux, darwin, cygwin, aix, freebsd
 
-Version 1.1 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.2 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/timedatectl.md

@@ -88,4 +88,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux
 
-Version 1.3 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.4 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/tracepath.md

@@ -156,4 +156,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux
 
-Version 1.1 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.2 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/traceroute.md

@@ -142,4 +142,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux, darwin, freebsd
 
-Version 1.2 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.3 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/ufw.md

@@ -0,0 +1,225 @@
+[Home](https://kellyjonbrazil.github.io/jc/)
+
+# jc.parsers.ufw
+jc - JSON CLI output utility `ufw status` command output parser
+
+Usage (cli):
+
+    $ ufw status | jc --ufw
+
+    or
+
+    $ jc ufw status
+
+Usage (module):
+
+    import jc.parsers.ufw
+    result = jc.parsers.ufw.parse(ufw_command_output)
+
+Schema:
+
+    {
+      "status":                     string,
+      "logging":                    string,
+      "logging_level":              string,
+      "default":                    string,
+      "new_profiles":               string,
+      "rules": [
+        {
+          "action":                 string,
+          "action_direction":       string,     # null if blank
+          "index":                  integer,    # null if blank
+          "network_protocol":       string,
+          "to_ip":                  string,
+          "to_ip_prefix":           integer,
+          "to_interface":           string,
+          "to_transport":           string,
+          "to_ports": [
+                                    integer
+          ],
+          "to_port_ranges": [
+            {
+              "start":              integer,
+              "end":                integer
+            }
+          ],
+          "to_service":             string,     # null if any to ports or port_ranges are set
+          "from_ip":                string,
+          "from_ip_prefix":         integer,
+          "from_interface":         string,
+          "from_transport":         string,
+          "from_ports": [
+                                    integer
+          ],
+          "from_port_ranges": [
+            {
+              "start":              integer,
+              "end":                integer
+            }
+          ],
+          "from_service":           string,     # null if any from ports or port_ranges are set
+          "comment":                string      # null if no comment
+        }
+      ]
+    }
+
+Examples:
+
+    $ ufw status verbose | jc --ufw -p
+    {
+      "status": "active",
+      "logging": "on",
+      "logging_level": "low",
+      "default": "deny (incoming), allow (outgoing), disabled (routed)",
+      "new_profiles": "skip",
+      "rules": [
+        {
+          "action": "ALLOW",
+          "action_direction": "IN",
+          "index": null,
+          "network_protocol": "ipv4",
+          "to_interface": "any",
+          "to_transport": "any",
+          "to_service": null,
+          "to_ports": [
+            22
+          ],
+          "to_ip": "0.0.0.0",
+          "to_ip_prefix": 0,
+          "comment": null,
+          "from_ip": "0.0.0.0",
+          "from_ip_prefix": 0,
+          "from_interface": "any",
+          "from_transport": "any",
+          "from_port_ranges": [
+            {
+              "start": 0,
+              "end": 65535
+            }
+          ],
+          "from_service": null
+        },
+        {
+          "action": "ALLOW",
+          "action_direction": "IN",
+          "index": null,
+          "network_protocol": "ipv4",
+          "to_interface": "any",
+          "to_transport": "tcp",
+          "to_service": null,
+          "to_ports": [
+            80,
+            443
+          ],
+          "to_ip": "0.0.0.0",
+          "to_ip_prefix": 0,
+          "comment": null,
+          "from_ip": "0.0.0.0",
+          "from_ip_prefix": 0,
+          "from_interface": "any",
+          "from_transport": "any",
+          "from_port_ranges": [
+            {
+              "start": 0,
+              "end": 65535
+            }
+          ],
+          "from_service": null
+        },
+        ...
+      ]
+    }
+
+    $ ufw status verbose | jc --ufw -p -r
+    {
+      "status": "active",
+      "logging": "on",
+      "logging_level": "low",
+      "default": "deny (incoming), allow (outgoing), disabled (routed)",
+      "new_profiles": "skip",
+      "rules": [
+        {
+          "action": "ALLOW",
+          "action_direction": "IN",
+          "index": null,
+          "network_protocol": "ipv4",
+          "to_interface": "any",
+          "to_transport": "any",
+          "to_service": null,
+          "to_ports": [
+            "22"
+          ],
+          "to_ip": "0.0.0.0",
+          "to_ip_prefix": "0",
+          "comment": null,
+          "from_ip": "0.0.0.0",
+          "from_ip_prefix": "0",
+          "from_interface": "any",
+          "from_transport": "any",
+          "from_port_ranges": [
+            {
+              "start": "0",
+              "end": "65535"
+            }
+          ],
+          "from_service": null
+        },
+        {
+          "action": "ALLOW",
+          "action_direction": "IN",
+          "index": null,
+          "network_protocol": "ipv4",
+          "to_interface": "any",
+          "to_transport": "tcp",
+          "to_service": null,
+          "to_ports": [
+            "80",
+            "443"
+          ],
+          "to_ip": "0.0.0.0",
+          "to_ip_prefix": "0",
+          "comment": null,
+          "from_ip": "0.0.0.0",
+          "from_ip_prefix": "0",
+          "from_interface": "any",
+          "from_transport": "any",
+          "from_port_ranges": [
+            {
+              "start": "0",
+              "end": "65535"
+            }
+          ],
+          "from_service": null
+        },
+        ...
+      ]
+    }
+
+
+## info
+```python
+info()
+```
+Provides parser metadata (version, author, etc.)
+
+## parse
+```python
+parse(data, raw=False, quiet=False)
+```
+
+Main text parsing function
+
+Parameters:
+
+    data:        (string)  text data to parse
+    raw:         (boolean) output preprocessed JSON if True
+    quiet:       (boolean) suppress warning messages if True
+
+Returns:
+
+    Dictionary. Raw or processed structured data.
+
+## Parser Information
+Compatibility:  linux
+
+Version 1.0 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/ufw_appinfo.md

@@ -0,0 +1,158 @@
+[Home](https://kellyjonbrazil.github.io/jc/)
+
+# jc.parsers.ufw_appinfo
+jc - JSON CLI output utility `ufw app info [application]` command output parser
+
+Supports individual apps via `ufw app info [application]` and all apps list via `ufw app info all`.
+
+Because `ufw` application definitions allow overlapping ports and port ranges, this parser preserves that behavior, but also provides `normalized` lists and ranges that remove duplicate ports and merge overlapping ranges.
+
+Usage (cli):
+
+    $ ufw app info OpenSSH | jc --ufw-appinfo
+
+    or
+
+    $ jc ufw app info OpenSSH
+
+Usage (module):
+
+    import jc.parsers.ufw_appinfo
+    result = jc.parsers.ufw_appinfo.parse(ufw_appinfo_command_output)
+
+Schema:
+
+    [
+      {
+        "profile":                  string,
+        "title":                    string,
+        "description":              string,
+        "tcp_list": [
+                                    integer
+        ],
+        "tcp_ranges": [
+          {
+            "start":                integer,      # 'any' is converted to start/end: 0/65535
+            "end":                  integer
+          }
+        ],
+        "udp_list": [
+                                    integer
+        ],
+        "udp_ranges": [
+          {
+            "start":                integer,      # 'any' is converted to start/end: 0/65535
+            "end":                  integer
+          }
+        ],
+        "normalized_tcp_list": [
+                                    integers      # duplicates and overlapping are removed
+        ],
+        "normalized_tcp_ranges": [
+          {
+            "start":                integer,      # 'any' is converted to start/end: 0/65535
+            "end":                  integers      # overlapping are merged
+          }
+        ],
+        "normalized_udp_list": [
+                                    integers      # duplicates and overlapping are removed
+        ],
+        "normalized_udp_ranges": [
+          {
+            "start":                integer,      # 'any' is converted to start/end: 0/65535
+            "end":                  integers      # overlapping are merged
+          }
+        ]
+      }
+    ]
+
+Examples:
+
+    $ ufw app info MSN | jc --ufw-appinfo -p
+    [
+      {
+        "profile": "MSN",
+        "title": "MSN Chat",
+        "description": "MSN chat protocol (with file transfer and voice)",
+        "tcp_list": [
+          1863,
+          6901
+        ],
+        "udp_list": [
+          1863,
+          6901
+        ],
+        "tcp_ranges": [
+          {
+            "start": 6891,
+            "end": 6900
+          }
+        ],
+        "normalized_tcp_list": [
+          1863,
+          6901
+        ],
+        "normalized_tcp_ranges": [
+          {
+            "start": 6891,
+            "end": 6900
+          }
+        ],
+        "normalized_udp_list": [
+          1863,
+          6901
+        ]
+      }
+    ]
+
+    $ ufw app info MSN | jc --ufw-appinfo -p -r
+    [
+      {
+        "profile": "MSN",
+        "title": "MSN Chat",
+        "description": "MSN chat protocol (with file transfer and voice)",
+        "tcp_list": [
+          "1863",
+          "6901"
+        ],
+        "udp_list": [
+          "1863",
+          "6901"
+        ],
+        "tcp_ranges": [
+          {
+            "start": "6891",
+            "end": "6900"
+          }
+        ]
+      }
+    ]
+
+
+## info
+```python
+info()
+```
+Provides parser metadata (version, author, etc.)
+
+## parse
+```python
+parse(data, raw=False, quiet=False)
+```
+
+Main text parsing function
+
+Parameters:
+
+    data:        (string)  text data to parse
+    raw:         (boolean) output preprocessed JSON if True
+    quiet:       (boolean) suppress warning messages if True
+
+Returns:
+
+    List of Dictionaries. Raw or processed structured data.
+
+## Parser Information
+Compatibility:  linux
+
+Version 1.0 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/upower.md

@@ -219,4 +219,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux
 
-Version 1.1 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.2 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/uptime.md

@@ -90,4 +90,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux, darwin, cygwin, aix, freebsd
 
-Version 1.4 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.5 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/vmstat.md

@@ -0,0 +1,149 @@
+[Home](https://kellyjonbrazil.github.io/jc/)
+
+# jc.parsers.vmstat
+jc - JSON CLI output utility `vmstat` command output parser
+
+Options supported: `-a`, `-w`, `-d`, `-t`
+
+The `epoch` calculated timestamp field is naive (i.e. based on the local time of the system the parser is run on)
+
+The `epoch_utc` calculated timestamp field is timezone-aware and is only available if the timezone field is UTC.
+
+Usage (cli):
+
+    $ vmstat | jc --vmstat
+
+    or
+
+    $ jc vmstat
+
+Usage (module):
+
+    import jc.parsers.vmstat
+    result = jc.parsers.vmstat.parse(vmstat_command_output)
+
+Schema:
+
+    [
+      {
+        "runnable_procs":                    integer,
+        "uninterruptible_sleeping_procs":    integer,
+        "virtual_mem_used":                  integer,
+        "free_mem":                          integer,
+        "buffer_mem":                        integer,
+        "cache_mem":                         integer,
+        "inactive_mem":                      integer,
+        "active_mem":                        integer,
+        "swap_in":                           integer,
+        "swap_out":                          integer,
+        "blocks_in":                         integer,
+        "blocks_out":                        integer,
+        "interrupts":                        integer,
+        "context_switches":                  integer,
+        "user_time":                         integer,
+        "system_time":                       integer,
+        "idle_time":                         integer,
+        "io_wait_time":                      integer,
+        "stolen_time":                       integer,
+        "disk":                              string,
+        "total_reads":                       integer,
+        "merged_reads":                      integer,
+        "sectors_read":                      integer,
+        "reading_ms":                        integer,
+        "total_writes":                      integer,
+        "merged_writes":                     integer,
+        "sectors_written":                   integer,
+        "writing_ms":                        integer,
+        "current_io":                        integer,
+        "io_seconds":                        integer,
+        "timestamp":                         string,
+        "timezone":                          string,
+        "epoch":                             integer,     # naive timestamp if -t flag is used
+        "epoch_utc":                         integer      # aware timestamp if -t flag is used and UTC TZ
+      }
+    ]
+
+Examples:
+
+    $ vmstat | jc --vmstat -p
+    [
+      {
+        "runnable_procs": 2,
+        "uninterruptible_sleeping_procs": 0,
+        "virtual_mem_used": 0,
+        "free_mem": 2794468,
+        "buffer_mem": 2108,
+        "cache_mem": 741208,
+        "inactive_mem": null,
+        "active_mem": null,
+        "swap_in": 0,
+        "swap_out": 0,
+        "blocks_in": 1,
+        "blocks_out": 3,
+        "interrupts": 29,
+        "context_switches": 57,
+        "user_time": 0,
+        "system_time": 0,
+        "idle_time": 99,
+        "io_wait_time": 0,
+        "stolen_time": 0,
+        "timestamp": null,
+        "timezone": null
+      }
+    ]
+
+    $ vmstat | jc --vmstat -p -r
+    [
+      {
+        "runnable_procs": "2",
+        "uninterruptible_sleeping_procs": "0",
+        "virtual_mem_used": "0",
+        "free_mem": "2794468",
+        "buffer_mem": "2108",
+        "cache_mem": "741208",
+        "inactive_mem": null,
+        "active_mem": null,
+        "swap_in": "0",
+        "swap_out": "0",
+        "blocks_in": "1",
+        "blocks_out": "3",
+        "interrupts": "29",
+        "context_switches": "57",
+        "user_time": "0",
+        "system_time": "0",
+        "idle_time": "99",
+        "io_wait_time": "0",
+        "stolen_time": "0",
+        "timestamp": null,
+        "timezone": null
+      }
+    ]
+
+
+## info
+```python
+info()
+```
+Provides parser metadata (version, author, etc.)
+
+## parse
+```python
+parse(data, raw=False, quiet=False)
+```
+
+Main text parsing function
+
+Parameters:
+
+    data:        (string)  text data to parse
+    raw:         (boolean) output preprocessed JSON if True
+    quiet:       (boolean) suppress warning messages if True
+
+Returns:
+
+    List of Dictionaries. Raw or processed structured data.
+
+## Parser Information
+Compatibility:  linux
+
+Version 1.0 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/vmstat_s.md

@@ -0,0 +1,110 @@
+[Home](https://kellyjonbrazil.github.io/jc/)
+
+# jc.parsers.vmstat_s
+jc - JSON CLI output utility `vmstat` command output streaming parser
+
+Options supported: `-a`, `-w`, `-d`, `-t`
+
+The `epoch` calculated timestamp field is naive (i.e. based on the local time of the system the parser is run on)
+
+The `epoch_utc` calculated timestamp field is timezone-aware and is only available if the timezone field is UTC.
+
+Usage (cli):
+
+    $ vmstat | jc --vmstat-s
+
+Usage (module):
+
+    import jc.parsers.vmstat_s
+    result = jc.parsers.vmstat_s.parse(vmstat_command_output.splitlines())    # result is an iterable object
+    for item in result:
+        # do something
+
+Schema:
+
+    {
+      "runnable_procs":                    integer,
+      "uninterruptible_sleeping_procs":    integer,
+      "virtual_mem_used":                  integer,
+      "free_mem":                          integer,
+      "buffer_mem":                        integer,
+      "cache_mem":                         integer,
+      "inactive_mem":                      integer,
+      "active_mem":                        integer,
+      "swap_in":                           integer,
+      "swap_out":                          integer,
+      "blocks_in":                         integer,
+      "blocks_out":                        integer,
+      "interrupts":                        integer,
+      "context_switches":                  integer,
+      "user_time":                         integer,
+      "system_time":                       integer,
+      "idle_time":                         integer,
+      "io_wait_time":                      integer,
+      "stolen_time":                       integer,
+      "disk":                              string,
+      "total_reads":                       integer,
+      "merged_reads":                      integer,
+      "sectors_read":                      integer,
+      "reading_ms":                        integer,
+      "total_writes":                      integer,
+      "merged_writes":                     integer,
+      "sectors_written":                   integer,
+      "writing_ms":                        integer,
+      "current_io":                        integer,
+      "io_seconds":                        integer,
+      "timestamp":                         string,
+      "timezone":                          string,
+      "epoch":                             integer,     # naive timestamp if -t flag is used
+      "epoch_utc":                         integer      # aware timestamp if -t flag is used and UTC TZ
+      "_jc_meta":                                       # This object only exists if using -qq or ignore_exceptions=True
+        {
+          "success":                       booean,      # true if successfully parsed, false if error
+          "error":                         string,      # exists if "success" is false
+          "line":                          string       # exists if "success" is false
+        }
+    }
+
+Examples:
+
+    $ vmstat | jc --vmstat-s
+    {"runnable_procs":2,"uninterruptible_sleeping_procs":0,"virtual_mem_used":0,"free_mem":2794468,"buffer_mem":2108,"cache_mem":741208,"inactive_mem":null,"active_mem":null,"swap_in":0,"swap_out":0,"blocks_in":1,"blocks_out":3,"interrupts":29,"context_switches":57,"user_time":0,"system_time":0,"idle_time":99,"io_wait_time":0,"stolen_time":0,"timestamp":null,"timezone":null}
+    ...
+
+    $ vmstat | jc --vmstat-s -r
+    {"runnable_procs":"2","uninterruptible_sleeping_procs":"0","virtual_mem_used":"0","free_mem":"2794468","buffer_mem":"2108","cache_mem":"741208","inactive_mem":null,"active_mem":null,"swap_in":"0","swap_out":"0","blocks_in":"1","blocks_out":"3","interrupts":"29","context_switches":"57","user_time":"0","system_time":"0","idle_time":"99","io_wait_time":"0","stolen_time":"0","timestamp":null,"timezone":null}
+    ...
+
+
+## info
+```python
+info()
+```
+Provides parser metadata (version, author, etc.)
+
+## parse
+```python
+parse(data, raw=False, quiet=False, ignore_exceptions=False)
+```
+
+Main text parsing generator function. Returns an iterator object.
+
+Parameters:
+
+    data:              (iterable)  line-based text data to parse (e.g. sys.stdin or str.splitlines())
+    raw:               (boolean)   output preprocessed JSON if True
+    quiet:             (boolean)   suppress warning messages if True
+    ignore_exceptions: (boolean)   ignore parsing exceptions if True
+
+Yields:
+
+    Dictionary. Raw or processed structured data.
+
+Returns:
+
+    Iterator object
+
+## Parser Information
+Compatibility:  linux
+
+Version 0.5 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/wc.md

@@ -79,4 +79,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux, darwin, cygwin, aix, freebsd
 
-Version 1.1 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.2 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/who.md

@@ -157,4 +157,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux, darwin, cygwin, aix, freebsd
 
-Version 1.3 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.4 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/xml.md

@@ -95,4 +95,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux, darwin, cygwin, win32, aix, freebsd
 
-Version 1.4 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.5 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/parsers/yaml.md

@@ -109,4 +109,4 @@ Returns:
 ## Parser Information
 Compatibility:  linux, darwin, cygwin, win32, aix, freebsd
 
-Version 1.4 by Kelly Brazil (kellyjonbrazil@gmail.com)
+Version 1.5 by Kelly Brazil (kellyjonbrazil@gmail.com)

docs/readme.md

@@ -10,62 +10,66 @@ For documentation on each parser, see the [documentation site](https://kellyjonb
 
 CLI Example:
 
-    $ ls -l /usr/bin | jc --ls -p
+    $ dig example.com | jc --dig -p
     [
       {
-        "filename": "apropos",
-        "link_to": "whatis",
-        "flags": "lrwxrwxrwx.",
-        "links": 1,
-        "owner": "root",
-        "group": "root",
-        "size": 6,
-        "date": "Aug 15 10:53"
-      },
-      {
-        "filename": "ar",
-        "flags": "-rwxr-xr-x.",
-        "links": 1,
-        "owner": "root",
-        "group": "root",
-        "size": 62744,
-        "date": "Aug 8 16:14"
-      },
-      {
-        "filename": "arch",
-        "flags": "-rwxr-xr-x.",
-        "links": 1,
-        "owner": "root",
-        "group": "root",
-        "size": 33080,
-        "date": "Aug 19 23:25"
-      },
-      ...
+        "id": 2951,
+        "opcode": "QUERY",
+        "status": "NOERROR",
+        "flags": [
+          "qr",
+          "rd",
+          "ra"
+        ],
+        "query_num": 1,
+        "answer_num": 1,
+        "authority_num": 0,
+        "additional_num": 1,
+        "opt_pseudosection": {
+          "edns": {
+            "version": 0,
+            "flags": [],
+            "udp": 4096
+          }
+        },
+        "question": {
+          "name": "example.com.",
+          "class": "IN",
+          "type": "A"
+        },
+        "answer": [
+          {
+            "name": "example.com.",
+            "class": "IN",
+            "type": "A",
+            "ttl": 39302,
+            "data": "93.184.216.34"
+          }
+        ],
+        "query_time": 49,
+        "server": "2600:1700:bab0:d40::1#53(2600:1700:bab0:d40::1)",
+        "when": "Fri Apr 16 16:05:10 PDT 2021",
+        "rcvd": 56,
+        "when_epoch": 1618614310,
+        "when_epoch_utc": null
+      }
     ]
 
 Module Example:
 
-    >>> import jc.parsers.ls
+    >>> import jc.parsers.dig
     >>>
-    >>> data='''-rwxr-xr-x  1 root  wheel    23648 May  3 22:26 cat
-    ... -rwxr-xr-x  1 root  wheel    30016 May  3 22:26 chmod
-    ... -rwxr-xr-x  1 root  wheel    29024 May  3 22:26 cp
-    ... -rwxr-xr-x  1 root  wheel   375824 May  3 22:26 csh
-    ... -rwxr-xr-x  1 root  wheel    28608 May  3 22:26 date
-    ... -rwxr-xr-x  1 root  wheel    32000 May  3 22:26 dd
-    ... -rwxr-xr-x  1 root  wheel    23392 May  3 22:26 df
-    ... -rwxr-xr-x  1 root  wheel    18128 May  3 22:26 echo'''
+    >>> import subprocess
+    >>> import jc.parsers.dig
     >>>
-    >>> jc.parsers.ls.parse(data)
-    [{'filename': 'cat', 'flags': '-rwxr-xr-x', 'links': 1, 'owner': 'root', 'group': 'wheel', 'size': 23648,
-    'date': 'May 3 22:26'}, {'filename': 'chmod', 'flags': '-rwxr-xr-x', 'links': 1, 'owner': 'root',
-    'group': 'wheel', 'size': 30016, 'date': 'May 3 22:26'}, {'filename': 'cp', 'flags': '-rwxr-xr-x',
-    'links': 1, 'owner': 'root', 'group': 'wheel', 'size': 29024, 'date': 'May 3 22:26'}, {'filename': 'csh',
-    'flags': '-rwxr-xr-x', 'links': 1, 'owner': 'root', 'group': 'wheel', 'size': 375824, 'date': 'May 3
-    22:26'}, {'filename': 'date', 'flags': '-rwxr-xr-x', 'links': 1, 'owner': 'root', 'group': 'wheel',
-    'size': 28608, 'date': 'May 3 22:26'}, {'filename': 'dd', 'flags': '-rwxr-xr-x', 'links': 1, 'owner':
-    'root', 'group': 'wheel', 'size': 32000, 'date': 'May 3 22:26'}, {'filename': 'df', 'flags':
-    '-rwxr-xr-x', 'links': 1, 'owner': 'root', 'group': 'wheel', 'size': 23392, 'date': 'May 3 22:26'},
-    {'filename': 'echo', 'flags': '-rwxr-xr-x', 'links': 1, 'owner': 'root', 'group': 'wheel', 'size': 18128,
-    'date': 'May 3 22:26'}]
+    >>> cmd_output = subprocess.check_output(['dig', 'example.com'], text=True)
+    >>> data = jc.parsers.dig.parse(cmd_output)
+    >>>
+    >>> data
+    [{'id': 64612, 'opcode': 'QUERY', 'status': 'NOERROR', 'flags': ['qr', 'rd', 'ra'], 'query_num': 1, 'answer_num':
+    1, 'authority_num': 0, 'additional_num': 1, 'opt_pseudosection': {'edns': {'version': 0, 'flags': [], 'udp':
+    4096}}, 'question': {'name': 'example.com.', 'class': 'IN', 'type': 'A'}, 'answer': [{'name': 'example.com.',
+    'class': 'IN', 'type': 'A', 'ttl': 29658, 'data': '93.184.216.34'}], 'query_time': 52, 'server':
+    '2600:1700:bab0:d40::1#53(2600:1700:bab0:d40::1)', 'when': 'Fri Apr 16 16:13:00 PDT 2021', 'rcvd': 56,
+    'when_epoch': 1618614780, 'when_epoch_utc': None}]
 

docs/utils.md

@@ -4,14 +4,16 @@ jc - JSON CLI output utility utils
 
 ## warning_message
 ```python
-warning_message(message)
+warning_message(message_lines)
 ```
 
-Prints a warning message for non-fatal issues
+Prints warning message for non-fatal issues. The first line is prepended with
+'jc:  Warning - ' and subsequent lines are indented. Wraps text as needed based
+on the terminal width.
 
 Parameters:
 
-    message:        (string) text of message
+    message:   (list) list of string lines
 
 Returns:
 
@@ -20,14 +22,16 @@ Returns:
 
 ## error_message
 ```python
-error_message(message)
+error_message(message_lines)
 ```
 
-Prints an error message for fatal issues
+Prints an error message for fatal issues. The first line is prepended with
+'jc:  Error - ' and subsequent lines are indented. Wraps text as needed based
+on the terminal width.
 
 Parameters:
 
-    message:        (string) text of message
+    message:   (list) list of string lines
 
 Returns:
 
@@ -69,6 +73,68 @@ Returns:
     Boolean      True if input string (data) contains non-whitespace characters, otherwise False
 
 
+## convert_to_int
+```python
+convert_to_int(value)
+```
+
+Converts string and float input to int. Strips all non-numeric characters from strings.
+
+Parameters:
+
+    value:          (string/integer/float) Input value
+
+Returns:
+
+    integer/None    Integer if successful conversion, otherwise None
+
+
+## convert_to_float
+```python
+convert_to_float(value)
+```
+
+Converts string and int input to float. Strips all non-numeric characters from strings.
+
+Parameters:
+
+    value:          (string) Input value
+
+Returns:
+
+    float/None      Float if successful conversion, otherwise None
+
+
+## convert_to_bool
+```python
+convert_to_bool(value)
+```
+
+Converts string, integer, or float input to boolean by checking for 'truthy' values
+
+Parameters:
+
+    value:          (string/integer/float) Input value
+
+Returns:
+
+    True/False      False unless a 'truthy' number or string is found ('y', 'yes', 'true', '1', 1, -1, etc.)
+
+
+## stream_success
+```python
+stream_success(output_line, ignore_exceptions)
+```
+Add `_jc_meta` object to output line if `ignore_exceptions=True`
+
+## stream_error
+```python
+stream_error(e, ignore_exceptions, line)
+```
+Reraise the stream exception with annotation or print an error `_jc_meta`
+field if `ignore_exceptions=True`
+
+
 ## timestamp
 ```python
 timestamp(datetime_string)
@@ -83,7 +149,7 @@ Parameters:
 Attributes:
 
     string              (str)   the input datetime string
-    format              (int)   the format rule that was used to decode the datetime string
+    format              (int)   the format rule that was used to decode the datetime string. None if conversion fails
     naive               (int)   timestamp based on locally configured timezone. None if conversion fails
     utc                 (int)   aware timestamp only if UTC timezone detected in datetime string. None if conversion fails
 

jc/__init__.py

@@ -8,65 +8,69 @@ For documentation on each parser, see the [documentation site](https://kellyjonb
 
 CLI Example:
 
-    $ ls -l /usr/bin | jc --ls -p
+    $ dig example.com | jc --dig -p
     [
       {
-        "filename": "apropos",
-        "link_to": "whatis",
-        "flags": "lrwxrwxrwx.",
-        "links": 1,
-        "owner": "root",
-        "group": "root",
-        "size": 6,
-        "date": "Aug 15 10:53"
-      },
-      {
-        "filename": "ar",
-        "flags": "-rwxr-xr-x.",
-        "links": 1,
-        "owner": "root",
-        "group": "root",
-        "size": 62744,
-        "date": "Aug 8 16:14"
-      },
-      {
-        "filename": "arch",
-        "flags": "-rwxr-xr-x.",
-        "links": 1,
-        "owner": "root",
-        "group": "root",
-        "size": 33080,
-        "date": "Aug 19 23:25"
-      },
-      ...
+        "id": 2951,
+        "opcode": "QUERY",
+        "status": "NOERROR",
+        "flags": [
+          "qr",
+          "rd",
+          "ra"
+        ],
+        "query_num": 1,
+        "answer_num": 1,
+        "authority_num": 0,
+        "additional_num": 1,
+        "opt_pseudosection": {
+          "edns": {
+            "version": 0,
+            "flags": [],
+            "udp": 4096
+          }
+        },
+        "question": {
+          "name": "example.com.",
+          "class": "IN",
+          "type": "A"
+        },
+        "answer": [
+          {
+            "name": "example.com.",
+            "class": "IN",
+            "type": "A",
+            "ttl": 39302,
+            "data": "93.184.216.34"
+          }
+        ],
+        "query_time": 49,
+        "server": "2600:1700:bab0:d40::1#53(2600:1700:bab0:d40::1)",
+        "when": "Fri Apr 16 16:05:10 PDT 2021",
+        "rcvd": 56,
+        "when_epoch": 1618614310,
+        "when_epoch_utc": null
+      }
     ]
 
 Module Example:
 
-    >>> import jc.parsers.ls
+    >>> import jc.parsers.dig
     >>>
-    >>> data='''-rwxr-xr-x  1 root  wheel    23648 May  3 22:26 cat
-    ... -rwxr-xr-x  1 root  wheel    30016 May  3 22:26 chmod
-    ... -rwxr-xr-x  1 root  wheel    29024 May  3 22:26 cp
-    ... -rwxr-xr-x  1 root  wheel   375824 May  3 22:26 csh
-    ... -rwxr-xr-x  1 root  wheel    28608 May  3 22:26 date
-    ... -rwxr-xr-x  1 root  wheel    32000 May  3 22:26 dd
-    ... -rwxr-xr-x  1 root  wheel    23392 May  3 22:26 df
-    ... -rwxr-xr-x  1 root  wheel    18128 May  3 22:26 echo'''
+    >>> import subprocess
+    >>> import jc.parsers.dig
     >>>
-    >>> jc.parsers.ls.parse(data)
-    [{'filename': 'cat', 'flags': '-rwxr-xr-x', 'links': 1, 'owner': 'root', 'group': 'wheel', 'size': 23648,
-    'date': 'May 3 22:26'}, {'filename': 'chmod', 'flags': '-rwxr-xr-x', 'links': 1, 'owner': 'root',
-    'group': 'wheel', 'size': 30016, 'date': 'May 3 22:26'}, {'filename': 'cp', 'flags': '-rwxr-xr-x',
-    'links': 1, 'owner': 'root', 'group': 'wheel', 'size': 29024, 'date': 'May 3 22:26'}, {'filename': 'csh',
-    'flags': '-rwxr-xr-x', 'links': 1, 'owner': 'root', 'group': 'wheel', 'size': 375824, 'date': 'May 3
-    22:26'}, {'filename': 'date', 'flags': '-rwxr-xr-x', 'links': 1, 'owner': 'root', 'group': 'wheel',
-    'size': 28608, 'date': 'May 3 22:26'}, {'filename': 'dd', 'flags': '-rwxr-xr-x', 'links': 1, 'owner':
-    'root', 'group': 'wheel', 'size': 32000, 'date': 'May 3 22:26'}, {'filename': 'df', 'flags':
-    '-rwxr-xr-x', 'links': 1, 'owner': 'root', 'group': 'wheel', 'size': 23392, 'date': 'May 3 22:26'},
-    {'filename': 'echo', 'flags': '-rwxr-xr-x', 'links': 1, 'owner': 'root', 'group': 'wheel', 'size': 18128,
-    'date': 'May 3 22:26'}]
+    >>> cmd_output = subprocess.check_output(['dig', 'example.com'], text=True)
+    >>> data = jc.parsers.dig.parse(cmd_output)
+    >>>
+    >>> data
+    [{'id': 64612, 'opcode': 'QUERY', 'status': 'NOERROR', 'flags': ['qr', 'rd', 'ra'], 'query_num': 1, 'answer_num':
+    1, 'authority_num': 0, 'additional_num': 1, 'opt_pseudosection': {'edns': {'version': 0, 'flags': [], 'udp':
+    4096}}, 'question': {'name': 'example.com.', 'class': 'IN', 'type': 'A'}, 'answer': [{'name': 'example.com.',
+    'class': 'IN', 'type': 'A', 'ttl': 29658, 'data': '93.184.216.34'}], 'query_time': 52, 'server':
+    '2600:1700:bab0:d40::1#53(2600:1700:bab0:d40::1)', 'when': 'Fri Apr 16 16:13:00 PDT 2021', 'rcvd': 56,
+    'when_epoch': 1618614780, 'when_epoch_utc': None}]
 """
 
 name = 'jc'
-__version__ = '1.15.1'
+__version__ = '1.17.0'

jc/cli.py

@@ -6,13 +6,18 @@ import sys
 import os
 import os.path
 import re
-import shlex
 import importlib
 import textwrap
 import signal
+import shlex
+import subprocess
 import json
 import jc
-import jc.appdirs as appdirs
+from jc import appdirs
+import jc.utils
+import jc.tracebackplus
+from jc.exceptions import LibraryNotInstalled, ParseError
+
 # make pygments import optional
 try:
     import pygments
@@ -21,9 +26,9 @@ try:
     from pygments.token import (Name, Number, String, Keyword)
     from pygments.lexers import JsonLexer
     from pygments.formatters import Terminal256Formatter
-    pygments_installed = True
+    PYGMENTS_INSTALLED = True
 except Exception:
-    pygments_installed = False
+    PYGMENTS_INSTALLED = False
 
 
 class info():
@@ -76,6 +81,7 @@ parsers = [
     'kv',
     'last',
     'ls',
+    'ls-s',
     'lsblk',
     'lsmod',
     'lsof',
@@ -84,11 +90,13 @@ parsers = [
     'ntpq',
     'passwd',
     'ping',
+    'ping-s',
     'pip-list',
     'pip-show',
     'ps',
     'route',
     'rpm-qi',
+    'sfdisk',
     'shadow',
     'ss',
     'stat',
@@ -97,13 +105,18 @@ parsers = [
     'systemctl-lj',
     'systemctl-ls',
     'systemctl-luf',
+    'systeminfo',
     'time',
     'timedatectl',
     'tracepath',
     'traceroute',
+    'ufw',
+    'ufw-appinfo',
     'uname',
     'upower',
     'uptime',
+    'vmstat',
+    'vmstat-s',
     'w',
     'wc',
     'who',
@@ -111,6 +124,9 @@ parsers = [
     'yaml'
 ]
 
+JC_ERROR_EXIT = 100
+
+
 # List of custom or override parsers.
 # Allow any <user_data_dir>/jc/jcparsers/*.py
 local_parsers = []
@@ -119,7 +135,7 @@ local_parsers_dir = os.path.join(data_dir, 'jcparsers')
 if os.path.isdir(local_parsers_dir):
     sys.path.append(data_dir)
     for name in os.listdir(local_parsers_dir):
-        if re.match(r'\w+\.py', name) and os.path.isfile(os.path.join(local_parsers_dir, name)):
+        if re.match(r'\w+\.py$', name) and os.path.isfile(os.path.join(local_parsers_dir, name)):
             plugin_name = name[0:-3]
             local_parsers.append(plugin_name)
             if plugin_name not in parsers:
@@ -128,7 +144,7 @@ if os.path.isdir(local_parsers_dir):
 
 # We only support 2.3.0+, pygments changed color names in 2.4.0.
 # startswith is sufficient and avoids potential exceptions from split and int.
-if pygments_installed:
+if PYGMENTS_INSTALLED:
     if pygments.__version__.startswith('2.3.'):
         PYGMENT_COLOR = {
             'black': '#ansiblack',
@@ -183,10 +199,9 @@ def set_env_colors(env_colors=None):
 
     Default colors:
 
-    JC_COLORS=blue,brightblack,magenta,green
+        JC_COLORS=blue,brightblack,magenta,green
     or
-    JC_COLORS=default,default,default,default
-
+        JC_COLORS=default,default,default,default
     """
     input_error = False
 
@@ -204,7 +219,7 @@ def set_env_colors(env_colors=None):
 
     # if there is an issue with the env variable, just set all colors to default and move on
     if input_error:
-        jc.utils.warning_message('could not parse JC_COLORS environment variable')
+        jc.utils.warning_message(['Could not parse JC_COLORS environment variable'])
         color_list = ['default', 'default', 'default', 'default']
 
     # Try the color set in the JC_COLORS env variable first. If it is set to default, then fall back to default colors
@@ -218,20 +233,17 @@ def set_env_colors(env_colors=None):
 
 def piped_output():
     """Return False if stdout is a TTY. True if output is being piped to another program"""
-    if sys.stdout.isatty():
-        return False
-    else:
-        return True
+    return not sys.stdout.isatty()
 
 
 def ctrlc(signum, frame):
     """Exit with error on SIGINT"""
-    sys.exit(1)
+    sys.exit(JC_ERROR_EXIT)
 
 
-def parser_shortname(parser_argument):
+def parser_shortname(parser_arg):
     """Return short name of the parser with dashes and no -- prefix"""
-    return parser_argument[2:]
+    return parser_arg[2:]
 
 
 def parser_argument(parser):
@@ -318,28 +330,29 @@ def helptext():
     Parsers:
 {parsers_string}
     Options:
-            -a               about jc
-            -d               debug - show traceback (-dd for verbose traceback)
-            -h               help (use -h --parser_name for parser documentation)
-            -m               monochrome output
-            -p               pretty print output
-            -q               quiet - suppress parser warnings
-            -r               raw JSON output
-            -v               version info
+            -a    about jc
+            -d    debug (-dd for verbose debug)
+            -h    help (-h --parser_name for parser documentation)
+            -m    monochrome output
+            -p    pretty print output
+            -q    quiet - suppress parser warnings (-qq to ignore streaming errors)
+            -r    raw JSON output
+            -u    unbuffer output
+            -v    version info
 
-    Example:
-            ls -al | jc --ls -p
+    Examples:
+            Standard Syntax:
+                $ dig www.google.com | jc --dig -p
 
-            or using the magic syntax:
+            Magic Syntax:
+                $ jc -p dig www.google.com
 
-            jc -p ls -al
-
-            For parser documentation:
-
-            jc -h --ls
+            Parser Documentation:
+                $ jc -h --dig
     '''
     return textwrap.dedent(helptext_string)
 
+
 def help_doc(options):
     """
     Returns the parser documentation if a parser is found in the arguments, otherwise
@@ -352,16 +365,16 @@ def help_doc(options):
             # load parser module just in time so we don't need to load all modules
             parser = parser_module(arg)
             compatible = ', '.join(parser.info.compatible)
-            doc_text = f'''{parser.__doc__}
-Compatibility:  {compatible}
-
-Version {parser.info.version} by {parser.info.author} ({parser.info.author_email})
-'''
+            doc_text = \
+                f'{parser.__doc__}\n'\
+                f'Compatibility:  {compatible}\n\n'\
+                f'Version {parser.info.version} by {parser.info.author} ({parser.info.author_email})\n'
 
             return doc_text
 
     return helptext()
 
+
 def versiontext():
     """Return the version text"""
     versiontext_string = f'''\
@@ -373,46 +386,49 @@ def versiontext():
 
 def json_out(data, pretty=False, env_colors=None, mono=False, piped_out=False):
     """Return a JSON formatted string. String may include color codes or be pretty printed."""
+    separators = (',', ':')
+    indent = None
+
+    if pretty:
+        separators = None
+        indent = 2
+
     if not mono and not piped_out:
         # set colors
         class JcStyle(Style):
             styles = set_env_colors(env_colors)
 
-        if pretty:
-            return str(highlight(json.dumps(data, indent=2, ensure_ascii=False),
-                                 JsonLexer(), Terminal256Formatter(style=JcStyle))[0:-1])
-        else:
-            return str(highlight(json.dumps(data, separators=(',', ':'), ensure_ascii=False),
-                                 JsonLexer(), Terminal256Formatter(style=JcStyle))[0:-1])
-    else:
-        if pretty:
-            return json.dumps(data, indent=2, ensure_ascii=False)
-        else:
-            return json.dumps(data, separators=(',', ':'), ensure_ascii=False)
+        return str(highlight(json.dumps(data, indent=indent, separators=separators, ensure_ascii=False),
+                             JsonLexer(), Terminal256Formatter(style=JcStyle))[0:-1])
+
+    return json.dumps(data, indent=indent, separators=separators, ensure_ascii=False)
 
 
-def generate_magic_command(args):
-    """
-    Return a tuple with a boolean and a command, where the boolean signifies that
-    the command is valid, and the command is either a command string or None.
+def magic_parser(args):
     """
+    Parse command arguments for magic syntax: jc -p ls -al
 
-    # Parse with magic syntax: jc -p ls -al
+    Return a tuple:
+        valid_command   (bool)  is this a valid command? (exists in magic dict)
+        run_command     (list)  list of the user's command to run. None if no command.
+        jc_parser       (str)   parser to use for this user's command.
+        jc_options      (list)  list of jc options
+    """
+    # bail immediately if there are no args or a parser is defined
     if len(args) <= 1 or args[1].startswith('--'):
-        return False, None
+        return False, None, None, []
 
-    # correctly parse escape characters and spaces with shlex
-    args_given = ' '.join(map(shlex.quote, args[1:])).split()
+    args_given = args[1:]
     options = []
 
     # find the options
     for arg in list(args_given):
         # parser found - use standard syntax
         if arg.startswith('--'):
-            return False, None
+            return False, None, None, []
 
         # option found - populate option list
-        elif arg.startswith('-'):
+        if arg.startswith('-'):
             options.extend(args_given.pop(0)[1:])
 
         # command found if iterator didn't already stop - stop iterating
@@ -421,11 +437,11 @@ def generate_magic_command(args):
 
     # if -h, -a, or -v found in options, then bail out
     if 'h' in options or 'a' in options or 'v' in options:
-        return False, None
+        return False, None, None, []
 
-    # all options popped and no command found - for case like 'jc -a'
+    # all options popped and no command found - for case like 'jc -x'
     if len(args_given) == 0:
-        return False, None
+        return False, None, None, []
 
     magic_dict = {}
     parser_info = about_jc()['parsers']
@@ -442,31 +458,37 @@ def generate_magic_command(args):
     # try to get a parser for two_word_command, otherwise get one for one_word_command
     found_parser = magic_dict.get(two_word_command, magic_dict.get(one_word_command))
 
-    # construct a new command line using the standard syntax: COMMAND | jc --PARSER -OPTIONS
-    run_command = ' '.join(args_given)
-    if found_parser:
-        cmd_options = ('-' + ''.join(options)) if options else ''
-        return True, ' '.join([run_command, '|', 'jc', found_parser, cmd_options])
-    else:
-        return False, run_command
+    return (
+        bool(found_parser),                 # was a suitable parser found?
+        args_given,                         # run_command
+        found_parser,                       # the parser selected
+        options                             # jc options to preserve
+    )
 
 
-def magic():
-    """Runs the command generated by generate_magic_command() to support magic syntax"""
-    valid_command, run_command = generate_magic_command(sys.argv)
-    if valid_command:
-        os.system(run_command)
-        sys.exit(0)
-    elif run_command is None:
-        return
-    else:
-        jc.utils.error_message(f'parser not found for "{run_command}". Use "jc -h" for help.')
-        sys.exit(1)
+def run_user_command(command):
+    """Use subprocess to run the user's command. Returns the STDOUT, STDERR, and the Exit Code as a tuple."""
+    proc = subprocess.Popen(command,
+                            stdout=subprocess.PIPE,
+                            stderr=subprocess.PIPE,
+                            close_fds=False,            # Allows inheriting file descriptors. Useful for process substitution
+                            universal_newlines=True)
+    stdout, stderr = proc.communicate()
+
+    return (
+        stdout or '\n',
+        stderr,
+        proc.returncode
+    )
+
+
+def combined_exit_code(program_exit=0, jc_exit=0):
+    exit_code = program_exit + jc_exit
+    exit_code = min(exit_code, 255)
+    return exit_code
 
 
 def main():
-    import jc.utils
-
     # break on ctrl-c keyboard interrupt
     signal.signal(signal.SIGINT, ctrlc)
 
@@ -476,29 +498,43 @@ def main():
     except AttributeError:
         pass
 
-    # try magic syntax first: e.g. jc -p ls -al
-    magic()
+    # enable colors for Windows cmd.exe terminal
+    if sys.platform.startswith('win32'):
+        os.system('')
 
+    # parse magic syntax first: e.g. jc -p ls -al
+    magic_options = []
+    valid_command, run_command, magic_found_parser, magic_options = magic_parser(sys.argv)
+
+    # set colors
     jc_colors = os.getenv('JC_COLORS')
 
+    # set options
     options = []
+    options.extend(magic_options)
 
-    # options
-    for opt in sys.argv:
-        if opt.startswith('-') and not opt.startswith('--'):
-            options.extend(opt[1:])
+    # find options if magic_parser did not find a command
+    if not valid_command:
+        for opt in sys.argv:
+            if opt.startswith('-') and not opt.startswith('--'):
+                options.extend(opt[1:])
 
     about = 'a' in options
     debug = 'd' in options
-    verbose_debug = True if options.count('d') > 1 else False
+    verbose_debug = options.count('d') > 1
     mono = 'm' in options
     help_me = 'h' in options
     pretty = 'p' in options
     quiet = 'q' in options
+    ignore_exceptions = options.count('q') > 1
     raw = 'r' in options
+    unbuffer = 'u' in options
     version_info = 'v' in options
 
-    if not pygments_installed:
+    if verbose_debug:
+        jc.tracebackplus.enable(context=11)
+
+    if not PYGMENTS_INSTALLED:
         mono = True
 
     if about:
@@ -513,44 +549,134 @@ def main():
         print(versiontext())
         sys.exit(0)
 
-    if verbose_debug:
-        import jc.tracebackplus
-        jc.tracebackplus.enable(context=11)
+    # if magic syntax used, try to run the command and error if it's not found, etc.
+    magic_stdout, magic_stderr, magic_exit_code = None, None, 0
+    if run_command:
+        try:
+            run_command_str = shlex.join(run_command)      # python 3.8+
+        except AttributeError:
+            run_command_str = ' '.join(run_command)        # older python versions
 
-    if sys.stdin.isatty():
-        jc.utils.error_message('Missing piped data. Use "jc -h" for help.')
-        sys.exit(1)
+    if valid_command:
+        try:
+            magic_stdout, magic_stderr, magic_exit_code = run_user_command(run_command)
+            if magic_stderr:
+                print(magic_stderr[:-1], file=sys.stderr)
 
-    data = sys.stdin.read()
+        except FileNotFoundError:
+            if debug:
+                raise
 
-    found = False
+            jc.utils.error_message([f'"{run_command_str}" command could not be found. For details use the -d or -dd option.'])
+            sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
 
-    for arg in sys.argv:
-        parser_name = parser_shortname(arg)
+        except OSError:
+            if debug:
+                raise
 
-        if parser_name in parsers:
-            # load parser module just in time so we don't need to load all modules
-            parser = parser_module(arg)
-            try:
-                result = parser.parse(data, raw=raw, quiet=quiet)
+            jc.utils.error_message([f'"{run_command_str}" command could not be run due to too many open files. For details use the -d or -dd option.'])
+            sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
+
+        except Exception:
+            if debug:
+                raise
+
+            jc.utils.error_message([f'"{run_command_str}" command could not be run. For details use the -d or -dd option.'])
+            sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
+
+    elif run_command is not None:
+        jc.utils.error_message([f'"{run_command_str}" cannot be used with Magic syntax. Use "jc -h" for help.'])
+        sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
+
+    # find the correct parser
+    if magic_found_parser:
+        parser = parser_module(magic_found_parser)
+        parser_name = parser_shortname(magic_found_parser)
+
+    else:
+        found = False
+        for arg in sys.argv:
+            parser_name = parser_shortname(arg)
+
+            if parser_name in parsers:
+                parser = parser_module(arg)
                 found = True
                 break
 
-            except Exception:
-                if debug:
-                    raise
-                else:
-                    import jc.utils
-                    jc.utils.error_message(
-                        f'{parser_name} parser could not parse the input data. Did you use the correct parser?\n'
-                        '             For details use the -d or -dd option. Use "jc -h" for help.')
-                    sys.exit(1)
+        if not found:
+            jc.utils.error_message(['Missing or incorrect arguments. Use "jc -h" for help.'])
+            sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
 
-    if not found:
-        jc.utils.error_message('Missing or incorrect arguments. Use "jc -h" for help.')
-        sys.exit(1)
+    # check for input errors (pipe vs magic)
+    if not sys.stdin.isatty() and magic_stdout:
+        jc.utils.error_message(['Piped data and Magic syntax used simultaneously. Use "jc -h" for help.'])
+        sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
 
-    print(json_out(result, pretty=pretty, env_colors=jc_colors, mono=mono, piped_out=piped_output()))
+    elif sys.stdin.isatty() and magic_stdout is None:
+        jc.utils.error_message(['Missing piped data. Use "jc -h" for help.'])
+        sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
+
+    # parse and print to stdout
+    try:
+        # differentiate between regular and streaming parsers
+
+        # streaming
+        if getattr(parser.info, 'streaming', None):
+            result = parser.parse(sys.stdin, raw=raw, quiet=quiet, ignore_exceptions=ignore_exceptions)
+            for line in result:
+                print(json_out(line,
+                               pretty=pretty,
+                               env_colors=jc_colors,
+                               mono=mono,
+                               piped_out=piped_output()),
+                      flush=unbuffer)
+
+            sys.exit(combined_exit_code(magic_exit_code, 0))
+
+        # regular
+        else:
+            data = magic_stdout or sys.stdin.read()
+            result = parser.parse(data, raw=raw, quiet=quiet)
+            print(json_out(result,
+                           pretty=pretty,
+                           env_colors=jc_colors,
+                           mono=mono,
+                           piped_out=piped_output()),
+                  flush=unbuffer)
+
+        sys.exit(combined_exit_code(magic_exit_code, 0))
+
+    except (ParseError, LibraryNotInstalled) as e:
+        if debug:
+            raise
+
+        jc.utils.error_message([f'Parser issue with {parser_name}:',
+                                f'{e.__class__.__name__}: {e}',
+                                'For details use the -d or -dd option. Use "jc -h" for help.'])
+        sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
+
+    except json.JSONDecodeError:
+        if debug:
+            raise
+
+        jc.utils.error_message(['There was an issue generating the JSON output.',
+                                'For details use the -d or -dd option.'])
+        sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
+
+    except Exception:
+        if debug:
+            raise
+
+        streaming_msg = ''
+        if getattr(parser.info, 'streaming', None):
+            streaming_msg = 'Use the -qq option to ignore streaming parser errors.'
+
+        jc.utils.error_message([
+            f'{parser_name} parser could not parse the input data. Did you use the correct parser?',
+            f'{streaming_msg}',
+            'For details use the -d or -dd option. Use "jc -h" for help.'
+        ])
+        sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
 
 
 if __name__ == '__main__':

jc/exceptions.py

@@ -0,0 +1,9 @@
+"""jc - JSON CLI output utility exceptions"""
+
+
+class ParseError(Exception):
+    pass
+
+
+class LibraryNotInstalled(Exception):
+    pass

jc/parsers/acpi.py

@@ -227,7 +227,7 @@ import jc.utils
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.1'
+    version = '1.2'
     description = '`acpi` command parser'
     author = 'Kelly Brazil'
     author_email = 'kellyjonbrazil@gmail.com'
@@ -256,38 +256,19 @@ def _process(proc_data):
     float_list = ['temperature']
 
     for entry in proc_data:
-        for key in int_list:
-            if key in entry:
-                try:
-                    entry[key] = int(entry[key])
-                except (ValueError):
-                    entry[key] = None
+        for key in entry:
+            if key in int_list:
+                entry[key] = jc.utils.convert_to_int(entry[key])
+            if key in float_list:
+                entry[key] = jc.utils.convert_to_float(entry[key])
 
         if 'trip_points' in entry:
             for tp in entry['trip_points']:
-                for key in int_list:
-                    if key in tp:
-                        try:
-                            tp[key] = int(tp[key])
-                        except (ValueError):
-                            tp[key] = None
-
-    for entry in proc_data:
-        for key in float_list:
-            if key in entry:
-                try:
-                    entry[key] = float(entry[key])
-                except (ValueError):
-                    entry[key] = None
-
-        if 'trip_points' in entry:
-            for tp in entry['trip_points']:
-                for key in float_list:
-                    if key in tp:
-                        try:
-                            tp[key] = float(tp[key])
-                        except (ValueError):
-                            tp[key] = None
+                for key in tp:
+                    if key in int_list:
+                        tp[key] = jc.utils.convert_to_int(tp[key])
+                    if key in float_list:
+                        tp[key] = jc.utils.convert_to_float(tp[key])
 
     for entry in proc_data:
         if 'until_charged' in entry:

jc/parsers/airport.py

@@ -80,7 +80,7 @@ import jc.utils
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.2'
+    version = '1.3'
     description = '`airport -I` command parser'
     author = 'Kelly Brazil'
     author_email = 'kellyjonbrazil@gmail.com'
@@ -111,10 +111,7 @@ def _process(proc_data):
                 'lasttxrate', 'maxrate', 'lastassocstatus', 'mcs']
     for key in proc_data:
         if key in int_list:
-            try:
-                proc_data[key] = int(proc_data[key])
-            except (ValueError):
-                proc_data[key] = None
+            proc_data[key] = jc.utils.convert_to_int(proc_data[key])
 
     return proc_data
 

jc/parsers/airport_s.py

@@ -109,7 +109,7 @@ import jc.parsers.universal
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.3'
+    version = '1.4'
     description = '`airport -s` command parser'
     author = 'Kelly Brazil'
     author_email = 'kellyjonbrazil@gmail.com'
@@ -137,23 +137,14 @@ def _process(proc_data):
     """
     for entry in proc_data:
 
-        # integers
+        # convert integers and booleans
         int_list = ['rssi']
-        for key in int_list:
-            if key in entry:
-                try:
-                    entry[key] = int(entry[key])
-                except (ValueError):
-                    entry[key] = None
-
-        # booleans
         bool_list = ['ht']
         for key in entry:
+            if key in int_list:
+                entry[key] = jc.utils.convert_to_int(entry[key])
             if key in bool_list:
-                try:
-                    entry[key] = True if entry[key] == 'Y' else False
-                except (ValueError):
-                    entry[key] = None
+                entry[key] = jc.utils.convert_to_bool(entry[key])
 
         if 'security' in entry:
             entry['security'] = entry['security'].split()

jc/parsers/arp.py

@@ -150,12 +150,9 @@ def _process(proc_data):
             entry['name'] = None
 
         int_list = ['expires']
-        for key in int_list:
-            if key in entry:
-                try:
-                    entry[key] = int(entry[key])
-                except (ValueError):
-                    entry[key] = None
+        for key in entry:
+            if key in int_list:
+                entry[key] = jc.utils.convert_to_int(entry[key])
 
     return proc_data
 

jc/parsers/blkid.py

@@ -121,7 +121,7 @@ import jc.utils
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.3'
+    version = '1.4'
     description = '`blkid` command parser'
     author = 'Kelly Brazil'
     author_email = 'kellyjonbrazil@gmail.com'
@@ -155,12 +155,9 @@ def _process(proc_data):
                     'id_part_entry_offset', 'id_part_entry_size', 'minimum_io_size', 'physical_sector_size',
                     'logical_sector_size', 'id_iolimit_minimum_io_size', 'id_iolimit_physical_sector_size',
                     'id_iolimit_logical_sector_size']
-        for key in int_list:
-            if key in entry:
-                try:
-                    entry[key] = int(entry[key])
-                except (ValueError):
-                    entry[key] = None
+        for key in entry:
+            if key in int_list:
+                entry[key] = jc.utils.convert_to_int(entry[key])
 
     return proc_data
 

jc/parsers/cksum.py

@@ -54,7 +54,7 @@ import jc.utils
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.1'
+    version = '1.2'
     description = '`cksum` and `sum` command parser'
     author = 'Kelly Brazil'
     author_email = 'kellyjonbrazil@gmail.com'
@@ -82,12 +82,10 @@ def _process(proc_data):
 
     for entry in proc_data:
         int_list = ['checksum', 'blocks']
-        for key in int_list:
-            if key in entry:
-                try:
-                    entry[key] = int(entry[key])
-                except (ValueError):
-                    entry[key] = None
+        for key in entry:
+            if key in int_list:
+                entry[key] = jc.utils.convert_to_int(entry[key])
+
     return proc_data
 
 

jc/parsers/df.py

@@ -98,7 +98,7 @@ import jc.parsers.universal
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.6'
+    version = '1.7'
     description = '`df` command parser'
     author = 'Kelly Brazil'
     author_email = 'kellyjonbrazil@gmail.com'
@@ -144,11 +144,7 @@ def _process(proc_data):
         # change any entry for key with '_blocks' in the name to int
         for k in entry:
             if '_blocks' in str(k):
-                try:
-                    blocks_int = int(entry[k])
-                    entry[k] = blocks_int
-                except (ValueError):
-                    entry[k] = None
+                entry[k] = jc.utils.convert_to_int(entry[k])
 
         # remove percent sign from 'use_percent', 'capacity_percent', and 'iused_percent'
         if 'use_percent' in entry:
@@ -162,13 +158,9 @@ def _process(proc_data):
 
         # change used, available, use_percent, capacity_percent, ifree, iused, iused_percent to int
         int_list = ['used', 'available', 'use_percent', 'capacity_percent', 'ifree', 'iused', 'iused_percent']
-        for key in int_list:
-            if key in entry:
-                try:
-                    key_int = int(entry[key])
-                    entry[key] = key_int
-                except (ValueError):
-                    entry[key] = None
+        for key in entry:
+            if key in int_list:
+                entry[key] = jc.utils.convert_to_int(entry[key])
 
     return proc_data
 

jc/parsers/dig.py

@@ -1,5 +1,9 @@
 """jc - JSON CLI output utility `dig` command output parser
 
+Options supported:
+- `+noall +answer` options are supported in cases where only the answer information is desired.
+- `+axfr` option is supported on its own
+
 The `when_epoch` calculated timestamp field is naive (i.e. based on the local time of the system the parser is run on)
 
 The `when_epoch_utc` calculated timestamp field is timezone-aware and is only available if the timezone field is UTC.
@@ -21,64 +25,84 @@ Schema:
 
     [
       {
-        "id":             integer,
-        "opcode":         string,
-        "status":         string,
+        "id":                   integer,
+        "opcode":               string,
+        "status":               string,
         "flags": [
-                          string
+                                string
         ],
-        "query_num":      integer,
-        "answer_num":     integer,
-        "authority_num":  integer,
-        "additional_num": integer,
+        "query_num":            integer,
+        "answer_num":           integer,
+        "authority_num":        integer,
+        "additional_num":       integer,
         "axfr": [
           {
-            "name":       string,
-            "class":      string,
-            "type":       string,
-            "ttl":        integer,
-            "data":       string
+            "name":             string,
+            "class":            string,
+            "type":             string,
+            "ttl":              integer,
+            "data":             string
           }
         ],
+        "opt_pseudosection": {
+          "edns": {
+            "version":          integer,
+            "flags": [
+                                string
+            ],
+            "udp":              integer
+          },
+          "cookie":             string
+        },
         "question": {
-          "name":         string,
-          "class":        string,
-          "type":         string
+          "name":               string,
+          "class":              string,
+          "type":               string
         },
         "answer": [
           {
-            "name":       string,
-            "class":      string,
-            "type":       string,
-            "ttl":        integer,
-            "data":       string
+            "name":             string,
+            "class":            string,
+            "type":             string,
+            "ttl":              integer,
+            "data":             string
+          }
+        ],
+        "additional": [
+          {
+            "name":             string,
+            "class":            string,
+            "type":             string,
+            "ttl":              integer,
+            "data":             string
           }
         ],
         "authority": [
           {
-            "name":       string,
-            "class":      string,
-            "type":       string,
-            "ttl":        integer,
-            "data":       string
+            "name":             string,
+            "class":            string,
+            "type":             string,
+            "ttl":              integer,
+            "data":             string
           }
         ],
-        "query_time":     integer,   # in msec
-        "server":         string,
-        "when":           string,
-        "when_epoch":     integer,   # naive timestamp if when field is parsable, else null
-        "when_epoch_utc": integer,   # timezone aware timestamp availabe for UTC, else null
-        "rcvd":           integer
-        "size":           string
+        "query_size":           integer,
+        "query_time":           integer,   # in msec
+        "server":               string,
+        "when":                 string,
+        "when_epoch":           integer,   # naive timestamp if when field is parsable, else null
+        "when_epoch_utc":       integer,   # timezone aware timestamp availabe for UTC, else null
+        "rcvd":                 integer
+        "size":                 string
       }
     ]
 
 Examples:
 
-    $ dig cnn.com www.cnn.com @205.251.194.64 | jc --dig -p
+    $ dig example.com | jc --dig -p
     [
       {
-        "id": 52172,
+        "id": 2951,
         "opcode": "QUERY",
         "status": "NOERROR",
         "flags": [
@@ -87,113 +111,35 @@ Examples:
           "ra"
         ],
         "query_num": 1,
-        "answer_num": 4,
+        "answer_num": 1,
         "authority_num": 0,
         "additional_num": 1,
+        "opt_pseudosection": {
+          "edns": {
+            "version": 0,
+            "flags": [],
+            "udp": 4096
+          }
+        },
         "question": {
-          "name": "cnn.com.",
+          "name": "example.com.",
           "class": "IN",
           "type": "A"
         },
         "answer": [
           {
-            "name": "cnn.com.",
+            "name": "example.com.",
             "class": "IN",
             "type": "A",
-            "ttl": 27,
-            "data": "151.101.65.67"
-          },
-          {
-            "name": "cnn.com.",
-            "class": "IN",
-            "type": "A",
-            "ttl": 27,
-            "data": "151.101.129.67"
-          },
-          {
-            "name": "cnn.com.",
-            "class": "IN",
-            "type": "A",
-            "ttl": 27,
-            "data": "151.101.1.67"
-          },
-          {
-            "name": "cnn.com.",
-            "class": "IN",
-            "type": "A",
-            "ttl": 27,
-            "data": "151.101.193.67"
+            "ttl": 39302,
+            "data": "93.184.216.34"
           }
         ],
-        "query_time": 38,
-        "server": "2600",
-        "when": "Tue Mar 30 20:07:59 PDT 2021",
-        "rcvd": 100,
-        "when_epoch": 1617160079,
-        "when_epoch_utc": null
-      },
-      {
-        "id": 36292,
-        "opcode": "QUERY",
-        "status": "NOERROR",
-        "flags": [
-          "qr",
-          "aa",
-          "rd"
-        ],
-        "query_num": 1,
-        "answer_num": 1,
-        "authority_num": 4,
-        "additional_num": 1,
-        "question": {
-          "name": "www.cnn.com.",
-          "class": "IN",
-          "type": "A"
-        },
-        "answer": [
-          {
-            "name": "www.cnn.com.",
-            "class": "IN",
-            "type": "CNAME",
-            "ttl": 300,
-            "data": "turner-tls.map.fastly.net."
-          }
-        ],
-        "authority": [
-          {
-            "name": "cnn.com.",
-            "class": "IN",
-            "type": "NS",
-            "ttl": 3600,
-            "data": "ns-1086.awsdns-07.org."
-          },
-          {
-            "name": "cnn.com.",
-            "class": "IN",
-            "type": "NS",
-            "ttl": 3600,
-            "data": "ns-1630.awsdns-11.co.uk."
-          },
-          {
-            "name": "cnn.com.",
-            "class": "IN",
-            "type": "NS",
-            "ttl": 3600,
-            "data": "ns-47.awsdns-05.com."
-          },
-          {
-            "name": "cnn.com.",
-            "class": "IN",
-            "type": "NS",
-            "ttl": 3600,
-            "data": "ns-576.awsdns-08.net."
-          }
-        ],
-        "query_time": 27,
-        "server": "205.251.194.64#53(205.251.194.64)",
-        "when": "Tue Mar 30 20:07:59 PDT 2021",
-        "rcvd": 212,
-        "when_epoch": 1617160079,
+        "query_time": 49,
+        "server": "2600:1700:bab0:d40::1#53(2600:1700:bab0:d40::1)",
+        "when": "Fri Apr 16 16:05:10 PDT 2021",
+        "rcvd": 56,
+        "when_epoch": 1618614310,
         "when_epoch_utc": null
       }
     ]
@@ -201,7 +147,7 @@ Examples:
     $ dig cnn.com www.cnn.com @205.251.194.64 | jc --dig -p -r
     [
       {
-        "id": "23843",
+        "id": "46052",
         "opcode": "QUERY",
         "status": "NOERROR",
         "flags": [
@@ -210,117 +156,41 @@ Examples:
           "ra"
         ],
         "query_num": "1",
-        "answer_num": "4",
+        "answer_num": "1",
         "authority_num": "0",
         "additional_num": "1",
+        "opt_pseudosection": {
+          "edns": {
+            "version": "0",
+            "flags": [],
+            "udp": "4096"
+          }
+        },
         "question": {
-          "name": "cnn.com.",
+          "name": "example.com.",
           "class": "IN",
           "type": "A"
         },
         "answer": [
           {
-            "name": "cnn.com.",
+            "name": "example.com.",
             "class": "IN",
             "type": "A",
-            "ttl": "30",
-            "data": "151.101.193.67"
-          },
-          {
-            "name": "cnn.com.",
-            "class": "IN",
-            "type": "A",
-            "ttl": "30",
-            "data": "151.101.1.67"
-          },
-          {
-            "name": "cnn.com.",
-            "class": "IN",
-            "type": "A",
-            "ttl": "30",
-            "data": "151.101.65.67"
-          },
-          {
-            "name": "cnn.com.",
-            "class": "IN",
-            "type": "A",
-            "ttl": "30",
-            "data": "151.101.129.67"
+            "ttl": "40426",
+            "data": "93.184.216.34"
           }
         ],
-        "query_time": "24 msec",
-        "server": "192.168.1.254#53(192.168.1.254)",
-        "when": "Tue Nov 12 07:16:19 PST 2019",
-        "rcvd": "100"
-      },
-      {
-        "id": "8266",
-        "opcode": "QUERY",
-        "status": "NOERROR",
-        "flags": [
-          "qr",
-          "aa",
-          "rd"
-        ],
-        "query_num": "1",
-        "answer_num": "1",
-        "authority_num": "4",
-        "additional_num": "1",
-        "question": {
-          "name": "www.cnn.com.",
-          "class": "IN",
-          "type": "A"
-        },
-        "answer": [
-          {
-            "name": "www.cnn.com.",
-            "class": "IN",
-            "type": "CNAME",
-            "ttl": "300",
-            "data": "turner-tls.map.fastly.net."
-          }
-        ],
-        "authority": [
-          {
-            "name": "cnn.com.",
-            "class": "IN",
-            "type": "NS",
-            "ttl": "3600",
-            "data": "ns-1086.awsdns-07.org."
-          },
-          {
-            "name": "cnn.com.",
-            "class": "IN",
-            "type": "NS",
-            "ttl": "3600",
-            "data": "ns-1630.awsdns-11.co.uk."
-          },
-          {
-            "name": "cnn.com.",
-            "class": "IN",
-            "type": "NS",
-            "ttl": "3600",
-            "data": "ns-47.awsdns-05.com."
-          },
-          {
-            "name": "cnn.com.",
-            "class": "IN",
-            "type": "NS",
-            "ttl": "3600",
-            "data": "ns-576.awsdns-08.net."
-          }
-        ],
-        "query_time": "26 msec",
-        "server": "205.251.194.64#53(205.251.194.64)",
-        "when": "Tue Nov 12 07:16:19 PST 2019",
-        "rcvd": "212"
+        "query_time": "48 msec",
+        "server": "2600:1700:bab0:d40::1#53(2600:1700:bab0:d40::1)",
+        "when": "Fri Apr 16 16:06:12 PDT 2021",
+        "rcvd": "56"
       }
     ]
 
     $ dig -x 1.1.1.1 | jc --dig -p
     [
       {
-        "id": 22191,
+        "id": 20785,
         "opcode": "QUERY",
         "status": "NOERROR",
         "flags": [
@@ -332,6 +202,13 @@ Examples:
         "answer_num": 1,
         "authority_num": 0,
         "additional_num": 1,
+        "opt_pseudosection": {
+          "edns": {
+            "version": 0,
+            "flags": [],
+            "udp": 4096
+          }
+        },
         "question": {
           "name": "1.1.1.1.in-addr.arpa.",
           "class": "IN",
@@ -346,11 +223,11 @@ Examples:
             "data": "one.one.one.one."
           }
         ],
-        "query_time": 44,
-        "server": "2600",
-        "when": "Tue Mar 30 20:10:34 PDT 2021",
+        "query_time": 40,
+        "server": "2600:1700:bab0:d40::1#53(2600:1700:bab0:d40::1)",
+        "when": "Sat Apr 17 14:50:50 PDT 2021",
         "rcvd": 78,
-        "when_epoch": 1617160234,
+        "when_epoch": 1618696250,
         "when_epoch_utc": null
       }
     ]
@@ -358,7 +235,7 @@ Examples:
     $ dig -x 1.1.1.1 | jc --dig -p -r
     [
       {
-        "id": "50986",
+        "id": "32644",
         "opcode": "QUERY",
         "status": "NOERROR",
         "flags": [
@@ -370,6 +247,13 @@ Examples:
         "answer_num": "1",
         "authority_num": "0",
         "additional_num": "1",
+        "opt_pseudosection": {
+          "edns": {
+            "version": "0",
+            "flags": [],
+            "udp": "4096"
+          }
+        },
         "question": {
           "name": "1.1.1.1.in-addr.arpa.",
           "class": "IN",
@@ -384,25 +268,61 @@ Examples:
             "data": "one.one.one.one."
           }
         ],
-        "query_time": "38 msec",
-        "server": "2600",
-        "when": "Tue Nov 12 07:17:19 PST 2019",
+        "query_time": "52 msec",
+        "server": "2600:1700:bab0:d40::1#53(2600:1700:bab0:d40::1)",
+        "when": "Sat Apr 17 14:51:46 PDT 2021",
         "rcvd": "78"
       }
     ]
+
+    $ dig +noall +answer cnn.com | jc --dig -p
+    [
+      {
+        "answer": [
+          {
+            "name": "cnn.com.",
+            "class": "IN",
+            "type": "A",
+            "ttl": 60,
+            "data": "151.101.193.67"
+          },
+          {
+            "name": "cnn.com.",
+            "class": "IN",
+            "type": "A",
+            "ttl": 60,
+            "data": "151.101.65.67"
+          },
+          {
+            "name": "cnn.com.",
+            "class": "IN",
+            "type": "A",
+            "ttl": 60,
+            "data": "151.101.1.67"
+          },
+          {
+            "name": "cnn.com.",
+            "class": "IN",
+            "type": "A",
+            "ttl": 60,
+            "data": "151.101.129.67"
+          }
+        ]
+      }
+    ]
 """
 import jc.utils
 
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.7'
+    version = '2.1'
     description = '`dig` command parser'
     author = 'Kelly Brazil'
     author_email = 'kellyjonbrazil@gmail.com'
 
     # compatible options: linux, darwin, cygwin, win32, aix, freebsd
-    compatible = ['linux', 'aix', 'freebsd', 'darwin']
+    compatible = ['linux', 'aix', 'freebsd', 'darwin', 'win32', 'cygwin']
     magic_commands = ['dig']
 
 
@@ -421,47 +341,36 @@ def _process(proc_data):
 
         List of Dictionaries. Structured data to conform to the schema.
     """
-
     for entry in proc_data:
-        int_list = ['id', 'query_num', 'answer_num', 'authority_num', 'additional_num', 'rcvd']
-        for key in int_list:
-            if key in entry:
-                try:
-                    key_int = int(entry[key])
-                    entry[key] = key_int
-                except (ValueError):
-                    entry[key] = None
+        int_list = ['id', 'query_num', 'answer_num', 'authority_num', 'additional_num', 'rcvd',
+                    'query_size', 'query_time']
+        for key in entry:
+            if key in int_list:
+                entry[key] = jc.utils.convert_to_int(entry[key])
 
         if 'axfr' in entry:
             for ax in entry['axfr']:
-                try:
-                    ttl_int = int(ax['ttl'])
-                    ax['ttl'] = ttl_int
-                except (ValueError):
-                    ax['ttl'] = None
+                ax['ttl'] = jc.utils.convert_to_int(ax['ttl'])
+
+        if 'opt_pseudosection' in entry:
+            if 'edns' in entry['opt_pseudosection']:
+                if 'version' in entry['opt_pseudosection']['edns']:
+                    entry['opt_pseudosection']['edns']['version'] = jc.utils.convert_to_int(entry['opt_pseudosection']['edns']['version'])
+
+                if 'udp' in entry['opt_pseudosection']['edns']:
+                    entry['opt_pseudosection']['edns']['udp'] = jc.utils.convert_to_int(entry['opt_pseudosection']['edns']['udp'])
 
         if 'answer' in entry:
             for ans in entry['answer']:
-                try:
-                    ttl_int = int(ans['ttl'])
-                    ans['ttl'] = ttl_int
-                except (ValueError):
-                    ans['ttl'] = None
+                ans['ttl'] = jc.utils.convert_to_int(ans['ttl'])
+
+        if 'additional' in entry:
+            for add in entry['additional']:
+                add['ttl'] = jc.utils.convert_to_int(add['ttl'])
 
         if 'authority' in entry:
             for auth in entry['authority']:
-                try:
-                    ttl_int = int(auth['ttl'])
-                    auth['ttl'] = ttl_int
-                except (ValueError):
-                    auth['ttl'] = None
-
-        if 'query_time' in entry:
-            try:
-                qt_int = int(entry['query_time'].split()[0])
-                entry['query_time'] = qt_int
-            except (ValueError):
-                entry['query_time'] = None
+                auth['ttl'] = jc.utils.convert_to_int(auth['ttl'])
 
         if 'when' in entry:
             ts = jc.utils.timestamp(entry['when'])
@@ -507,6 +416,31 @@ def _parse_flags_line(flagsline):
             'additional_num': additional_num}
 
 
+def _parse_opt_pseudosection(optline):
+    # ;; OPT PSEUDOSECTION:
+    # ; EDNS: version: 0, flags:; udp: 4096
+    # ; COOKIE: 1cbc06703eaef210
+    if optline.startswith('; EDNS:'):
+        optline_list = optline.replace(',', ' ').split(';')
+        optline_first = optline_list[1]
+        optline_rest = optline_list[2]
+        _, _, ver, _, *flags = optline_first.split()
+        udp = optline_rest.split()[-1]
+
+        return {
+            'edns': {
+                'version': ver,
+                'flags': flags,
+                'udp': udp
+            }
+        }
+
+    elif optline.startswith('; COOKIE:'):
+        return {
+            'cookie': optline.split()[2]
+        }
+
+
 def _parse_question(question):
     # ;www.cnn.com.           IN  A
     question = question.split()
@@ -519,22 +453,6 @@ def _parse_question(question):
             'type': dns_type}
 
 
-def _parse_authority(authority):
-    # cnn.com.      3600    IN  NS  ns-1086.awsdns-07.org.
-    authority = authority.split()
-    authority_name = authority[0]
-    authority_class = authority[2]
-    authority_type = authority[3]
-    authority_ttl = authority[1]
-    authority_data = authority[4]
-
-    return {'name': authority_name,
-            'class': authority_class,
-            'type': authority_type,
-            'ttl': authority_ttl,
-            'data': authority_data}
-
-
 def _parse_answer(answer):
     # www.cnn.com.        5   IN  CNAME   turner-tls.map.fastly.net.
     answer = answer.split(maxsplit=4)
@@ -574,6 +492,28 @@ def _parse_axfr(axfr):
             'data': axfr_data}
 
 
+def _parse_footer(footer):
+    # footer consists of 4 lines
+    # footer line 1
+    if footer.startswith(';; Query time:'):
+        return {'query_time': footer.split(':')[1].lstrip()}
+
+    # footer line 2
+    if footer.startswith(';; SERVER:'):
+        return {'server': footer.split(':', maxsplit=1)[1].lstrip()}
+
+    # footer line 3
+    if footer.startswith(';; WHEN:'):
+        return {'when': footer.split(':', maxsplit=1)[1].lstrip()}
+
+    # footer line 4 (last line)
+    if footer.startswith(';; MSG SIZE  rcvd:'):
+        return {'rcvd': footer.split(':')[1].lstrip()}
+
+    elif footer.startswith(';; XFR size:'):
+        return {'size': footer.split(':')[1].lstrip()}
+
+
 def parse(data, raw=False, quiet=False):
     """
     Main text parsing function
@@ -597,106 +537,112 @@ def parse(data, raw=False, quiet=False):
     # remove blank lines
     cleandata = list(filter(None, cleandata))
 
-    question = False
-    authority = False
-    answer = False
-    axfr = False
-
+    # section can be: header, flags, question, authority, answer, axfr, additional, opt_pseudosection, footer
+    section = ''
     output_entry = {}
+    answer_list = []
 
     if jc.utils.has_data(data):
         for line in cleandata:
 
+            # identify sections
+            if line.startswith(';; Got answer:'):
+                section = ''
+                continue
+
             if line.startswith('; <<>> ') and ' axfr ' in line.lower():
-                question = False
-                authority = False
-                answer = False
-                axfr = True
+                section = 'axfr'
                 axfr_list = []
                 continue
 
-            if ';' not in line and axfr:
-                axfr_list.append(_parse_axfr(line))
-                output_entry.update({'axfr': axfr_list})
-                continue
-
             if line.startswith(';; ->>HEADER<<-'):
+                section = 'header'
+                if output_entry:
+                    raw_output.append(output_entry)
                 output_entry = {}
                 output_entry.update(_parse_header(line))
                 continue
 
             if line.startswith(';; flags:'):
+                section = 'flags'
                 output_entry.update(_parse_flags_line(line))
                 continue
 
-            if line.startswith(';; QUESTION SECTION:'):
-                question = True
-                authority = False
-                answer = False
-                axfr = False
+            if line.startswith(';; OPT PSEUDOSECTION:'):
+                section = 'opt_pseudosection'
                 continue
 
-            if question:
-                output_entry['question'] = _parse_question(line)
-                question = False
-                authority = False
-                answer = False
-                axfr = False
+            if line.startswith(';; QUESTION SECTION:'):
+                section = 'question'
                 continue
 
             if line.startswith(';; AUTHORITY SECTION:'):
-                question = False
-                authority = True
-                answer = False
-                axfr = False
+                section = 'authority'
                 authority_list = []
                 continue
 
-            if ';' not in line and authority:
-                authority_list.append(_parse_authority(line))
-                output_entry.update({'authority': authority_list})
-                continue
-
             if line.startswith(';; ANSWER SECTION:'):
-                question = False
-                authority = False
-                answer = True
-                axfr = False
+                section = 'answer'
                 answer_list = []
                 continue
 
-            if ';' not in line and answer:
+            if line.startswith(';; ADDITIONAL SECTION:'):
+                section = 'additional'
+                additional_list = []
+                continue
+
+            if line.startswith(';; Query time:'):
+                section = 'footer'
+                output_entry.update(_parse_footer(line))
+                continue
+
+            # parse sections
+
+            if line.startswith(';; QUERY SIZE:'):
+                output_entry.update({'query_size': line.split(': ', maxsplit=1)[1]})
+                continue
+
+            if not line.startswith(';') and section == 'axfr':
+                axfr_list.append(_parse_axfr(line))
+                output_entry.update({'axfr': axfr_list})
+                continue
+
+            if section == 'opt_pseudosection':
+                if 'opt_pseudosection' not in output_entry:
+                    output_entry['opt_pseudosection'] = {}
+                output_entry['opt_pseudosection'].update(_parse_opt_pseudosection(line))
+                continue
+
+            if section == 'question':
+                output_entry['question'] = _parse_question(line)
+                continue
+
+            if not line.startswith(';') and section == 'authority':
+                authority_list.append(_parse_answer(line))
+                output_entry.update({'authority': authority_list})
+                continue
+
+            # https://github.com/kellyjonbrazil/jc/issues/133
+            # to allow parsing of output that only has the answer section - e.g:
+            # dig +noall +answer example.com
+            # we allow section to be 'answer' (normal output) or
+            # '', which means +noall +answer was used.
+            if not line.startswith(';') and (section == 'answer' or section == ''):
                 answer_list.append(_parse_answer(line))
                 output_entry.update({'answer': answer_list})
                 continue
 
-            # footer consists of 4 lines
-            # footer line 1
-            if line.startswith(';; Query time:'):
-                output_entry.update({'query_time': line.split(':')[1].lstrip()})
+            if not line.startswith(';') and section == 'additional':
+                additional_list.append(_parse_answer(line))
+                output_entry.update({'additional': additional_list})
                 continue
 
-            # footer line 2
-            if line.startswith(';; SERVER:'):
-                output_entry.update({'server': line.split(':')[1].lstrip()})
+            if section == 'footer':
+                output_entry.update(_parse_footer(line))
                 continue
 
-            # footer line 3
-            if line.startswith(';; WHEN:'):
-                output_entry.update({'when': line.split(':', maxsplit=1)[1].lstrip()})
-                continue
-
-            # footer line 4 (last line)
-            if line.startswith(';; MSG SIZE  rcvd:'):
-                output_entry.update({'rcvd': line.split(':')[1].lstrip()})
-
-                if output_entry:
-                    raw_output.append(output_entry)
-            elif line.startswith(';; XFR size:'):
-                output_entry.update({'size': line.split(':')[1].lstrip()})
-
-                if output_entry:
-                    raw_output.append(output_entry)
+        if output_entry:
+            raw_output.append(output_entry)
 
         raw_output = list(filter(None, raw_output))
 

jc/parsers/dir.py

@@ -6,16 +6,14 @@ Options supported:
 - `/C, /-C`
 - `/S`
 
+The "Magic" syntax is not supported since the `dir` command is a shell builtin.
+
 The `epoch` calculated timestamp field is naive (i.e. based on the local time of the system the parser is run on)
 
 Usage (cli):
 
     C:> dir | jc --dir
 
-    or
-
-    C:> jc dir
-
 Usage (module):
 
     import jc.parsers.dir
@@ -121,14 +119,13 @@ import jc.utils
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.1'
+    version = '1.3'
     description = '`dir` command parser'
     author = 'Rasheed Elsaleh'
     author_email = 'rasheed@rebelliondefense.com'
 
     # compatible options: win32
     compatible = ['win32']
-    magic_commands = ['dir']
 
 
 __version__ = info.version
@@ -155,14 +152,9 @@ def _process(proc_data):
 
         # add ints
         int_list = ["size"]
-        for key in int_list:
-            if entry.get(key):
-                try:
-                    key_int = int(entry[key].replace(",", ""))
-                except ValueError:
-                    entry[key] = None
-                else:
-                    entry[key] = key_int
+        for key in entry:
+            if key in int_list:
+                entry[key] = jc.utils.convert_to_int(entry[key])
 
     return proc_data
 

jc/parsers/dmidecode.py

@@ -125,7 +125,7 @@ import jc.utils
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.2'
+    version = '1.3'
     description = '`dmidecode` command parser'
     author = 'Kelly Brazil'
     author_email = 'kellyjonbrazil@gmail.com'
@@ -153,13 +153,9 @@ def _process(proc_data):
     """
     for entry in proc_data:
         int_list = ['type', 'bytes']
-        for key in int_list:
-            if key in entry:
-                try:
-                    key_int = int(entry[key])
-                    entry[key] = key_int
-                except (ValueError):
-                    entry[key] = None
+        for key in entry:
+            if key in int_list:
+                entry[key] = jc.utils.convert_to_int(entry[key])
 
         if not entry['values']:
             entry['values'] = None

jc/parsers/du.py

@@ -88,7 +88,7 @@ import jc.parsers.universal
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.3'
+    version = '1.4'
     description = '`du` command parser'
     author = 'Kelly Brazil'
     author_email = 'kellyjonbrazil@gmail.com'
@@ -116,13 +116,9 @@ def _process(proc_data):
     """
     int_list = ['size']
     for entry in proc_data:
-        for key in int_list:
-            if key in entry:
-                try:
-                    key_int = int(entry[key])
-                    entry[key] = key_int
-                except (ValueError):
-                    entry[key] = None
+        for key in entry:
+            if key in int_list:
+                entry[key] = jc.utils.convert_to_int(entry[key])
 
     return proc_data
 

jc/parsers/file.py

@@ -130,7 +130,7 @@ def parse(data, raw=False, quiet=False):
                 )
             except IndexError:
                 if not warned:
-                    jc.utils.warning_message('Filenames with newline characters detected. Some filenames may be truncated.')
+                    jc.utils.warning_message(['Filenames with newline characters detected. Some filenames may be truncated.'])
                     warned = True
 
     if raw:

jc/parsers/foo.py

@@ -65,7 +65,10 @@ def _process(proc_data):
         List of Dictionaries. Structured to conform to the schema.
     """
 
+    # process the data here
     # rebuild output for added semantic information
+    # use helper functions in jc.utils for int, float, bool conversions and timestamps
+
     return proc_data
 
 
@@ -91,10 +94,9 @@ def parse(data, raw=False, quiet=False):
     if jc.utils.has_data(data):
 
         for line in filter(None, data.splitlines()):
-            # parse the content
+            #
+            # parse the content here
+            #
             pass
 
-    if raw:
-        return raw_output
-    else:
-        return _process(raw_output)
+    return raw_output if raw else _process(raw_output)

jc/parsers/foo_s.py

@@ -0,0 +1,115 @@
+"""jc - JSON CLI output utility `foo` command output streaming parser
+
+<<Short foo description and caveats>>
+
+Usage (cli):
+
+    $ foo | jc --foo-s
+
+Usage (module):
+
+    import jc.parsers.foo_s
+    result = jc.parsers.foo_s.parse(foo_command_output.splitlines())    # result is an iterable object
+    for item in result:
+        # do something
+
+Schema:
+
+    {
+      "foo":            string,
+      "_jc_meta":                    # This object only exists if using -qq or ignore_exceptions=True
+        {
+          "success":    booean,      # true if successfully parsed, false if error
+          "error":      string,      # exists if "success" is false
+          "line":       string       # exists if "success" is false
+        }
+    }
+
+Examples:
+
+    $ foo | jc --foo-s
+    {example output}
+    ...
+
+    $ foo | jc --foo-s -r
+    {example output}
+    ...
+"""
+import jc.utils
+from jc.utils import stream_success, stream_error
+from jc.exceptions import ParseError
+
+
+class info():
+    """Provides parser metadata (version, author, etc.)"""
+    version = '1.0'
+    description = '`foo` command streaming parser'
+    author = 'John Doe'
+    author_email = 'johndoe@gmail.com'
+
+    # compatible options: linux, darwin, cygwin, win32, aix, freebsd
+    compatible = ['linux', 'darwin', 'cygwin', 'aix', 'freebsd']
+    streaming = True
+
+
+__version__ = info.version
+
+
+def _process(proc_data):
+    """
+    Final processing to conform to the schema.
+
+    Parameters:
+
+        proc_data:   (Dictionary) raw structured data to process
+
+    Returns:
+
+        Dictionary. Structured data to conform to the schema.
+    """
+    #
+    # process the data here
+    # rebuild output for added semantic information
+    # use helper functions in jc.utils for int, float, bool conversions and timestamps
+    #
+
+    return proc_data
+
+
+def parse(data, raw=False, quiet=False, ignore_exceptions=False):
+    """
+    Main text parsing generator function. Returns an iterator object.
+
+    Parameters:
+
+        data:              (iterable)  line-based text data to parse (e.g. sys.stdin or str.splitlines())
+        raw:               (boolean)   output preprocessed JSON if True
+        quiet:             (boolean)   suppress warning messages if True
+        ignore_exceptions: (boolean)   ignore parsing exceptions if True
+
+    Yields:
+
+        Dictionary. Raw or processed structured data.
+
+    Returns:
+
+        Iterator object
+    """
+    if not quiet:
+        jc.utils.compatibility(__name__, info.compatible)
+
+    for line in data:
+        try:
+            output_line = {}
+
+            #
+            # parse the input here
+            #
+
+            if output_line:
+                yield stream_success(output_line, ignore_exceptions) if raw else stream_success(_process(output_line), ignore_exceptions)
+            else:
+                raise ParseError('Not foo data')
+
+        except Exception as e:
+            yield stream_error(e, ignore_exceptions, line)

jc/parsers/free.py

@@ -73,7 +73,7 @@ import jc.parsers.universal
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.3'
+    version = '1.4'
     description = '`free` command parser'
     author = 'Kelly Brazil'
     author_email = 'kellyjonbrazil@gmail.com'
@@ -101,13 +101,9 @@ def _process(proc_data):
 
     for entry in proc_data:
         int_list = ['total', 'used', 'free', 'shared', 'buff_cache', 'available']
-        for key in int_list:
-            if key in entry:
-                try:
-                    key_int = int(entry[key])
-                    entry[key] = key_int
-                except (ValueError):
-                    entry[key] = None
+        for key in entry:
+            if key in int_list:
+                entry[key] = jc.utils.convert_to_int(entry[key])
 
     return proc_data
 

jc/parsers/fstab.py

@@ -85,7 +85,7 @@ import jc.utils
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.4'
+    version = '1.5'
     description = '`/etc/fstab` file parser'
     author = 'Kelly Brazil'
     author_email = 'kellyjonbrazil@gmail.com'
@@ -111,13 +111,9 @@ def _process(proc_data):
     """
     for entry in proc_data:
         int_list = ['fs_freq', 'fs_passno']
-        for key in int_list:
-            if key in entry:
-                try:
-                    key_int = int(entry[key])
-                    entry[key] = key_int
-                except (ValueError):
-                    entry[key] = None
+        for key in entry:
+            if key in int_list:
+                entry[key] = jc.utils.convert_to_int(entry[key])
 
     return proc_data
 

jc/parsers/group.py

@@ -109,7 +109,7 @@ import jc.utils
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.2'
+    version = '1.3'
     description = '`/etc/group` file parser'
     author = 'Kelly Brazil'
     author_email = 'kellyjonbrazil@gmail.com'
@@ -136,13 +136,9 @@ def _process(proc_data):
     """
     for entry in proc_data:
         int_list = ['gid']
-        for key in int_list:
-            if key in entry:
-                try:
-                    key_int = int(entry[key])
-                    entry[key] = key_int
-                except (ValueError):
-                    entry[key] = None
+        for key in entry:
+            if key in int_list:
+                entry[key] = jc.utils.convert_to_int(entry[key])
 
         if entry['members'] == ['']:
             entry['members'] = []

jc/parsers/hash.py

@@ -38,7 +38,7 @@ import jc.parsers.universal
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.1'
+    version = '1.2'
     description = '`hash` command parser'
     author = 'Kelly Brazil'
     author_email = 'kellyjonbrazil@gmail.com'
@@ -63,15 +63,10 @@ def _process(proc_data):
         List of Dictionaries. Structured data to conform to the schema.
     """
     for entry in proc_data:
-        # change to int
         int_list = ['hits']
-        for key in int_list:
-            if key in entry:
-                try:
-                    key_int = int(entry[key])
-                    entry[key] = key_int
-                except (ValueError):
-                    entry[key] = None
+        for key in entry:
+            if key in int_list:
+                entry[key] = jc.utils.convert_to_int(entry[key])
 
     return proc_data
 

jc/parsers/hciconfig.py

@@ -317,7 +317,7 @@ import jc.utils
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.1'
+    version = '1.2'
     description = '`hciconfig` command parser'
     author = 'Kelly Brazil'
     author_email = 'kellyjonbrazil@gmail.com'
@@ -346,15 +346,11 @@ def _process(proc_data):
 
     for entry in proc_data:
 
-        # integers
         int_list = ['acl_mtu', 'acl_mtu_packets', 'sco_mtu', 'sco_mtu_packets', 'rx_bytes', 'rx_acl', 'rx_sco',
                     'rx_events', 'rx_errors', 'tx_bytes', 'tx_acl', 'tx_sco', 'tx_commands', 'tx_errors']
-        for key in int_list:
-            if key in entry:
-                try:
-                    entry[key] = int(entry[key])
-                except (ValueError):
-                    entry[key] = None
+        for key in entry:
+            if key in int_list:
+                entry[key] = jc.utils.convert_to_int(entry[key])
 
         if 'service_classes' in entry and len(entry['service_classes']) == 1 and 'Unspecified' in entry['service_classes']:
             entry['service_classes'] = None

jc/parsers/history.py

@@ -2,6 +2,8 @@
 
 This parser will output a list of dictionaries each containing `line` and `command` keys. If you would like a simple dictionary output, then use the `-r` command-line option or the `raw=True` argument in the `parse()` function.
 
+The "Magic" syntax is not supported since the `history` command is a shell builtin.
+
 Usage (cli):
 
     $ history | jc --history
@@ -57,7 +59,7 @@ import jc.utils
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.4'
+    version = '1.5'
     description = '`history` command parser'
     author = 'Kelly Brazil'
     author_email = 'kellyjonbrazil@gmail.com'
@@ -82,12 +84,10 @@ def _process(proc_data):
 
         List of Dictionaries. Structured data to conform to the schema.
     """
-
-    # rebuild output for added semantic information
     processed = []
     for k, v in proc_data.items():
         proc_line = {
-            'line': int(k) if k.isdigit() else None,
+            'line': jc.utils.convert_to_int(k),
             'command': v,
         }
         processed.append(proc_line)

jc/parsers/id.py

@@ -105,7 +105,7 @@ import jc.utils
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.2'
+    version = '1.3'
     description = '`id` command parser'
     author = 'Kelly Brazil'
     author_email = 'kellyjonbrazil@gmail.com'
@@ -133,25 +133,16 @@ def _process(proc_data):
     """
     if 'uid' in proc_data:
         if 'id' in proc_data['uid']:
-            try:
-                proc_data['uid']['id'] = int(proc_data['uid']['id'])
-            except (ValueError):
-                proc_data['uid']['id'] = None
+            proc_data['uid']['id'] = jc.utils.convert_to_int(proc_data['uid']['id'])
 
     if 'gid' in proc_data:
         if 'id' in proc_data['gid']:
-            try:
-                proc_data['gid']['id'] = int(proc_data['gid']['id'])
-            except (ValueError):
-                proc_data['gid']['id'] = None
+            proc_data['gid']['id'] = jc.utils.convert_to_int(proc_data['gid']['id'])
 
     if 'groups' in proc_data:
         for group in proc_data['groups']:
             if 'id' in group:
-                try:
-                    group['id'] = int(group['id'])
-                except (ValueError):
-                    group['id'] = None
+                group['id'] = jc.utils.convert_to_int(group['id'])
 
     return proc_data
 

jc/parsers/ifconfig.py

@@ -188,7 +188,7 @@ import jc.utils
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.9'
+    version = '1.10'
     description = '`ifconfig` command parser'
     author = 'Kelly Brazil'
     author_email = 'kellyjonbrazil@gmail.com'
@@ -264,7 +264,7 @@ class _IfconfigParser(object):
         :return:
         """
         for attr in kwargs.keys():
-            if attr not in IfconfigParser.attributes:
+            if attr not in _IfconfigParser.attributes:
                 raise ValueError("Attribute [{}] not supported.".format(attr))
 
         filtered_interfaces = []
@@ -398,7 +398,8 @@ class _IfconfigParser(object):
                 if match:
                     details = match.groupdict()
                     for k, v in details.items():
-                        if isinstance(v, str): details[k] = v.strip()
+                        if isinstance(v, str):
+                            details[k] = v.strip()
                     _interface.update(details)
             if _interface is not None:
                 available_interfaces[_interface['name']] = self.update_interface_details(_interface)
@@ -433,13 +434,9 @@ def _process(proc_data):
         int_list = ['flags', 'mtu', 'ipv6_mask', 'rx_packets', 'rx_bytes', 'rx_errors', 'rx_dropped', 'rx_overruns',
                     'rx_frame', 'tx_packets', 'tx_bytes', 'tx_errors', 'tx_dropped', 'tx_overruns', 'tx_carrier',
                     'tx_collisions', 'metric']
-        for key in int_list:
-            if key in entry:
-                try:
-                    key_int = int(entry[key])
-                    entry[key] = key_int
-                except (ValueError, TypeError):
-                    entry[key] = None
+        for key in entry:
+            if key in int_list:
+                entry[key] = jc.utils.convert_to_int(entry[key])
 
         # convert OSX-style subnet mask to dotted quad
         if 'ipv4_mask' in entry:

jc/parsers/iptables.py

@@ -163,7 +163,7 @@ import jc.utils
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.5'
+    version = '1.6'
     description = '`iptables` command parser'
     author = 'Kelly Brazil'
     author_email = 'kellyjonbrazil@gmail.com'
@@ -191,13 +191,9 @@ def _process(proc_data):
     for entry in proc_data:
         for rule in entry['rules']:
             int_list = ['num', 'pkts']
-            for key in int_list:
-                if key in rule:
-                    try:
-                        key_int = int(rule[key])
-                        rule[key] = key_int
-                    except (ValueError):
-                        rule[key] = None
+            for key in rule:
+                if key in int_list:
+                    rule[key] = jc.utils.convert_to_int(rule[key])
 
             if 'bytes' in rule:
                 multiplier = 1
@@ -218,7 +214,7 @@ def _process(proc_data):
                     rule['bytes'] = rule['bytes'].rstrip('P')
 
                 try:
-                    bytes_int = int(rule['bytes'])
+                    bytes_int = jc.utils.convert_to_int(rule['bytes'])
                     rule['bytes'] = bytes_int * multiplier
                 except (ValueError):
                     rule['bytes'] = None

jc/parsers/jobs.py

@@ -2,14 +2,12 @@
 
 Also supports the `-l` option.
 
+The "Magic" syntax is not supported since the `jobs` command is a shell builtin.
+
 Usage (cli):
 
     $ jobs | jc --jobs
 
-    or
-
-    $ jc jobs
-
 Usage (module):
 
     import jc.parsers.jobs
@@ -95,7 +93,7 @@ import jc.utils
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.3'
+    version = '1.4'
     description = '`jobs` command parser'
     author = 'Kelly Brazil'
     author_email = 'kellyjonbrazil@gmail.com'
@@ -122,13 +120,9 @@ def _process(proc_data):
     """
     for entry in proc_data:
         int_list = ['job_number', 'pid']
-        for key in int_list:
-            if key in entry:
-                try:
-                    key_int = int(entry[key])
-                    entry[key] = key_int
-                except (ValueError):
-                    entry[key] = None
+        for key in entry:
+            if key in int_list:
+                entry[key] = jc.utils.convert_to_int(entry[key])
 
     return proc_data
 

jc/parsers/kv.py

@@ -35,7 +35,7 @@ Examples:
     # but can be preserved with the -r argument
     occupation:"Engineer"
 
-    $ cat keyvalue.txt | jc --ini -p
+    $ cat keyvalue.txt | jc --kv -p
     {
       "name": "John Doe",
       "address": "555 California Drive",

jc/parsers/last.py

@@ -104,7 +104,7 @@ import jc.utils
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.6'
+    version = '1.7'
     description = '`last` and `lastb` command parser'
     author = 'Kelly Brazil'
     author_email = 'kellyjonbrazil@gmail.com'
@@ -161,7 +161,7 @@ def _process(proc_data):
             entry['logout_epoch'] = timestamp.naive
 
         if 'login_epoch' in entry and 'logout_epoch' in entry:
-            entry['duration_seconds'] = int(entry['logout_epoch']) - int(entry['login_epoch'])
+            entry['duration_seconds'] = entry['logout_epoch'] - entry['login_epoch']
 
         if 'duration' in entry and re.match(r'^\d+\+', entry['duration']):
             m = re.match(r'^(?P<days>\d+)\+(?P<hours>\d\d):(?P<minutes>\d\d)', entry['duration'])

jc/parsers/ls.py

@@ -3,7 +3,6 @@
 Options supported:
 - `lbaR1`
 - `--time-style=full-iso`
-- `-h`: File sizes will be available in text form with `-r` but larger file sizes with human readable suffixes will be converted to `Null` in the default view since the parser attempts to convert this field to an integer.
 
 Note: The `-1`, `-l`, or `-b` option of `ls` should be used to correctly parse filenames that include newline characters. Since `ls` does not encode newlines in filenames when outputting to a pipe it will cause `jc` to see multiple files instead of a single file if `-1`, `-l`, or `-b` is not used. Alternatively, `vdir` can be used, which is the same as running `ls -lb`.
 
@@ -108,7 +107,7 @@ import jc.utils
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.8'
+    version = '1.9'
     description = '`ls` command parser'
     author = 'Kelly Brazil'
     author_email = 'kellyjonbrazil@gmail.com'
@@ -135,13 +134,9 @@ def _process(proc_data):
     """
     for entry in proc_data:
         int_list = ['links', 'size']
-        for key in int_list:
-            if key in entry:
-                try:
-                    key_int = int(entry[key])
-                    entry[key] = key_int
-                except (ValueError):
-                    entry[key] = None
+        for key in entry:
+            if key in int_list:
+                entry[key] = jc.utils.convert_to_int(entry[key])
 
         if 'date' in entry:
             # to speed up processing only try to convert the date if it's not the default format
@@ -260,7 +255,7 @@ def parse(data, raw=False, quiet=False):
                     continue
 
                 if not quiet and next_is_parent and not entry.endswith(':') and not warned:
-                    jc.utils.warning_message('Newline characters detected. Filenames probably corrupted. Use ls -l or -b instead.')
+                    jc.utils.warning_message(['Newline characters detected. Filenames probably corrupted. Use ls -l or -b instead.'])
                     warned = True
 
                 output_line['filename'] = entry

jc/parsers/ls_s.py

@@ -0,0 +1,178 @@
+"""jc - JSON CLI output utility `ls` and `vdir` command output streaming parser
+
+This streaming parser requires the `-l` option to be used on `ls`. If there are newline characters in the filename, then make sure to use the `-b` option on `ls`.
+
+The `jc` `-q` option can be used to ignore parsing errors. (e.g. filenames with newline characters, but `-b` was not used)
+
+The `epoch` calculated timestamp field is naive (i.e. based on the local time of the system the parser is run on)
+
+The `epoch_utc` calculated timestamp field is timezone-aware and is only available if the timezone field is UTC.
+
+Usage (cli):
+
+    $ ls | jc --ls-s
+
+Usage (module):
+
+    import jc.parsers.ls_s
+    result = jc.parsers.ls_s.parse(ls_command_output.splitlines())    # result is an iterable object
+    for item in result:
+        # do something
+
+Schema:
+
+    {
+      "filename":       string,
+      "flags":          string,
+      "links":          integer,
+      "parent":         string,
+      "owner":          string,
+      "group":          string,
+      "size":           integer,
+      "date":           string,
+      "epoch":          integer,     # naive timestamp if date field exists and can be converted
+      "epoch_utc":      integer,     # timezone aware timestamp if date field is in UTC and can be converted
+      "_jc_meta":                    # This object only exists if using -qq or ignore_exceptions=True
+        {
+          "success":    booean,      # true if successfully parsed, false if error
+          "error":      string,      # exists if "success" is false
+          "line":       string       # exists if "success" is false
+        }
+    }
+
+Examples:
+
+    $ ls -l /usr/bin | jc --ls-s
+    {"filename":"2to3-","flags":"-rwxr-xr-x","links":4,"owner":"root","group":"wheel","size":925,"date":"Feb 22 2019"}
+    {"filename":"2to3-2.7","link_to":"../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/2to3-2.7","flags":"lrwxr-xr-x","links":1,"owner":"root","group":"wheel","size":74,"date":"May 4 2019"}
+    {"filename":"AssetCacheLocatorUtil","flags":"-rwxr-xr-x","links":1,"owner":"root","group":"wheel","size":55152,"date":"May 3 2019"}
+    ...
+
+    $ ls -l /usr/bin | jc --ls-s -r
+    {"filename":"2to3-","flags":"-rwxr-xr-x","links":"4","owner":"root","group":"wheel","size":"925","date":"Feb 22 2019"}
+    {"filename":"2to3-2.7","link_to":"../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/2to3-2.7","flags":"lrwxr-xr-x","links":"1","owner":"root","group":"wheel","size":"74","date":"May 4 2019"}
+    {"filename":"AssetCacheLocatorUtil","flags":"-rwxr-xr-x","links":"1","owner":"root","group":"wheel","size":"55152","date":"May 3 2019"}
+    ...
+"""
+import re
+import jc.utils
+from jc.utils import stream_success, stream_error
+from jc.exceptions import ParseError
+
+
+class info():
+    """Provides parser metadata (version, author, etc.)"""
+    version = '0.5'
+    description = '`ls` command streaming parser'
+    author = 'Kelly Brazil'
+    author_email = 'kellyjonbrazil@gmail.com'
+
+    # compatible options: linux, darwin, cygwin, win32, aix, freebsd
+    compatible = ['linux', 'darwin', 'cygwin', 'aix', 'freebsd']
+    streaming = True
+
+
+__version__ = info.version
+
+
+def _process(proc_data):
+    """
+    Final processing to conform to the schema.
+
+    Parameters:
+
+        proc_data:   (Dictionary) raw structured data to process
+
+    Returns:
+
+        Dictionary. Structured data to conform to the schema.
+    """
+    int_list = ['links', 'size']
+    for key in proc_data:
+        if key in int_list:
+            proc_data[key] = jc.utils.convert_to_int(proc_data[key])
+
+    if 'date' in proc_data:
+        # to speed up processing only try to convert the date if it's not the default format
+        if not re.match(r'[a-zA-Z]{3}\s{1,2}\d{1,2}\s{1,2}[0-9:]{4,5}', proc_data['date']):
+            ts = jc.utils.timestamp(proc_data['date'])
+            proc_data['epoch'] = ts.naive
+            proc_data['epoch_utc'] = ts.utc
+
+    return proc_data
+
+
+def parse(data, raw=False, quiet=False, ignore_exceptions=False):
+    """
+    Main text parsing generator function. Returns an iterator object.
+
+    Parameters:
+
+        data:              (iterable)  line-based text data to parse (e.g. sys.stdin or str.splitlines())
+        raw:               (boolean)   output preprocessed JSON if True
+        quiet:             (boolean)   suppress warning messages if True
+        ignore_exceptions: (boolean)   ignore parsing exceptions if True
+
+    Yields:
+
+        Dictionary. Raw or processed structured data.
+
+    Returns:
+
+        Iterator object
+    """
+    if not quiet:
+        jc.utils.compatibility(__name__, info.compatible)
+
+    parent = ''
+
+    for line in data:
+        try:
+
+            # skip line if it starts with 'total 1234'
+            if re.match(r'total [0-9]+', line):
+                continue
+
+            # skip blank lines
+            if line.strip() == '':
+                continue
+
+            # Look for parent line if glob or -R is used
+            if not re.match(r'[-dclpsbDCMnP?]([-r][-w][-xsS]){2}([-r][-w][-xtT])[+]?', line) \
+               and line.strip().endswith(':'):
+                parent = line.strip()[:-1]
+                continue
+
+            if not re.match(r'[-dclpsbDCMnP?]([-r][-w][-xsS]){2}([-r][-w][-xtT])[+]?', line):
+                raise ParseError('Not ls -l data')
+
+            parsed_line = line.strip().split(maxsplit=8)
+            output_line = {}
+
+            # split filenames and links
+            if len(parsed_line) == 9:
+                filename_field = parsed_line[8].split(' -> ')
+            else:
+                # in case of filenames starting with a newline character
+                filename_field = ['']
+
+            # create output object
+            output_line['filename'] = filename_field[0]
+
+            if len(filename_field) > 1:
+                output_line['link_to'] = filename_field[1]
+
+            if parent:
+                output_line['parent'] = parent
+
+            output_line['flags'] = parsed_line[0]
+            output_line['links'] = parsed_line[1]
+            output_line['owner'] = parsed_line[2]
+            output_line['group'] = parsed_line[3]
+            output_line['size'] = parsed_line[4]
+            output_line['date'] = ' '.join(parsed_line[5:8])
+
+            yield stream_success(output_line, ignore_exceptions) if raw else stream_success(_process(output_line), ignore_exceptions)
+
+        except Exception as e:
+            yield stream_error(e, ignore_exceptions, line)

jc/parsers/lsblk.py

@@ -269,7 +269,7 @@ import jc.parsers.universal
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.6'
+    version = '1.7'
     description = '`lsblk` command parser'
     author = 'Kelly Brazil'
     author_email = 'kellyjonbrazil@gmail.com'
@@ -295,25 +295,14 @@ def _process(proc_data):
         List of Dictionaries. Structured data to conform to the schema.
     """
     for entry in proc_data:
-        # boolean changes
+        # boolean and integer changes
         bool_list = ['rm', 'ro', 'rota', 'disc_zero', 'rand']
-        for key in bool_list:
-            if key in entry:
-                try:
-                    key_bool = bool(int(entry[key]))
-                    entry[key] = key_bool
-                except (ValueError):
-                    entry[key] = None
-
-        # integer changes
         int_list = ['ra', 'alignment', 'min_io', 'opt_io', 'phy_sec', 'log_sec', 'rq_size', 'disc_aln']
-        for key in int_list:
-            if key in entry:
-                try:
-                    key_int = int(entry[key])
-                    entry[key] = key_int
-                except (ValueError):
-                    entry[key] = None
+        for key in entry:
+            if key in bool_list:
+                entry[key] = jc.utils.convert_to_bool(entry[key])
+            if key in int_list:
+                entry[key] = jc.utils.convert_to_int(entry[key])
 
     return proc_data
 

jc/parsers/lsmod.py

@@ -126,7 +126,7 @@ import jc.parsers.universal
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.4'
+    version = '1.5'
     description = '`lsmod` command parser'
     author = 'Kelly Brazil'
     author_email = 'kellyjonbrazil@gmail.com'
@@ -152,15 +152,10 @@ def _process(proc_data):
         List of Dictionaries. Structured data to conform to the schema.
     """
     for entry in proc_data:
-        # integer changes
         int_list = ['size', 'used']
-        for key in int_list:
-            if key in entry:
-                try:
-                    key_int = int(entry[key])
-                    entry[key] = key_int
-                except (ValueError):
-                    entry[key] = None
+        for key in entry:
+            if key in int_list:
+                entry[key] = jc.utils.convert_to_int(entry[key])
 
     return proc_data
 

jc/parsers/lsof.py

@@ -120,7 +120,7 @@ import jc.parsers.universal
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.3'
+    version = '1.4'
     description = '`lsof` command parser'
     author = 'Kelly Brazil'
     author_email = 'kellyjonbrazil@gmail.com'
@@ -146,15 +146,11 @@ def _process(proc_data):
         List of Dictionaries. Structured data to conform to the schema.
     """
     for entry in proc_data:
-        # integer changes
         int_list = ['pid', 'tid', 'size_off', 'node']
-        for key in int_list:
-            if key in entry:
-                try:
-                    key_int = int(entry[key])
-                    entry[key] = key_int
-                except (ValueError, TypeError):
-                    entry[key] = None
+        for key in entry:
+            if key in int_list:
+                entry[key] = jc.utils.convert_to_int(entry[key])
+
     return proc_data
 
 

jc/parsers/netstat.py

@@ -349,11 +349,12 @@ Examples:
       }
     ]
 """
+import jc.utils
 
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.9'
+    version = '1.10'
     description = '`netstat` command parser'
     author = 'Kelly Brazil'
     author_email = 'kellyjonbrazil@gmail.com'
@@ -379,7 +380,7 @@ def _process(proc_data):
         List of Dictionaries. Structured data to conform to the schema.
     """
     for entry in proc_data:
-        # integer changes
+        # integer and float conversions
         int_list = ['recv_q', 'send_q', 'pid', 'refcnt', 'inode', 'unit', 'vendor', 'class',
                     'osx_flags', 'subcla', 'pcbcount', 'rcvbuf', 'sndbuf', 'rxbytes', 'txbytes',
                     'route_refs', 'use', 'mtu', 'mss', 'window', 'irtt', 'metric', 'ipkts',
@@ -387,35 +388,23 @@ def _process(proc_data):
                     'tx_ok', 'tx_err', 'tx_drp', 'tx_ovr', 'idrop', 'ibytes', 'obytes', 'r_mbuf',
                     's_mbuf', 'r_clus', 's_clus', 'r_hiwa', 's_hiwa', 'r_lowa', 's_lowa', 'r_bcnt',
                     's_bcnt', 'r_bmax', 's_bmax', 'rexmit', 'ooorcv', '0_win']
-        for key in int_list:
-            if key in entry:
-                try:
-                    key_int = int(entry[key])
-                    entry[key] = key_int
-                except (ValueError):
-                    entry[key] = None
-
-        # float changes
         float_list = ['rexmt', 'persist', 'keep', '2msl', 'delack', 'rcvtime']
-        for key in float_list:
-            if key in entry:
-                try:
-                    key_float = float(entry[key])
-                    entry[key] = key_float
-                except (ValueError):
-                    entry[key] = None
+        for key in entry:
+            if key in int_list:
+                entry[key] = jc.utils.convert_to_int(entry[key])
+            if key in float_list:
+                entry[key] = jc.utils.convert_to_float(entry[key])
 
+        # add number keys
         if 'local_port' in entry:
-            try:
-                entry['local_port_num'] = int(entry['local_port'])
-            except (ValueError):
-                pass
+            local_num = jc.utils.convert_to_int(entry['local_port'])
+            if local_num:
+                entry['local_port_num'] = local_num
 
         if 'foreign_port' in entry:
-            try:
-                entry['foreign_port_num'] = int(entry['foreign_port'])
-            except (ValueError):
-                pass
+            foreign_num = jc.utils.convert_to_int(entry['foreign_port'])
+            if foreign_num:
+                entry['foreign_port_num'] = foreign_num
 
     return proc_data
 

jc/parsers/ntpq.py

@@ -207,7 +207,7 @@ import jc.parsers.universal
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.4'
+    version = '1.5'
     description = '`ntpq -p` command parser'
     author = 'Kelly Brazil'
     author_email = 'kellyjonbrazil@gmail.com'
@@ -240,20 +240,12 @@ def _process(proc_data):
         entry['state'] = entry.pop('s')
 
         int_list = ['st', 'when', 'poll', 'reach']
-        for key in int_list:
-            if key in entry:
-                try:
-                    entry[key] = int(entry[key])
-                except (ValueError):
-                    entry[key] = None
-
         float_list = ['delay', 'offset', 'jitter']
-        for key in float_list:
-            if key in entry:
-                try:
-                    entry[key] = float(entry[key])
-                except (ValueError):
-                    entry[key] = None
+        for key in entry:
+            if key in int_list:
+                entry[key] = jc.utils.convert_to_int(entry[key])
+            if key in float_list:
+                entry[key] = jc.utils.convert_to_float(entry[key])
 
     return proc_data
 

jc/parsers/passwd.py

@@ -94,7 +94,7 @@ import jc.utils
 
 class info():
     """Provides parser metadata (version, author, etc.)"""
-    version = '1.2'
+    version = '1.3'
     description = '`/etc/passwd` file parser'
     author = 'Kelly Brazil'
     author_email = 'kellyjonbrazil@gmail.com'
@@ -121,13 +121,9 @@ def _process(proc_data):
     """
     for entry in proc_data:
         int_list = ['uid', 'gid']
-        for key in int_list:
-            if key in entry:
-                try:
-                    key_int = int(entry[key])
-                    entry[key] = key_int
-                except (ValueError):
-                    entry[key] = None
+        for key in entry:
+            if key in int_list:
+                entry[key] = jc.utils.convert_to_int(entry[key])
 
     return proc_data