Try the new
jcweb demo!
JC is now available as an Ansible filter plugin in the
community.generalcollection! See this blog post for an example.
JC
JSON CLI output utility
jc JSONifies the output of many CLI tools and file-types for easier parsing in scripts. See the Parsers section for supported commands and file-types.
This allows further command-line processing of output with tools like jq by piping commands:
ls -l /usr/bin | jc --ls | jq '.[] | select(.size > 50000000)'
{
"filename": "docker",
"flags": "-rwxr-xr-x",
"links": 1,
"owner": "root",
"group": "root",
"size": 68677120,
"date": "Aug 14 19:41"
}
or using the alternative "magic" syntax:
jc ls -l /usr/bin | jq '.[] | select(.size > 50000000)'
{
"filename": "docker",
"flags": "-rwxr-xr-x",
"links": 1,
"owner": "root",
"group": "root",
"size": 68677120,
"date": "Aug 14 19:41"
}
The jc parsers can also be used as python modules. In this case the output will be a python dictionary, or list of dictionaries, instead of JSON:
>>> import jc.parsers.ls
>>>
>>> data='''-rwxr-xr-x 1 root wheel 23648 May 3 22:26 cat
... -rwxr-xr-x 1 root wheel 30016 May 3 22:26 chmod
... -rwxr-xr-x 1 root wheel 29024 May 3 22:26 cp
... -rwxr-xr-x 1 root wheel 375824 May 3 22:26 csh
... -rwxr-xr-x 1 root wheel 28608 May 3 22:26 date
... -rwxr-xr-x 1 root wheel 32000 May 3 22:26 dd
... -rwxr-xr-x 1 root wheel 23392 May 3 22:26 df
... -rwxr-xr-x 1 root wheel 18128 May 3 22:26 echo'''
>>>
>>> jc.parsers.ls.parse(data)
[{'filename': 'cat', 'flags': '-rwxr-xr-x', 'links': 1, 'owner': 'root', 'group': 'wheel', 'size': 23648,
'date': 'May 3 22:26'}, {'filename': 'chmod', 'flags': '-rwxr-xr-x', 'links': 1, 'owner': 'root',
'group': 'wheel', 'size': 30016, 'date': 'May 3 22:26'}, {'filename': 'cp', 'flags': '-rwxr-xr-x',
'links': 1, 'owner': 'root', 'group': 'wheel', 'size': 29024, 'date': 'May 3 22:26'}, {'filename': 'csh',
'flags': '-rwxr-xr-x', 'links': 1, 'owner': 'root', 'group': 'wheel', 'size': 375824, 'date': 'May 3
22:26'}, {'filename': 'date', 'flags': '-rwxr-xr-x', 'links': 1, 'owner': 'root', 'group': 'wheel',
'size': 28608, 'date': 'May 3 22:26'}, {'filename': 'dd', 'flags': '-rwxr-xr-x', 'links': 1, 'owner':
'root', 'group': 'wheel', 'size': 32000, 'date': 'May 3 22:26'}, {'filename': 'df', 'flags':
'-rwxr-xr-x', 'links': 1, 'owner': 'root', 'group': 'wheel', 'size': 23392, 'date': 'May 3 22:26'},
{'filename': 'echo', 'flags': '-rwxr-xr-x', 'links': 1, 'owner': 'root', 'group': 'wheel', 'size': 18128,
'date': 'May 3 22:26'}]
Two representations of the data are possible. The default representation uses a strict schema per parser and converts known numbers to int/float JSON values. Certain known values of None are converted to JSON null, known boolean values are converted, and, in some cases, additional semantic context fields are added.
Note: Some parsers have calculated epoch timestamp fields added to the output. Unless a timestamp field name has a
_utcsuffix it is considered naive. (i.e. based on the local timezone of the system thejcparser was run on).If a UTC timezone can be detected in the text of the command output, the timestamp will be timezone aware and have a
_utcsuffix on the key name. (e.g.epoch_utc) No other timezones are supported for aware timestamps.
To access the raw, pre-processed JSON, use the -r cli option or the raw=True function parameter in parse().
Schemas for each parser can be found in the docs/parsers folder.
Release notes can be found here.
Why Would Anyone Do This!?
For more information on the motivations for this project, please see my blog post.
See also:
Installation
There are several ways to get jc. You can install via pip; other OS package repositories like apt-get, dnf, zypper, pacman, nix-env, guix, brew, or portsnap; via DEB/RPM packages; or by downloading the correct binary for your architecture and running it anywhere on your filesystem.
Pip (macOS, linux, unix, Windows)
pip3 install jc
OS Package Repositories
| OS | Command |
|---|---|
| Debian/Ubuntu linux | apt-get install jc |
| Fedora linux | dnf install jc |
| openSUSE linux | zypper install jc |
| Arch linux | pacman -S jc |
| NixOS linux | nix-env -iA nixpkgs.jc or nix-env -iA nixos.jc |
| Guix System linux | guix install jc |
| MacOS | brew install jc |
| FreeBSD | portsnap fetch update && cd /usr/ports/textproc/py-jc && make install clean |
| Ansible filter plugin | ansible-galaxy collection install community.general |
For more packages and binaries, see https://kellyjonbrazil.github.io/jc-packaging/.
Usage
jc accepts piped input from STDIN and outputs a JSON representation of the previous command's output to STDOUT.
COMMAND | jc PARSER [OPTIONS]
Alternatively, the "magic" syntax can be used by prepending jc to the command to be converted. Options can be passed to jc immediately before the command is given. (Note: command aliases and shell builtins are not supported)
jc [OPTIONS] COMMAND
The JSON output can be compact (default) or pretty formatted with the -p option.
Note: For best results set the
LANGlocale environment variable toC. For example, either by setting directly on the command-line:$ LANG=C date | jc --date, or by exporting to the environment before running commands:$ export LANG=C.
Parsers
--acpienables the acpi command parser--airportenables the airport -I command parser--airport-senables the airport -s command parser--arpenables the arp command parser--blkidenables the blkid command parser--cksumenables the cksum command parser--crontabenables the crontab command and file parser--crontab-uenables the crontab file parser with user support--csvenables the CSV file parser--dateenables the date command parser--dfenables the df command parser--digenables the dig command parser--direnables the dir command parser--dmidecodeenables the dmidecode command parser--dpkg-lenables the dpkg -l command parser--duenables the du command parser--envenables the env command parser--fileenables the file command parser--freeenables the free command parser--fstabenables the fstab file parser--groupenables the /etc/group file parser--gshadowenables the /etc/gshadow file parser--hashenables the hash command parser--hashsumenables the hashsum command parser (md5sum, shasum, etc.)--hciconfigenables the hciconfig command parser--historyenables the history command parser--hostsenables the /etc/hosts file parser--idenables the id command parser--ifconfigenables the ifconfig command parser--inienables the INI file parser--iptablesenables the iptables command parser--iw-scanenables the iw dev scan command parser--jobsenables the jobs command parser--kvenables the Key/Value file parser--lastenables the last and lastb command parser--lsenables the ls command parser--lsblkenables the lsblk command parser--lsmodenables the lsmod command parser--lsofenables the lsof command parser--mountenables the mount command parser--netstatenables the netstat command parser--ntpqenables the ntpq -p command parser--passwdenables the /etc/passwd file parser--pingenables the ping command parser--pip-listenables the pip list command parser--pip-showenables the pip show command parser--psenables the ps command parser--routeenables the route command parser--shadowenables the /etc/shadow file parser--ssenables the ss command parser--statenables the stat command parser--sysctlenables the sysctl command parser--systemctlenables the systemctl command parser--systemctl-ljenables the systemctl list-jobs command parser--systemctl-lsenables the systemctl list-sockets command parser--systemctl-lufenables the systemctl list-unit-files command parser--timeenables the /usr/bin/time command parser--timedatectlenables the timedatectl status command parser--tracepathenables the tracepath command parser--tracerouteenables the traceroute command parser--unameenables the uname -a command parser--upowerenables the upower command parser--uptimeenables the uptime command parser--wenables the w command parser--wcenables the wc command parser--whoenables the who command parser--xmlenables the XML file parser--yamlenables the YAML file parser
Options
-aaboutjc. Prints information aboutjcand the parsers (in JSON, of course!)-ddebug mode. Prints trace messages if parsing issues are encountered (use-ddfor verbose debugging)-hjchelp-mmonochrome JSON output-ppretty format the JSON output-qquiet mode. Suppresses parser warning messages-rraw output. Provides a more literal JSON output with all values as strings and no additional semantic processing-vversion information
Setting Custom Colors via Environment Variable
You can specify custom colors via the JC_COLORS environment variable. The JC_COLORS environment variable takes four comma separated string values in the following format:
JC_COLORS=<keyname_color>,<keyword_color>,<number_color>,<string_color>
Where colors are: black, red, green, yellow, blue, magenta, cyan, gray, brightblack, brightred, brightgreen, brightyellow, brightblue, brightmagenta, brightcyan, white, or default
For example, to set to the default colors:
JC_COLORS=blue,brightblack,magenta,green
or
JC_COLORS=default,default,default,default
Custom Parsers
Custom local parser plugins may be placed in a jc/jcparsers folder in your local "App data directory":
- Linux/unix:
$HOME/.local/share/jc/jcparsers - macOS:
$HOME/Library/Application Support/jc/jcparsers - Windows:
$LOCALAPPDATA\jc\jc\jcparsers
Local parser plugins are standard python module files. Use the jc/parsers/foo.py parser as a template and simply place a .py file in the jcparsers subfolder.
Local plugin filenames must be valid python module names, therefore must consist entirely of alphanumerics and start with a letter. Local plugins may override default plugins.
Note: The application data directory follows the XDG Base Directory Specification
Compatibility
Some parsers like ls, ps, dig, etc. will work on any platform. Other parsers that are platform-specific will generate a warning message if they are used on an unsupported platform. To see all parser information, including compatibility, run jc -ap.
You may still use a parser on an unsupported platform - for example, you may want to parse a file with linux lsof output on an macOS laptop. In that case you can suppress the warning message with the -q cli option or the quiet=True function parameter in parse():
cat lsof.out | jc --lsof -q
Tested on:
- Centos 7.7
- Ubuntu 18.04
- Ubuntu 20.04
- Fedora32
- macOS 10.11.6
- macOS 10.14.6
- NixOS
- FreeBSD12
- Windows 10
Contributions
Feel free to add/improve code or parsers! You can use the jc/parsers/foo.py parser as a template and submit your parser with a pull request.
Please see the Contributing Guidelines for more information.
Acknowledgments
- Local parser plugin feature contributed by Dean Serenevy
- CI automation and code optimizations by philippeitis
ifconfig-parsermodule by KnightWhoSayNixmltodictmodule by Martín Blechruamel.yamlmodule by Anthon van der Neuttrparsemodule by Luis Benitez- Parsing code from Conor Heine adapted for some parsers
- Excellent constructive feedback from Ilya Sher
Examples
Here are some examples of jc output. For more examples, see EXAMPLES.md or the parser documentation.
arp
arp | jc --arp -p # or: jc -p arp
[
{
"address": "gateway",
"hwtype": "ether",
"hwaddress": "00:50:56:f7:4a:fc",
"flags_mask": "C",
"iface": "ens33"
},
{
"address": "192.168.71.1",
"hwtype": "ether",
"hwaddress": "00:50:56:c0:00:08",
"flags_mask": "C",
"iface": "ens33"
},
{
"address": "192.168.71.254",
"hwtype": "ether",
"hwaddress": "00:50:56:fe:7a:b4",
"flags_mask": "C",
"iface": "ens33"
}
]
CSV files
cat homes.csv
"Sell", "List", "Living", "Rooms", "Beds", "Baths", "Age", "Acres", "Taxes"
142, 160, 28, 10, 5, 3, 60, 0.28, 3167
175, 180, 18, 8, 4, 1, 12, 0.43, 4033
129, 132, 13, 6, 3, 1, 41, 0.33, 1471
...
cat homes.csv | jc --csv -p
[
{
"Sell": "142",
"List": "160",
"Living": "28",
"Rooms": "10",
"Beds": "5",
"Baths": "3",
"Age": "60",
"Acres": "0.28",
"Taxes": "3167"
},
{
"Sell": "175",
"List": "180",
"Living": "18",
"Rooms": "8",
"Beds": "4",
"Baths": "1",
"Age": "12",
"Acres": "0.43",
"Taxes": "4033"
},
{
"Sell": "129",
"List": "132",
"Living": "13",
"Rooms": "6",
"Beds": "3",
"Baths": "1",
"Age": "41",
"Acres": "0.33",
"Taxes": "1471"
}
]
dig
dig cnn.com @205.251.194.64 | jc --dig -p # or: jc -p dig cnn.com @205.251.194.64
[
{
"id": 52172,
"opcode": "QUERY",
"status": "NOERROR",
"flags": [
"qr",
"rd",
"ra"
],
"query_num": 1,
"answer_num": 1,
"authority_num": 0,
"additional_num": 1,
"question": {
"name": "cnn.com.",
"class": "IN",
"type": "A"
},
"answer": [
{
"name": "cnn.com.",
"class": "IN",
"type": "A",
"ttl": 27,
"data": "151.101.65.67"
}
],
"query_time": 38,
"server": "2600",
"when": "Tue Mar 30 20:07:59 PDT 2021",
"rcvd": 100,
"when_epoch": 1617160079,
"when_epoch_utc": null
}
]
/etc/hosts file
cat /etc/hosts | jc --hosts -p
[
{
"ip": "127.0.0.1",
"hostname": [
"localhost"
]
},
{
"ip": "::1",
"hostname": [
"ip6-localhost",
"ip6-loopback"
]
},
{
"ip": "fe00::0",
"hostname": [
"ip6-localnet"
]
}
]
ifconfig
ifconfig | jc --ifconfig -p # or: jc -p ifconfig
[
{
"name": "ens33",
"flags": 4163,
"state": [
"UP",
"BROADCAST",
"RUNNING",
"MULTICAST"
],
"mtu": 1500,
"ipv4_addr": "192.168.71.137",
"ipv4_mask": "255.255.255.0",
"ipv4_bcast": "192.168.71.255",
"ipv6_addr": "fe80::c1cb:715d:bc3e:b8a0",
"ipv6_mask": 64,
"ipv6_scope": "0x20",
"mac_addr": "00:0c:29:3b:58:0e",
"type": "Ethernet",
"rx_packets": 8061,
"rx_bytes": 1514413,
"rx_errors": 0,
"rx_dropped": 0,
"rx_overruns": 0,
"rx_frame": 0,
"tx_packets": 4502,
"tx_bytes": 866622,
"tx_errors": 0,
"tx_dropped": 0,
"tx_overruns": 0,
"tx_carrier": 0,
"tx_collisions": 0,
"metric": null
}
]
INI files
cat example.ini
[DEFAULT]
ServerAliveInterval = 45
Compression = yes
CompressionLevel = 9
ForwardX11 = yes
[bitbucket.org]
User = hg
[topsecret.server.com]
Port = 50022
ForwardX11 = no
cat example.ini | jc --ini -p
{
"bitbucket.org": {
"serveraliveinterval": "45",
"compression": "yes",
"compressionlevel": "9",
"forwardx11": "yes",
"user": "hg"
},
"topsecret.server.com": {
"serveraliveinterval": "45",
"compression": "yes",
"compressionlevel": "9",
"forwardx11": "no",
"port": "50022"
}
}
ls
$ ls -l /usr/bin | jc --ls -p # or: jc -p ls -l /usr/bin
[
{
"filename": "apropos",
"link_to": "whatis",
"flags": "lrwxrwxrwx.",
"links": 1,
"owner": "root",
"group": "root",
"size": 6,
"date": "Aug 15 10:53"
},
{
"filename": "ar",
"flags": "-rwxr-xr-x.",
"links": 1,
"owner": "root",
"group": "root",
"size": 62744,
"date": "Aug 8 16:14"
},
{
"filename": "arch",
"flags": "-rwxr-xr-x.",
"links": 1,
"owner": "root",
"group": "root",
"size": 33080,
"date": "Aug 19 23:25"
}
]
netstat
netstat -apee | jc --netstat -p # or: jc -p netstat -apee
[
{
"proto": "tcp",
"recv_q": 0,
"send_q": 0,
"local_address": "localhost",
"foreign_address": "0.0.0.0",
"state": "LISTEN",
"user": "systemd-resolve",
"inode": 26958,
"program_name": "systemd-resolve",
"kind": "network",
"pid": 887,
"local_port": "domain",
"foreign_port": "*",
"transport_protocol": "tcp",
"network_protocol": "ipv4"
},
{
"proto": "tcp6",
"recv_q": 0,
"send_q": 0,
"local_address": "[::]",
"foreign_address": "[::]",
"state": "LISTEN",
"user": "root",
"inode": 30510,
"program_name": "sshd",
"kind": "network",
"pid": 1186,
"local_port": "ssh",
"foreign_port": "*",
"transport_protocol": "tcp",
"network_protocol": "ipv6"
},
{
"proto": "udp",
"recv_q": 0,
"send_q": 0,
"local_address": "localhost",
"foreign_address": "0.0.0.0",
"state": null,
"user": "systemd-resolve",
"inode": 26957,
"program_name": "systemd-resolve",
"kind": "network",
"pid": 887,
"local_port": "domain",
"foreign_port": "*",
"transport_protocol": "udp",
"network_protocol": "ipv4"
},
{
"proto": "raw6",
"recv_q": 0,
"send_q": 0,
"local_address": "[::]",
"foreign_address": "[::]",
"state": "7",
"user": "systemd-network",
"inode": 27001,
"program_name": "systemd-network",
"kind": "network",
"pid": 867,
"local_port": "ipv6-icmp",
"foreign_port": "*",
"transport_protocol": null,
"network_protocol": "ipv6"
},
{
"proto": "unix",
"refcnt": 2,
"flags": null,
"type": "DGRAM",
"state": null,
"inode": 33322,
"program_name": "systemd",
"path": "/run/user/1000/systemd/notify",
"kind": "socket",
"pid": 1607
}
]
/etc/passwd file
cat /etc/passwd | jc --passwd -p
[
{
"username": "root",
"password": "*",
"uid": 0,
"gid": 0,
"comment": "System Administrator",
"home": "/var/root",
"shell": "/bin/sh"
},
{
"username": "daemon",
"password": "*",
"uid": 1,
"gid": 1,
"comment": "System Services",
"home": "/var/root",
"shell": "/usr/bin/false"
}
]
ping
ping 8.8.8.8 -c 3 | jc --ping -p # or: jc -p ping 8.8.8.8 -c 3
{
"destination_ip": "8.8.8.8",
"data_bytes": 56,
"pattern": null,
"destination": "8.8.8.8",
"packets_transmitted": 3,
"packets_received": 3,
"packet_loss_percent": 0.0,
"duplicates": 0,
"time_ms": 2005.0,
"round_trip_ms_min": 23.835,
"round_trip_ms_avg": 30.46,
"round_trip_ms_max": 34.838,
"round_trip_ms_stddev": 4.766,
"responses": [
{
"type": "reply",
"timestamp": null,
"bytes": 64,
"response_ip": "8.8.8.8",
"icmp_seq": 1,
"ttl": 118,
"time_ms": 23.8,
"duplicate": false
},
{
"type": "reply",
"timestamp": null,
"bytes": 64,
"response_ip": "8.8.8.8",
"icmp_seq": 2,
"ttl": 118,
"time_ms": 34.8,
"duplicate": false
},
{
"type": "reply",
"timestamp": null,
"bytes": 64,
"response_ip": "8.8.8.8",
"icmp_seq": 3,
"ttl": 118,
"time_ms": 32.7,
"duplicate": false
}
]
}
ps
ps axu | jc --ps -p # or: jc -p ps axu
[
{
"user": "root",
"pid": 1,
"cpu_percent": 0.0,
"mem_percent": 0.1,
"vsz": 128072,
"rss": 6784,
"tty": null,
"stat": "Ss",
"start": "Nov09",
"time": "0:08",
"command": "/usr/lib/systemd/systemd --switched-root --system --deserialize 22"
},
{
"user": "root",
"pid": 2,
"cpu_percent": 0.0,
"mem_percent": 0.0,
"vsz": 0,
"rss": 0,
"tty": null,
"stat": "S",
"start": "Nov09",
"time": "0:00",
"command": "[kthreadd]"
},
{
"user": "root",
"pid": 4,
"cpu_percent": 0.0,
"mem_percent": 0.0,
"vsz": 0,
"rss": 0,
"tty": null,
"stat": "S<",
"start": "Nov09",
"time": "0:00",
"command": "[kworker/0:0H]"
}
]
traceroute
traceroute -m 2 8.8.8.8 | jc --traceroute -p # or: jc -p traceroute -m 2 8.8.8.8
{
"destination_ip": "8.8.8.8",
"destination_name": "8.8.8.8",
"hops": [
{
"hop": 1,
"probes": [
{
"annotation": null,
"asn": null,
"ip": "192.168.1.254",
"name": "dsldevice.local.net",
"rtt": 6.616
},
{
"annotation": null,
"asn": null,
"ip": "192.168.1.254",
"name": "dsldevice.local.net",
"rtt": 6.413
},
{
"annotation": null,
"asn": null,
"ip": "192.168.1.254",
"name": "dsldevice.local.net",
"rtt": 6.308
}
]
},
{
"hop": 2,
"probes": [
{
"annotation": null,
"asn": null,
"ip": "76.220.24.1",
"name": "76-220-24-1.lightspeed.sntcca.sbcglobal.net",
"rtt": 29.367
},
{
"annotation": null,
"asn": null,
"ip": "76.220.24.1",
"name": "76-220-24-1.lightspeed.sntcca.sbcglobal.net",
"rtt": 40.197
},
{
"annotation": null,
"asn": null,
"ip": "76.220.24.1",
"name": "76-220-24-1.lightspeed.sntcca.sbcglobal.net",
"rtt": 29.162
}
]
}
]
}
uptime
uptime | jc --uptime -p # or: jc -p uptime
{
"time": "11:35",
"uptime": "3 days, 4:03",
"users": 5,
"load_1m": 1.88,
"load_5m": 2.0,
"load_15m": 1.94,
"time_hour": 11,
"time_minute": 35,
"time_second": null,
"uptime_days": 3,
"uptime_hours": 4,
"uptime_minutes": 3,
"uptime_total_seconds": 273780
}
XML files
cat cd_catalog.xml
<?xml version="1.0" encoding="UTF-8"?>
<CATALOG>
<CD>
<TITLE>Empire Burlesque</TITLE>
<ARTIST>Bob Dylan</ARTIST>
<COUNTRY>USA</COUNTRY>
<COMPANY>Columbia</COMPANY>
<PRICE>10.90</PRICE>
<YEAR>1985</YEAR>
</CD>
<CD>
<TITLE>Hide your heart</TITLE>
<ARTIST>Bonnie Tyler</ARTIST>
<COUNTRY>UK</COUNTRY>
<COMPANY>CBS Records</COMPANY>
<PRICE>9.90</PRICE>
<YEAR>1988</YEAR>
</CD>
...
cat cd_catalog.xml | jc --xml -p
{
"CATALOG": {
"CD": [
{
"TITLE": "Empire Burlesque",
"ARTIST": "Bob Dylan",
"COUNTRY": "USA",
"COMPANY": "Columbia",
"PRICE": "10.90",
"YEAR": "1985"
},
{
"TITLE": "Hide your heart",
"ARTIST": "Bonnie Tyler",
"COUNTRY": "UK",
"COMPANY": "CBS Records",
"PRICE": "9.90",
"YEAR": "1988"
}
]
}
}
YAML files
cat istio.yaml
apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
name: "default"
namespace: "default"
spec:
peers:
- mtls: {}
---
apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
name: "default"
namespace: "default"
spec:
host: "*.default.svc.cluster.local"
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
cat istio.yaml | jc --yaml -p
[
{
"apiVersion": "authentication.istio.io/v1alpha1",
"kind": "Policy",
"metadata": {
"name": "default",
"namespace": "default"
},
"spec": {
"peers": [
{
"mtls": {}
}
]
}
},
{
"apiVersion": "networking.istio.io/v1alpha3",
"kind": "DestinationRule",
"metadata": {
"name": "default",
"namespace": "default"
},
"spec": {
"host": "*.default.svc.cluster.local",
"trafficPolicy": {
"tls": {
"mode": "ISTIO_MUTUAL"
}
}
}
}
]
© 2019-2021 Kelly Brazil