From d64c4cb39056bb869fc117477612ae5ae2ef3d85 Mon Sep 17 00:00:00 2001 From: Kelly Brazil Date: Mon, 20 Sep 2021 13:04:26 -0700 Subject: [PATCH] add streaming parsers section --- man/jc.1 | 72 +++++++++++++++++++++++++++++++++++++- templates/manpage_template | 70 ++++++++++++++++++++++++++++++++++++ 2 files changed, 141 insertions(+), 1 deletion(-) diff --git a/man/jc.1 b/man/jc.1 index 5543eeb9..b96cbf98 100644 --- a/man/jc.1 +++ b/man/jc.1 @@ -1,4 +1,4 @@ -.TH jc 1 2021-09-17 1.17.0 "JSON CLI output utility" +.TH jc 1 2021-09-20 1.17.0 "JSON CLI output utility" .SH NAME jc \- JSONifies the output of many CLI tools and file-types .SH SYNOPSIS @@ -473,6 +473,76 @@ or JC_COLORS=default,default,default,default .RE + +.SH STREAMING PARSERS +Most parsers load all of the data from \fBSTDIN\fP, parse it, then output the entire JSON document serially. There are some streaming parsers (e.g. \fBls-s\fP and \fBping-s\fP) that immediately start processing and outputing the data line-by-line as JSON Lines (aka NDJSON) while it is being received from \fBSTDIN\fP. This can significantly reduce the amount of memory required to parse large amounts of command output (e.g. \fBls -lR /\fP) and can sometimes process the data more quickly. Streaming parsers have slightly different behavior than standard parsers as outlined below. + +.RS +Note: Streaming parsers cannot be used with the "magic" syntax +.RE + +\fBIgnoring Errors\fP + +You may want to ignore parsing errors when using streaming parsers since these may be used in long-lived processing pipelines and errors can break the pipe. To ignore parsing errors, use the \fB-q\fP cli option or the \fBquiet=True\fP argument with the \fBparse()\fP function. This will add a \fB_meta\fP object to the JSON output with a \fBsuccess\fP attribute. If \fBsuccess\fP is \fBtrue\fP, then there were no issues parsing the line. If \fBsuccess\fP is \fBfalse\fP, then a parsing issue was found and \fBerror\fP and \fBline\fP fields will be added to include a short error description and the contents of the unparsable line, respectively: + +.RS +Successfully parsed line with \fB-q\fP option: +.RS +{ + + "command_data": "data", + + "_meta": { + + "success": true + + } + +} +.RE + +Unsuccessfully parsed line with \fB-q\fP option: +.RS +{ + + "_meta": { + + "success": false, + + "error": "error parsing line", + + "line": "original line data" + + } + +} +.RE + +.RE +\fBUnbuffering Output\fP + +Most operating systems will buffer output that is being piped from process to process. The buffer is usually around 4KB. When viewing the output in the terminal the OS buffer is not engaged so output is immediately displayed on the screen. When piping multiple processes together, though, it may seem as if the output is hanging when the input data is very slow (e.g. \fBping\fP): + +.RS +$ ping 1.1.1.1 | jc --ping-s | jq + + +.RE + +This is because the OS engages the 4KB buffer between \fBjc\fP and \fBjq\fP in this example. To display the data on the terminal in realtime, you can disable the buffer with the \fB-u\fP (unbuffer) cli option: + +.RS +$ ping 1.1.1.1 | jc --ping-s -u | jq + +{"type":"reply","pattern":null,"timestamp":null,"bytes":"64","response_ip":"1.1.1.1","icmp_seq":"1","ttl":"128","time_ms":"24.6","duplicate":false} + +{"type":"reply","pattern":null,"timestamp":null,"bytes":"64","response_ip":"1.1.1.1","icmp_seq":"2","ttl":"128","time_ms":"26.8","duplicate":false} + +etc... + +Note: Unbuffered output can be slower for faster data streams. +.RE + .SH CUSTOM PARSERS Custom local parser plugins may be placed in a \fBjc/jcparsers\fP folder in your local "App data directory": diff --git a/templates/manpage_template b/templates/manpage_template index f99ed411..12dbfc9a 100644 --- a/templates/manpage_template +++ b/templates/manpage_template @@ -98,6 +98,76 @@ or JC_COLORS=default,default,default,default .RE + +.SH STREAMING PARSERS +Most parsers load all of the data from \fBSTDIN\fP, parse it, then output the entire JSON document serially. There are some streaming parsers (e.g. \fBls-s\fP and \fBping-s\fP) that immediately start processing and outputing the data line-by-line as JSON Lines (aka NDJSON) while it is being received from \fBSTDIN\fP. This can significantly reduce the amount of memory required to parse large amounts of command output (e.g. \fBls -lR /\fP) and can sometimes process the data more quickly. Streaming parsers have slightly different behavior than standard parsers as outlined below. + +.RS +Note: Streaming parsers cannot be used with the "magic" syntax +.RE + +\fBIgnoring Errors\fP + +You may want to ignore parsing errors when using streaming parsers since these may be used in long-lived processing pipelines and errors can break the pipe. To ignore parsing errors, use the \fB-q\fP cli option or the \fBquiet=True\fP argument with the \fBparse()\fP function. This will add a \fB_meta\fP object to the JSON output with a \fBsuccess\fP attribute. If \fBsuccess\fP is \fBtrue\fP, then there were no issues parsing the line. If \fBsuccess\fP is \fBfalse\fP, then a parsing issue was found and \fBerror\fP and \fBline\fP fields will be added to include a short error description and the contents of the unparsable line, respectively: + +.RS +Successfully parsed line with \fB-q\fP option: +.RS +{ + + "command_data": "data", + + "_meta": { + + "success": true + + } + +} +.RE + +Unsuccessfully parsed line with \fB-q\fP option: +.RS +{ + + "_meta": { + + "success": false, + + "error": "error parsing line", + + "line": "original line data" + + } + +} +.RE + +.RE +\fBUnbuffering Output\fP + +Most operating systems will buffer output that is being piped from process to process. The buffer is usually around 4KB. When viewing the output in the terminal the OS buffer is not engaged so output is immediately displayed on the screen. When piping multiple processes together, though, it may seem as if the output is hanging when the input data is very slow (e.g. \fBping\fP): + +.RS +$ ping 1.1.1.1 | jc --ping-s | jq + + +.RE + +This is because the OS engages the 4KB buffer between \fBjc\fP and \fBjq\fP in this example. To display the data on the terminal in realtime, you can disable the buffer with the \fB-u\fP (unbuffer) cli option: + +.RS +$ ping 1.1.1.1 | jc --ping-s -u | jq + +{"type":"reply","pattern":null,"timestamp":null,"bytes":"64","response_ip":"1.1.1.1","icmp_seq":"1","ttl":"128","time_ms":"24.6","duplicate":false} + +{"type":"reply","pattern":null,"timestamp":null,"bytes":"64","response_ip":"1.1.1.1","icmp_seq":"2","ttl":"128","time_ms":"26.8","duplicate":false} + +etc... + +Note: Unbuffered output can be slower for faster data streams. +.RE + .SH CUSTOM PARSERS Custom local parser plugins may be placed in a \fBjc/jcparsers\fP folder in your local "App data directory":