1
0
mirror of https://github.com/kellyjonbrazil/jc.git synced 2026-04-03 17:44:07 +02:00

Compare commits

..

215 Commits

Author SHA1 Message Date
Kelly Brazil
bcff00799f Merge pull request #328 from kellyjonbrazil/dev
Dev v1.22.3
2022-12-16 15:31:16 -06:00
Kelly Brazil
0dc76621bc doc update 2022-12-16 12:54:07 -08:00
Kelly Brazil
1040c5706f split prefix and ports. remove utc timestamps 2022-12-14 11:55:46 -08:00
Kelly Brazil
265d08e3bd doc update 2022-12-13 17:32:12 -08:00
Kelly Brazil
491fce7052 add openvpn-status.log file parser 2022-12-13 17:31:05 -08:00
Kelly Brazil
3404bc4840 add pgpass tests 2022-12-13 14:57:22 -08:00
Kelly Brazil
064e3f6ac0 try changing test dates for windows compatibility 2022-12-13 14:11:12 -08:00
Kelly Brazil
71f0c4f9da doc update 2022-12-13 13:46:33 -08:00
Kelly Brazil
18143608dc doc update 2022-12-13 13:30:50 -08:00
Kelly Brazil
83a6d92449 fix tests for additional timestamps 2022-12-13 13:30:31 -08:00
Kelly Brazil
ac1f690c54 add epoch timestamps 2022-12-13 13:29:54 -08:00
Kelly Brazil
b063f1bfb4 add google big table time format 2022-12-13 13:10:34 -08:00
Kelly Brazil
4c5caa7b86 add iso attribute to timestamp 2022-12-13 12:47:31 -08:00
Kelly Brazil
85edda6e5f Merge branch 'dev' of https://github.com/kellyjonbrazil/jc into dev 2022-12-13 12:15:28 -08:00
Kelly Brazil
6e1a4a103c formatting and typing fixes 2022-12-13 12:13:27 -08:00
Kelly Brazil
2879c084e5 doc update 2022-12-13 09:01:13 -08:00
Kelly Brazil
4ecc94e531 formatting and typing fixes 2022-12-13 09:01:04 -08:00
Kelly Brazil
b8ef583b93 Merge pull request #327 from graipher/master
Add parser for cbt
2022-12-13 10:40:46 -06:00
Andreas Weiden
fd61e19135 Add raw schema 2022-12-12 19:48:33 +01:00
Andreas Weiden
79011465af Fix docstring 2022-12-12 19:46:45 +01:00
Andreas Weiden
27acedf8b7 Add newlines at end of files 2022-12-12 15:43:27 +01:00
Andreas Weiden
532c37140c Move test stuff to fixtures 2022-12-12 15:33:40 +01:00
Andreas Weiden
7f4a951065 Add parser to table in readme 2022-12-12 15:18:39 +01:00
Andreas Weiden
b9fb7fad9c Add parser for cbt 2022-12-12 15:10:59 +01:00
Kelly Brazil
be85d78f55 doc update 2022-12-05 14:54:23 -08:00
Kelly Brazil
23e02090e0 add pgpass parser 2022-12-05 14:43:48 -08:00
Kelly Brazil
50f2a811ad doc update 2022-12-05 12:43:45 -08:00
Kelly Brazil
28ee448c44 remove unneeded type ignore comments 2022-12-05 12:42:55 -08:00
Kelly Brazil
688a2099b5 relax JSONDictType - unions and Dicts cause a lot of friction 2022-12-05 12:15:35 -08:00
Kelly Brazil
6f0a53ed02 fix doc title 2022-12-02 16:10:00 -08:00
Kelly Brazil
8a14de663e doc update 2022-12-02 15:16:46 -08:00
Kelly Brazil
09344e938a add lane info lines 2022-12-02 15:16:07 -08:00
Kelly Brazil
bf41140322 fix tests for missing ipv6 addresses 2022-12-02 11:16:13 -08:00
Kelly Brazil
689a85db9b fix for missing ipv6 addresses and scope_id 2022-12-02 11:09:17 -08:00
Kelly Brazil
5e0d206e7a minor cleanup 2022-11-22 13:18:08 -08:00
Kelly Brazil
61addd7950 doc update 2022-11-22 13:12:40 -08:00
Kelly Brazil
60bb9e2aa9 doc update 2022-11-22 13:11:41 -08:00
Kelly Brazil
842fbbab64 formatting 2022-11-22 13:11:10 -08:00
Kelly Brazil
bbd227caf4 tighten up blank line checking 2022-11-22 13:10:58 -08:00
Kelly Brazil
975b4f5e4f add clf-s parser tests 2022-11-22 13:10:15 -08:00
Kelly Brazil
06840931ba doc update 2022-11-21 16:54:42 -08:00
Kelly Brazil
9f4327f517 add clf-s streaming parser 2022-11-21 16:54:13 -08:00
Kelly Brazil
3f13c70dfa formatting 2022-11-21 12:12:16 -08:00
Kelly Brazil
26f8803b23 add support for unparsable lines 2022-11-21 12:09:19 -08:00
Kelly Brazil
60f1e79b2f fix clf request string parsing and add tests 2022-11-21 11:00:58 -08:00
Kelly Brazil
5ab2ebe45a add CLF timestamp support 2022-11-21 09:27:37 -08:00
Kelly Brazil
9c8fe80d6d add more processing and timestamp 2022-11-21 09:27:21 -08:00
Kelly Brazil
1e7e22330f doc update 2022-11-20 20:45:22 -08:00
Kelly Brazil
7244868fbd initial common log format parser 2022-11-20 20:43:49 -08:00
Kelly Brazil
86ed39ecdd tighten up efi split 2022-11-18 16:19:55 -08:00
Kelly Brazil
94d87b726f add efi partition split 2022-11-18 14:20:18 -08:00
Kelly Brazil
0984a1ec26 fix git-log tests and docs 2022-11-18 13:54:59 -08:00
Kelly Brazil
de5da060ce Merge pull request #322 from adamwolf/git-log-blank-author
Fix git log parsing with empty name or email
2022-11-18 13:18:06 -06:00
Kelly Brazil
592435572c Merge branch 'dev' of https://github.com/kellyjonbrazil/jc into dev
add in PR 323
2022-11-18 11:11:57 -08:00
Kelly Brazil
3172a18a46 Merge pull request #323 from kianmeng/fix-typos
Fix typos
2022-11-18 13:05:36 -06:00
Kelly Brazil
7f1c57b89c version bump 2022-11-17 13:39:59 -06:00
Kian-Meng Ang
39555a48b5 Fix typos
Found via `codespell -S ./tests/fixtures -L
chage,ro,ist,ans,unx,respons,technik`
2022-11-16 10:01:58 +08:00
Adam Wolf
e4cdfa13ca Fix git log parsing with empty name or email
Sometimes, folks leave their name or email blank in on their
git commits.  Previously, a blank name crashed the git log
parser.
2022-11-11 13:33:01 -06:00
Kelly Brazil
cb4011bc03 formatting 2022-11-08 14:23:32 -08:00
Kelly Brazil
299b0faf7c Merge pull request #319 from kellyjonbrazil/dev
Dev v1.22.2
2022-11-08 16:32:03 +00:00
Kelly Brazil
2ffd698c03 update du docs 2022-11-08 08:13:41 -08:00
Kelly Brazil
dde54690fc Merge pull request #318 from kellyjonbrazil/master
sync to dev
2022-11-08 16:07:56 +00:00
Kelly Brazil
8d03055b34 add tests 2022-11-07 16:41:13 -08:00
Kelly Brazil
8a850be857 doc update 2022-11-07 13:26:16 -08:00
Kelly Brazil
2a530712cf add os-prober parser 2022-11-07 13:23:55 -08:00
Kelly Brazil
fd22c7dc3a add git-ls-remote parser 2022-11-07 13:02:55 -08:00
Kelly Brazil
b884f6aacc doc update 2022-11-07 10:44:37 -08:00
Kelly Brazil
ce680d4082 add cidr-style freebsd ipv4 support and freebsd options 2022-11-07 10:40:07 -08:00
Kelly Brazil
644d3f350d add more bsd tests 2022-11-06 12:12:07 -08:00
Kelly Brazil
0144863d33 new ifconfig parser with additional fields. tests passing 2022-11-06 12:04:34 -08:00
Kelly Brazil
a6b56519a2 add more bsd fields to ifconfig 2022-11-05 18:47:44 -07:00
Kelly Brazil
0a71caf9cd formatting for regexes 2022-11-05 17:12:50 -07:00
Kelly Brazil
35e74328c4 tests passing for ifconfig 2022-11-05 16:36:31 -07:00
Kelly Brazil
e46ac0ff7e initial commit of new ifconfig parser 2022-11-05 12:13:01 -07:00
Kelly Brazil
17fe6c7691 semver tests and doc update 2022-11-04 16:03:39 -07:00
Kelly Brazil
01ca7a69c4 formatting 2022-11-04 15:24:34 -07:00
Kelly Brazil
6d04db6113 add semver parser 2022-11-04 15:17:47 -07:00
Kelly Brazil
7c899abb15 add support for multiple include paths 2022-11-04 14:42:04 -07:00
Kelly Brazil
89d4df2a05 document ifconfig limitations and recommend using ip 2022-11-04 09:25:17 -07:00
Kelly Brazil
71c8364f80 doc fix 2022-11-04 09:19:42 -07:00
Kelly Brazil
214cd6b9e0 Merge pull request #310 from villesinisalo/readme_arch
Readme: recommend plain Pacman on Arch
2022-11-03 15:34:19 +00:00
Ville Sinisalo
73c280de3a use plain Pacman on Arch 2022-11-03 11:29:06 +02:00
Kelly Brazil
23e1dd3e35 use dict constructor for xmltodict to suppress !!omap comments in YAML output 2022-11-02 12:00:45 -07:00
Kelly Brazil
2b060aae0d clean up raw/processed logic 2022-11-01 19:53:44 -07:00
Kelly Brazil
186ad73651 add raw option to xml parser for _ attribute prefix 2022-11-01 18:09:05 -07:00
Kelly Brazil
de7a010f62 update test templates 2022-11-01 17:01:21 -07:00
Kelly Brazil
ac1bcd2918 doc update 2022-11-01 15:28:50 -07:00
Kelly Brazil
a2e6243282 doc update and additional tests 2022-11-01 13:50:01 -07:00
Kelly Brazil
01f92ced81 add docs and tests for findmnt 2022-11-01 13:15:31 -07:00
Kelly Brazil
b493bcf4fa update type annotations 2022-10-31 17:37:01 -07:00
Kelly Brazil
f6ee30be20 formatting 2022-10-31 17:30:51 -07:00
Kelly Brazil
50da124ea7 initial findmnt parser 2022-10-31 17:22:56 -07:00
Kelly Brazil
5e22f9e2bd formatting 2022-10-31 10:29:43 -07:00
Kelly Brazil
a384eb4c15 add sshd_conf tests 2022-10-31 09:32:58 -07:00
Kelly Brazil
dc4620eeb2 doc update 2022-10-31 09:22:39 -07:00
Kelly Brazil
d7cfa38eee ignore Match blocks 2022-10-28 16:36:52 -07:00
Kelly Brazil
a27110ebe5 formatting 2022-10-28 15:25:55 -07:00
Kelly Brazil
f5988527fb add sshd_conf parser 2022-10-28 15:15:02 -07:00
Kelly Brazil
c405309742 allow debug of exceptions 2022-10-28 15:14:53 -07:00
Kelly Brazil
747d12224f allow parser_info and get_help to use module objects as input 2022-10-28 10:36:54 -07:00
Kelly Brazil
2b621ab68e use exceptions instead of signal handlers to catch piperror and sigint 2022-10-28 10:11:19 -07:00
Kelly Brazil
8689865d31 fix exit on interrupt exit code. also add message to stderr 2022-10-27 11:23:43 -07:00
Kelly Brazil
358324533d doc update 2022-10-26 15:39:32 -07:00
Kelly Brazil
bc5821e69f force ci tests 2022-10-26 15:36:22 -07:00
Kelly Brazil
d5b478c968 bump checkout and setup-python versions 2022-10-26 15:35:41 -07:00
Kelly Brazil
368eba1826 bump checkout and setup-python versions 2022-10-26 15:34:46 -07:00
Kelly Brazil
6cffb449f4 Merge branch 'dev' of https://github.com/kellyjonbrazil/jc into dev 2022-10-26 15:23:35 -07:00
Kelly Brazil
d79d9c7f13 simplify return 2022-10-26 15:21:37 -07:00
Kelly Brazil
179822b994 add python 3.11 2022-10-26 15:18:33 -07:00
Kelly Brazil
ba369a0b73 add python 3.11 2022-10-26 15:18:04 -07:00
Kelly Brazil
6a5251f0ef doc update 2022-10-25 15:50:45 -07:00
Kelly Brazil
004fd74748 add type annotations 2022-10-25 15:21:42 -07:00
Kelly Brazil
e8d6d4c080 relax input_type_check to allow object argument 2022-10-25 15:21:26 -07:00
Kelly Brazil
8644f70db4 fix typos 2022-10-25 13:46:22 -07:00
Kelly Brazil
72f233b186 version bump 2022-10-25 11:53:08 -07:00
Kelly Brazil
fc85950a73 doc update 2022-10-25 11:49:02 -07:00
Kelly Brazil
fd5cbbb4d5 add csv utf-8 bom tests 2022-10-25 11:46:31 -07:00
Kelly Brazil
888b6bd6d5 fix for UTF-8 csv files with leading BOM bytes 2022-10-25 11:18:22 -07:00
Kelly Brazil
81ea83064c Merge pull request #306 from kellyjonbrazil/dev
Dev v1.22.1
2022-10-24 16:30:36 +00:00
Kelly Brazil
45859b01e5 doc update 2022-10-24 09:17:09 -07:00
Kelly Brazil
5b8f166169 doc update 2022-10-24 09:15:58 -07:00
Kelly Brazil
7e045ba7b0 doc update 2022-10-24 09:14:01 -07:00
Kelly Brazil
6247975ee6 update jc-web link to render 2022-10-23 13:20:50 -07:00
Kelly Brazil
297451230c lspci docs and tests 2022-10-23 09:15:23 -07:00
Kelly Brazil
754d555768 doc update 2022-10-21 13:09:13 -07:00
Kelly Brazil
b61a91f82d add pci-ids and udevadm tests 2022-10-21 12:57:28 -07:00
Kelly Brazil
1ac7a724bd integer conversions 2022-10-21 12:45:19 -07:00
Kelly Brazil
2fcf46505c add schema, typing, and doc update 2022-10-21 12:44:57 -07:00
Kelly Brazil
fdec4f08c0 add schema docs for pci-ids 2022-10-21 11:50:06 -07:00
Kelly Brazil
076c197385 more explicit type annotations 2022-10-21 10:56:10 -07:00
Kelly Brazil
3432e830f2 doc update 2022-10-19 13:46:32 -07:00
Kelly Brazil
14a237749c type annotation cleanup 2022-10-19 13:32:16 -07:00
Kelly Brazil
1559c4751e type annotate decorator function 2022-10-19 12:02:12 -07:00
Kelly Brazil
af7c4ce8ec doc update 2022-10-19 08:33:09 -07:00
Kelly Brazil
5763ce6160 Don't get too fancy with dict type annotations 2022-10-19 08:30:09 -07:00
Kelly Brazil
d7ca6caae8 use jc_types 2022-10-18 15:46:29 -07:00
Kelly Brazil
2c4f232aaa use jc_types 2022-10-18 15:45:29 -07:00
Kelly Brazil
bddfa71fa6 use object for typing since these functions accept other types 2022-10-18 15:44:53 -07:00
Kelly Brazil
f537ab058b use Tuple for <3.9 compatibility 2022-10-18 11:27:12 -07:00
Kelly Brazil
1b44b5b05a use typing Tuple for <3.9 compatibility 2022-10-18 11:24:55 -07:00
Kelly Brazil
f6e971c652 fix metadata return type 2022-10-18 11:04:08 -07:00
Kelly Brazil
dfa7a71f53 move custom types to jc_types 2022-10-18 11:01:59 -07:00
Kelly Brazil
3639e02843 doc update 2022-10-18 09:37:10 -07:00
Kelly Brazil
27b6e79c8a ignore mocking typing errors 2022-10-18 09:36:17 -07:00
Kelly Brazil
f370151b54 make iso_datetime deprecated only. no need to make hidden. 2022-10-18 09:35:59 -07:00
Kelly Brazil
597a8f468e normalize show_hidden and show_deprecated to be False by default 2022-10-18 09:35:24 -07:00
Kelly Brazil
8bd7a00410 add py.typed to package 2022-10-18 09:14:43 -07:00
Kelly Brazil
4867052972 fix AboutJCType 2022-10-15 18:27:55 -07:00
Kelly Brazil
688c3a34f6 define CustomColorType 2022-10-15 18:21:53 -07:00
Kelly Brazil
e357b27433 more granular type annotations 2022-10-15 17:53:02 -07:00
Kelly Brazil
dd70bf92f3 doc update 2022-10-15 14:43:45 -07:00
Kelly Brazil
0a97523928 doc update 2022-10-15 14:36:35 -07:00
Kelly Brazil
f67bd02283 More granular type annotations 2022-10-15 14:34:09 -07:00
Kelly Brazil
ef1055a9b6 more granular type annotations 2022-10-15 14:06:38 -07:00
Kelly Brazil
cd970b5871 TypedDict fixup for older python versions 2022-10-15 13:53:56 -07:00
Kelly Brazil
11b0863a65 Use TypedDict for more granular typing 2022-10-15 13:44:44 -07:00
Kelly Brazil
5b8cb497de add more type annotations 2022-10-14 17:28:34 -07:00
Kelly Brazil
38e2addbb6 simplify type annotation for MetadataType 2022-10-14 17:23:38 -07:00
Kelly Brazil
58158ce8b1 add type annotations 2022-10-14 17:12:58 -07:00
Kelly Brazil
5ca281f02e add type annotations 2022-10-14 15:54:26 -07:00
Kelly Brazil
ac39ce6b98 formatting 2022-10-14 14:07:05 -07:00
Kelly Brazil
38fb0a6828 allow parser module as parse() argument 2022-10-14 14:05:16 -07:00
Kelly Brazil
38c41cfaf4 formatting 2022-10-14 13:49:10 -07:00
Kelly Brazil
95ba628ac3 add more type annotations 2022-10-14 13:38:05 -07:00
Kelly Brazil
a53f2ecbaf split slot info 2022-10-12 09:07:17 -07:00
Kelly Brazil
aa1ff55bbe lower-case key names 2022-10-12 08:48:42 -07:00
Kelly Brazil
097d013447 remove bail if a, h, or v in magic_options to keep from confusing behavior 2022-10-11 17:18:55 -07:00
Kelly Brazil
3916623c46 remove multiple ands in conditional 2022-10-11 16:22:10 -07:00
Kelly Brazil
42c50c5fa1 remove unneeded fields 2022-10-11 13:44:28 -07:00
Kelly Brazil
ea515bbecc add show_hidden and show_deprecated tests 2022-10-11 12:19:11 -07:00
Kelly Brazil
9f8006060b deprecated parser support in lib 2022-10-10 16:34:32 -07:00
Kelly Brazil
6cd51c4426 add ip_split field 2022-10-10 12:21:02 -07:00
Kelly Brazil
b1e96bb50c add ip_split field 2022-10-10 12:19:39 -07:00
Kelly Brazil
50d3bda3b7 prefix ids with underscores for easier querying with jq, etc. 2022-10-10 11:37:33 -07:00
Kelly Brazil
3dfa4a1bbc rename iso-datetime to datetime-iso 2022-10-09 11:05:32 -07:00
Kelly Brazil
d1922acfc8 add datetime_iso tests 2022-10-09 11:03:31 -07:00
Kelly Brazil
3947e424e7 rename iso-datetime parser to datetime-iso 2022-10-09 11:00:02 -07:00
Kelly Brazil
25e202a09e doc update 2022-10-08 09:04:21 -07:00
Kelly Brazil
8eab3c1590 fixes for cli refactor 2022-10-08 09:04:04 -07:00
Kelly Brazil
4f294aa0ef add pci_ids parser 2022-10-07 16:36:46 -07:00
Kelly Brazil
81aa09a2e2 initial lspci parser 2022-10-07 13:06:20 -07:00
Kelly Brazil
10fc8cb48d add udevadm parser 2022-10-07 11:28:56 -07:00
Kelly Brazil
186f0708cc doc update 2022-10-07 09:34:42 -07:00
Kelly Brazil
622b3ff9ed add __slots__ to timestamp class 2022-10-06 09:05:36 -07:00
Kelly Brazil
c06963c3d6 doc update 2022-10-04 15:16:35 -07:00
Kelly Brazil
9009313748 move error codes from attribute to global constant 2022-10-04 15:04:00 -07:00
Kelly Brazil
d55a8c3079 clean up exit methods 2022-10-04 14:52:51 -07:00
Kelly Brazil
fee6a61f19 Merge pull request #299 from kellyjonbrazil/cli-refactor
Cli refactor
2022-10-04 21:18:58 +00:00
Kelly Brazil
6f1ef09d2a formatting 2022-10-04 14:13:48 -07:00
Kelly Brazil
cf6c13e605 remove old cli module 2022-10-04 14:07:02 -07:00
Kelly Brazil
028f55910a fix for rare instances when the output is a list of lists (yaml | yaml -> json) 2022-10-04 12:14:32 -07:00
Kelly Brazil
d7684d39a8 set magic_run_command_str in magic_parser 2022-10-04 11:48:36 -07:00
Kelly Brazil
f652ccd4b1 fix tests for env_colors 2022-10-04 11:39:50 -07:00
Kelly Brazil
6be92498bc remove self.env_colors attribute 2022-10-04 11:32:18 -07:00
Kelly Brazil
677e04ab7d cleanup 2022-10-04 11:15:37 -07:00
Kelly Brazil
46fdc457fc fix exit codes 2022-10-04 10:10:59 -07:00
Kelly Brazil
0306b6b73b simplify file open method 2022-10-04 09:38:46 -07:00
Kelly Brazil
c881653d55 simplify the run() method 2022-10-04 09:26:40 -07:00
Kelly Brazil
7650d831e3 add slots 2022-10-03 18:03:06 -07:00
Kelly Brazil
e7a8cc3b8b fix unbuffer 2022-10-03 17:38:55 -07:00
Kelly Brazil
d173b2f237 fix magic parser for cases where standard parser is found 2022-10-03 15:04:18 -07:00
Kelly Brazil
49ba6ed0f2 formatting 2022-10-03 13:13:10 -07:00
Kelly Brazil
2792d05c7f fix magic parser and tests 2022-10-03 13:03:42 -07:00
Kelly Brazil
cacda0f3cc fix cli tests 2022-10-03 12:49:13 -07:00
Kelly Brazil
c420547ff8 simplify monochrome settings 2022-10-03 11:30:29 -07:00
Kelly Brazil
83d388613f cleanup 2022-10-03 08:58:09 -07:00
Kelly Brazil
094b059aea fix magic options 2022-10-02 19:30:50 -07:00
Kelly Brazil
e36a8c627b fix meta timestamp 2022-10-02 18:55:54 -07:00
Kelly Brazil
d341e91290 cleanup 2022-10-02 18:29:02 -07:00
Kelly Brazil
557afc95bd initial working 2022-10-02 18:20:14 -07:00
Kelly Brazil
cb9979ac94 refactor cli 2022-10-02 16:58:20 -07:00
Kelly Brazil
5486957141 version bump 2022-09-29 09:14:53 -07:00
Kelly Brazil
d49155df95 make regex patterns raw strings to reduce pytest warnings 2022-09-28 15:09:06 -07:00
Kelly Brazil
0dd4b5f620 parser version bump 2022-09-28 14:56:37 -07:00
Kelly Brazil
690603bfda fix proc-pid-stat parser for command names with spaces and newlines 2022-09-28 14:55:25 -07:00
228 changed files with 46644 additions and 2035 deletions

View File

@@ -14,10 +14,10 @@ jobs:
strategy:
matrix:
os: [macos-latest, ubuntu-latest, windows-latest]
python-version: ["3.7", "3.8", "3.9", "3.10"]
python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v3
- name: "Set up timezone to America/Los_Angeles"
uses: szenius/set-timezone@v1.0
with:
@@ -25,7 +25,7 @@ jobs:
timezoneMacos: "America/Los_Angeles"
timezoneWindows: "Pacific Standard Time"
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v1
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies

View File

@@ -1,5 +1,52 @@
jc changelog
20221216 v1.22.3
- Add Common Log Format and Combined Log Format file parser (standard and streaming)
- Add PostgreSQL password file parser
- Add openvpn-status.log file parser
- Add `cbt` command parser (Google Big Table)
- Enhance `ifconfig` parser with interface lane information on BSD
- Enhance `ifconfig` parser with additional IPv6 `scope_id` info for BSD
- Fix `ifconfig` parser to capture some IPv6 addresses missed on BSD
- Fix `git-log` and `git-log-s` parsers for failure on empty author name
- Update `os-prober` parser with split EFI partition fields
- Add ISO string attribute (`.iso`) to `jc.utils.timestamp()`
- Fix several documentation typos
20221107 v1.22.2
- add `sshd_conf` parser for `sshd` configuration files and `sshd -T` output
- add `findmnt` command parser
- add `git ls-remote` command parser
- add `os-prober` command parser
- add SemVer string parser
- enhance the `ifconfig` parser so it can output multiple IPv4 and IPv6 addresses
- enhance the `ifconfig` parser so it can output additional fields common on BSD
- enhance `xml` parser with optional `_` prefix for attributes instead of
`@` by using the `--raw` option. This can make it easier to filter the
JSON output in some tools.
- fix the `xml` parser to output a normal Dictionary instead of OrderdDict.
This cleans up YAML output. (No `!!omap` comments)
- fix `csv` and `csv-s` parsers for UTF-8 encoded CSV files with leading BOM bytes
- fix exit code to be non-zero on interrupt
- allow parser module objects to be used as arguments to `jc.get_help()` and `jc.parser_info()`
- catch unexpected exceptions in the CLI
- add error message on interrupt to STDERR
- add python 3.11 tests to github actions
20221024 v1.22.1
- add `udevadm` command parser
- add `lspci` command parser
- add `pci.ids` file parser
- fix `proc-pid-stat` parser for command names with spaces and newlines
- enhance `ip-address` parser to add `ip_split` field
- rename `iso-datetime` parser to `datetime-iso`. A deprecation warning will
display until `iso-datetime` is removed in a future version.
- refactor cli module
- optimize performance of calculated timestamps
- add more type annotations
- add support for deprecating parsers
- move jc-web demo site from heroku to render.com
20220926 v1.22.0
- Add /proc file parsers for linux. Support for the following files:
`/proc/buddyinfo`
@@ -547,7 +594,7 @@ jc changelog
20200211 v1.7.3
- Add alternative 'magic' syntax: e.g. `jc ls -al`
- Options can now be condensed (e.g. -prq is equivalant to -p -r -q)
- Options can now be condensed (e.g. -prq is equivalent to -p -r -q)
20200208 v1.7.2
- Include test fixtures in wheel and sdist

View File

@@ -1707,6 +1707,12 @@ echo 192.168.2.10/24 | jc --ip-address -p
"ip": "192.168.2.10",
"ip_compressed": "192.168.2.10",
"ip_exploded": "192.168.2.10",
"ip_split": [
"192",
"168",
"2",
"10"
],
"scope_id": null,
"ipv4_mapped": null,
"six_to_four": null,
@@ -1819,7 +1825,7 @@ iptables --line-numbers -v -L -t nat | jc --iptables -p # or: jc -p ip
```
### ISO Datetime string
```bash
echo "2022-07-20T14:52:45Z" | jc --iso-datetime -p
echo "2022-07-20T14:52:45Z" | jc --datetime-iso -p
```
```json
{

View File

@@ -1,2 +1,3 @@
include jc/py.typed
include man/jc.1
include CHANGELOG

View File

@@ -3,7 +3,7 @@
> Check out the `jc` Python [package documentation](https://github.com/kellyjonbrazil/jc/tree/master/docs) for developers
> Try the `jc` [web demo](https://jc-web-demo.herokuapp.com/)
> Try the `jc` [web demo](https://jc-web.onrender.com/)
> JC is [now available](https://galaxy.ansible.com/community/general) as an
Ansible filter plugin in the `community.general` collection. See this
@@ -114,7 +114,7 @@ pip3 install jc
| Debian/Ubuntu linux | `apt-get install jc` |
| Fedora linux | `dnf install jc` |
| openSUSE linux | `zypper install jc` |
| Archlinux Community Repository | `paru -S jc` or `aura -S jc` or `yay -S jc` |
| Arch linux | `pacman -S jc` |
| NixOS linux | `nix-env -iA nixpkgs.jc` or `nix-env -iA nixos.jc` |
| Guix System linux | `guix install jc` |
| Gentoo Linux | `emerge dev-python/jc` |
@@ -161,15 +161,19 @@ option.
| ` --asciitable` | ASCII and Unicode table parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/asciitable) |
| ` --asciitable-m` | multi-line ASCII and Unicode table parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/asciitable_m) |
| ` --blkid` | `blkid` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/blkid) |
| ` --cbt` | `cbt` (Google Bigtable) command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/cbt) |
| ` --cef` | CEF string parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/cef) |
| ` --cef-s` | CEF string streaming parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/cef_s) |
| ` --chage` | `chage --list` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/chage) |
| ` --cksum` | `cksum` and `sum` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/cksum) |
| ` --clf` | Common and Combined Log Format file parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/clf) |
| ` --clf-s` | Common and Combined Log Format file streaming parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/clf_s) |
| ` --crontab` | `crontab` command and file parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/crontab) |
| ` --crontab-u` | `crontab` file parser with user support | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/crontab_u) |
| ` --csv` | CSV file parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/csv) |
| ` --csv-s` | CSV file streaming parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/csv_s) |
| ` --date` | `date` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/date) |
| ` --datetime-iso` | ISO 8601 Datetime string parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/datetime_iso) |
| ` --df` | `df` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/df) |
| ` --dig` | `dig` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/dig) |
| ` --dir` | `dir` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/dir) |
@@ -179,11 +183,13 @@ option.
| `--email-address` | Email Address string parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/email_address) |
| ` --env` | `env` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/env) |
| ` --file` | `file` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/file) |
| ` --findmnt` | `findmnt` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/findmnt) |
| ` --finger` | `finger` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/finger) |
| ` --free` | `free` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/free) |
| ` --fstab` | `/etc/fstab` file parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/fstab) |
| ` --git-log` | `git log` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/git_log) |
| ` --git-log-s` | `git log` command streaming parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/git_log_s) |
| `--git-ls-remote` | `git ls-remote` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/git_ls_remote) |
| ` --gpg` | `gpg --with-colons` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/gpg) |
| ` --group` | `/etc/group` file parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/group) |
| ` --gshadow` | `/etc/gshadow` file parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/gshadow) |
@@ -199,7 +205,6 @@ option.
| ` --iostat-s` | `iostat` command streaming parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/iostat_s) |
| ` --ip-address` | IPv4 and IPv6 Address string parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/ip_address) |
| ` --iptables` | `iptables` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/iptables) |
| ` --iso-datetime` | ISO 8601 Datetime string parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/iso_datetime) |
| ` --iw-scan` | `iw dev [device] scan` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/iw_scan) |
| ` --jar-manifest` | Java MANIFEST.MF file parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/jar_manifest) |
| ` --jobs` | `jobs` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/jobs) |
@@ -211,6 +216,7 @@ option.
| ` --lsblk` | `lsblk` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/lsblk) |
| ` --lsmod` | `lsmod` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/lsmod) |
| ` --lsof` | `lsof` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/lsof) |
| ` --lspci` | `lspci -mmv` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/lspci) |
| ` --lsusb` | `lsusb` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/lsusb) |
| ` --m3u` | M3U and M3U8 file parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/m3u) |
| ` --mdadm` | `mdadm` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/mdadm) |
@@ -220,7 +226,11 @@ option.
| ` --netstat` | `netstat` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/netstat) |
| ` --nmcli` | `nmcli` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/nmcli) |
| ` --ntpq` | `ntpq -p` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/ntpq) |
| ` --openvpn` | openvpn-status.log file parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/openvpn) |
| ` --os-prober` | `os-prober` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/os_prober) |
| ` --passwd` | `/etc/passwd` file parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/passwd) |
| ` --pci-ids` | `pci.ids` file parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/pci_ids) |
| ` --pgpass` | PostgreSQL password file parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/pgpass) |
| ` --pidstat` | `pidstat -H` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/pidstat) |
| ` --pidstat-s` | `pidstat -H` command streaming parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/pidstat_s) |
| ` --ping` | `ping` and `ping6` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/ping) |
@@ -235,9 +245,11 @@ option.
| ` --rpm-qi` | `rpm -qi` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/rpm_qi) |
| ` --rsync` | `rsync` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/rsync) |
| ` --rsync-s` | `rsync` command streaming parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/rsync_s) |
| ` --semver` | Semantic Version string parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/semver) |
| ` --sfdisk` | `sfdisk` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/sfdisk) |
| ` --shadow` | `/etc/shadow` file parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/shadow) |
| ` --ss` | `ss` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/ss) |
| ` --sshd-conf` | sshd config file and `sshd -T` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/sshd_conf) |
| ` --stat` | `stat` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/stat) |
| ` --stat-s` | `stat` command streaming parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/stat_s) |
| ` --sysctl` | `sysctl` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/sysctl) |
@@ -257,6 +269,7 @@ option.
| ` --top-s` | `top -b` command streaming parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/top_s) |
| ` --tracepath` | `tracepath` and `tracepath6` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/tracepath) |
| ` --traceroute` | `traceroute` and `traceroute6` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/traceroute) |
| ` --udevadm` | `udevadm info` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/udevadm) |
| ` --ufw` | `ufw status` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/ufw) |
| ` --ufw-appinfo` | `ufw app info [application]` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/ufw_appinfo) |
| ` --uname` | `uname -a` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/uname) |
@@ -382,7 +395,7 @@ option.
### Streaming Parsers
Most parsers load all of the data from `STDIN`, parse it, then output the entire
JSON document serially. There are some streaming parsers (e.g. `ls-s` and
`ping-s`) that immediately start processing and outputing the data line-by-line
`ping-s`) that immediately start processing and outputting the data line-by-line
as [JSON Lines](https://jsonlines.org/) (aka [NDJSON](http://ndjson.org/)) while
it is being received from `STDIN`. This can significantly reduce the amount of
memory required to parse large amounts of command output (e.g. `ls -lR /`) and

View File

@@ -3,8 +3,8 @@ _jc()
local cur prev words cword jc_commands jc_parsers jc_options \
jc_about_options jc_about_mod_options jc_help_options jc_special_options
jc_commands=(acpi airport arp blkid chage cksum crontab date df dig dmidecode dpkg du env file finger free git gpg hciconfig id ifconfig iostat iptables iw jobs last lastb ls lsblk lsmod lsof lsusb md5 md5sum mdadm mount mpstat netstat nmcli ntpq pidstat ping ping6 pip pip3 postconf printenv ps route rpm rsync sfdisk sha1sum sha224sum sha256sum sha384sum sha512sum shasum ss stat sum sysctl systemctl systeminfo timedatectl top tracepath tracepath6 traceroute traceroute6 ufw uname update-alternatives upower uptime vdir vmstat w wc who xrandr zipinfo)
jc_parsers=(--acpi --airport --airport-s --arp --asciitable --asciitable-m --blkid --cef --cef-s --chage --cksum --crontab --crontab-u --csv --csv-s --date --df --dig --dir --dmidecode --dpkg-l --du --email-address --env --file --finger --free --fstab --git-log --git-log-s --gpg --group --gshadow --hash --hashsum --hciconfig --history --hosts --id --ifconfig --ini --iostat --iostat-s --ip-address --iptables --iso-datetime --iw-scan --jar-manifest --jobs --jwt --kv --last --ls --ls-s --lsblk --lsmod --lsof --lsusb --m3u --mdadm --mount --mpstat --mpstat-s --netstat --nmcli --ntpq --passwd --pidstat --pidstat-s --ping --ping-s --pip-list --pip-show --plist --postconf --proc --proc-buddyinfo --proc-consoles --proc-cpuinfo --proc-crypto --proc-devices --proc-diskstats --proc-filesystems --proc-interrupts --proc-iomem --proc-ioports --proc-loadavg --proc-locks --proc-meminfo --proc-modules --proc-mtrr --proc-pagetypeinfo --proc-partitions --proc-slabinfo --proc-softirqs --proc-stat --proc-swaps --proc-uptime --proc-version --proc-vmallocinfo --proc-vmstat --proc-zoneinfo --proc-driver-rtc --proc-net-arp --proc-net-dev --proc-net-dev-mcast --proc-net-if-inet6 --proc-net-igmp --proc-net-igmp6 --proc-net-ipv6-route --proc-net-netlink --proc-net-netstat --proc-net-packet --proc-net-protocols --proc-net-route --proc-net-unix --proc-pid-fdinfo --proc-pid-io --proc-pid-maps --proc-pid-mountinfo --proc-pid-numa-maps --proc-pid-smaps --proc-pid-stat --proc-pid-statm --proc-pid-status --ps --route --rpm-qi --rsync --rsync-s --sfdisk --shadow --ss --stat --stat-s --sysctl --syslog --syslog-s --syslog-bsd --syslog-bsd-s --systemctl --systemctl-lj --systemctl-ls --systemctl-luf --systeminfo --time --timedatectl --timestamp --top --top-s --tracepath --traceroute --ufw --ufw-appinfo --uname --update-alt-gs --update-alt-q --upower --uptime --url --vmstat --vmstat-s --w --wc --who --x509-cert --xml --xrandr --yaml --zipinfo)
jc_commands=(acpi airport arp blkid cbt chage cksum crontab date df dig dmidecode dpkg du env file findmnt finger free git gpg hciconfig id ifconfig iostat iptables iw jobs last lastb ls lsblk lsmod lsof lspci lsusb md5 md5sum mdadm mount mpstat netstat nmcli ntpq os-prober pidstat ping ping6 pip pip3 postconf printenv ps route rpm rsync sfdisk sha1sum sha224sum sha256sum sha384sum sha512sum shasum ss sshd stat sum sysctl systemctl systeminfo timedatectl top tracepath tracepath6 traceroute traceroute6 udevadm ufw uname update-alternatives upower uptime vdir vmstat w wc who xrandr zipinfo)
jc_parsers=(--acpi --airport --airport-s --arp --asciitable --asciitable-m --blkid --cbt --cef --cef-s --chage --cksum --clf --clf-s --crontab --crontab-u --csv --csv-s --date --datetime-iso --df --dig --dir --dmidecode --dpkg-l --du --email-address --env --file --findmnt --finger --free --fstab --git-log --git-log-s --git-ls-remote --gpg --group --gshadow --hash --hashsum --hciconfig --history --hosts --id --ifconfig --ini --iostat --iostat-s --ip-address --iptables --iw-scan --jar-manifest --jobs --jwt --kv --last --ls --ls-s --lsblk --lsmod --lsof --lspci --lsusb --m3u --mdadm --mount --mpstat --mpstat-s --netstat --nmcli --ntpq --openvpn --os-prober --passwd --pci-ids --pgpass --pidstat --pidstat-s --ping --ping-s --pip-list --pip-show --plist --postconf --proc --proc-buddyinfo --proc-consoles --proc-cpuinfo --proc-crypto --proc-devices --proc-diskstats --proc-filesystems --proc-interrupts --proc-iomem --proc-ioports --proc-loadavg --proc-locks --proc-meminfo --proc-modules --proc-mtrr --proc-pagetypeinfo --proc-partitions --proc-slabinfo --proc-softirqs --proc-stat --proc-swaps --proc-uptime --proc-version --proc-vmallocinfo --proc-vmstat --proc-zoneinfo --proc-driver-rtc --proc-net-arp --proc-net-dev --proc-net-dev-mcast --proc-net-if-inet6 --proc-net-igmp --proc-net-igmp6 --proc-net-ipv6-route --proc-net-netlink --proc-net-netstat --proc-net-packet --proc-net-protocols --proc-net-route --proc-net-unix --proc-pid-fdinfo --proc-pid-io --proc-pid-maps --proc-pid-mountinfo --proc-pid-numa-maps --proc-pid-smaps --proc-pid-stat --proc-pid-statm --proc-pid-status --ps --route --rpm-qi --rsync --rsync-s --semver --sfdisk --shadow --ss --sshd-conf --stat --stat-s --sysctl --syslog --syslog-s --syslog-bsd --syslog-bsd-s --systemctl --systemctl-lj --systemctl-ls --systemctl-luf --systeminfo --time --timedatectl --timestamp --top --top-s --tracepath --traceroute --udevadm --ufw --ufw-appinfo --uname --update-alt-gs --update-alt-q --upower --uptime --url --vmstat --vmstat-s --w --wc --who --x509-cert --xml --xrandr --yaml --zipinfo)
jc_options=(--force-color -C --debug -d --monochrome -m --meta-out -M --pretty -p --quiet -q --raw -r --unbuffer -u --yaml-out -y)
jc_about_options=(--about -a)
jc_about_mod_options=(--pretty -p --yaml-out -y --monochrome -m --force-color -C)

View File

@@ -9,12 +9,13 @@ _jc() {
jc_help_options jc_help_options_describe \
jc_special_options jc_special_options_describe
jc_commands=(acpi airport arp blkid chage cksum crontab date df dig dmidecode dpkg du env file finger free git gpg hciconfig id ifconfig iostat iptables iw jobs last lastb ls lsblk lsmod lsof lsusb md5 md5sum mdadm mount mpstat netstat nmcli ntpq pidstat ping ping6 pip pip3 postconf printenv ps route rpm rsync sfdisk sha1sum sha224sum sha256sum sha384sum sha512sum shasum ss stat sum sysctl systemctl systeminfo timedatectl top tracepath tracepath6 traceroute traceroute6 ufw uname update-alternatives upower uptime vdir vmstat w wc who xrandr zipinfo)
jc_commands=(acpi airport arp blkid cbt chage cksum crontab date df dig dmidecode dpkg du env file findmnt finger free git gpg hciconfig id ifconfig iostat iptables iw jobs last lastb ls lsblk lsmod lsof lspci lsusb md5 md5sum mdadm mount mpstat netstat nmcli ntpq os-prober pidstat ping ping6 pip pip3 postconf printenv ps route rpm rsync sfdisk sha1sum sha224sum sha256sum sha384sum sha512sum shasum ss sshd stat sum sysctl systemctl systeminfo timedatectl top tracepath tracepath6 traceroute traceroute6 udevadm ufw uname update-alternatives upower uptime vdir vmstat w wc who xrandr zipinfo)
jc_commands_describe=(
'acpi:run "acpi" command with magic syntax.'
'airport:run "airport" command with magic syntax.'
'arp:run "arp" command with magic syntax.'
'blkid:run "blkid" command with magic syntax.'
'cbt:run "cbt" command with magic syntax.'
'chage:run "chage" command with magic syntax.'
'cksum:run "cksum" command with magic syntax.'
'crontab:run "crontab" command with magic syntax.'
@@ -26,6 +27,7 @@ _jc() {
'du:run "du" command with magic syntax.'
'env:run "env" command with magic syntax.'
'file:run "file" command with magic syntax.'
'findmnt:run "findmnt" command with magic syntax.'
'finger:run "finger" command with magic syntax.'
'free:run "free" command with magic syntax.'
'git:run "git" command with magic syntax.'
@@ -43,6 +45,7 @@ _jc() {
'lsblk:run "lsblk" command with magic syntax.'
'lsmod:run "lsmod" command with magic syntax.'
'lsof:run "lsof" command with magic syntax.'
'lspci:run "lspci" command with magic syntax.'
'lsusb:run "lsusb" command with magic syntax.'
'md5:run "md5" command with magic syntax.'
'md5sum:run "md5sum" command with magic syntax.'
@@ -52,6 +55,7 @@ _jc() {
'netstat:run "netstat" command with magic syntax.'
'nmcli:run "nmcli" command with magic syntax.'
'ntpq:run "ntpq" command with magic syntax.'
'os-prober:run "os-prober" command with magic syntax.'
'pidstat:run "pidstat" command with magic syntax.'
'ping:run "ping" command with magic syntax.'
'ping6:run "ping6" command with magic syntax.'
@@ -71,6 +75,7 @@ _jc() {
'sha512sum:run "sha512sum" command with magic syntax.'
'shasum:run "shasum" command with magic syntax.'
'ss:run "ss" command with magic syntax.'
'sshd:run "sshd" command with magic syntax.'
'stat:run "stat" command with magic syntax.'
'sum:run "sum" command with magic syntax.'
'sysctl:run "sysctl" command with magic syntax.'
@@ -82,6 +87,7 @@ _jc() {
'tracepath6:run "tracepath6" command with magic syntax.'
'traceroute:run "traceroute" command with magic syntax.'
'traceroute6:run "traceroute6" command with magic syntax.'
'udevadm:run "udevadm" command with magic syntax.'
'ufw:run "ufw" command with magic syntax.'
'uname:run "uname" command with magic syntax.'
'update-alternatives:run "update-alternatives" command with magic syntax.'
@@ -95,7 +101,7 @@ _jc() {
'xrandr:run "xrandr" command with magic syntax.'
'zipinfo:run "zipinfo" command with magic syntax.'
)
jc_parsers=(--acpi --airport --airport-s --arp --asciitable --asciitable-m --blkid --cef --cef-s --chage --cksum --crontab --crontab-u --csv --csv-s --date --df --dig --dir --dmidecode --dpkg-l --du --email-address --env --file --finger --free --fstab --git-log --git-log-s --gpg --group --gshadow --hash --hashsum --hciconfig --history --hosts --id --ifconfig --ini --iostat --iostat-s --ip-address --iptables --iso-datetime --iw-scan --jar-manifest --jobs --jwt --kv --last --ls --ls-s --lsblk --lsmod --lsof --lsusb --m3u --mdadm --mount --mpstat --mpstat-s --netstat --nmcli --ntpq --passwd --pidstat --pidstat-s --ping --ping-s --pip-list --pip-show --plist --postconf --proc --proc-buddyinfo --proc-consoles --proc-cpuinfo --proc-crypto --proc-devices --proc-diskstats --proc-filesystems --proc-interrupts --proc-iomem --proc-ioports --proc-loadavg --proc-locks --proc-meminfo --proc-modules --proc-mtrr --proc-pagetypeinfo --proc-partitions --proc-slabinfo --proc-softirqs --proc-stat --proc-swaps --proc-uptime --proc-version --proc-vmallocinfo --proc-vmstat --proc-zoneinfo --proc-driver-rtc --proc-net-arp --proc-net-dev --proc-net-dev-mcast --proc-net-if-inet6 --proc-net-igmp --proc-net-igmp6 --proc-net-ipv6-route --proc-net-netlink --proc-net-netstat --proc-net-packet --proc-net-protocols --proc-net-route --proc-net-unix --proc-pid-fdinfo --proc-pid-io --proc-pid-maps --proc-pid-mountinfo --proc-pid-numa-maps --proc-pid-smaps --proc-pid-stat --proc-pid-statm --proc-pid-status --ps --route --rpm-qi --rsync --rsync-s --sfdisk --shadow --ss --stat --stat-s --sysctl --syslog --syslog-s --syslog-bsd --syslog-bsd-s --systemctl --systemctl-lj --systemctl-ls --systemctl-luf --systeminfo --time --timedatectl --timestamp --top --top-s --tracepath --traceroute --ufw --ufw-appinfo --uname --update-alt-gs --update-alt-q --upower --uptime --url --vmstat --vmstat-s --w --wc --who --x509-cert --xml --xrandr --yaml --zipinfo)
jc_parsers=(--acpi --airport --airport-s --arp --asciitable --asciitable-m --blkid --cbt --cef --cef-s --chage --cksum --clf --clf-s --crontab --crontab-u --csv --csv-s --date --datetime-iso --df --dig --dir --dmidecode --dpkg-l --du --email-address --env --file --findmnt --finger --free --fstab --git-log --git-log-s --git-ls-remote --gpg --group --gshadow --hash --hashsum --hciconfig --history --hosts --id --ifconfig --ini --iostat --iostat-s --ip-address --iptables --iw-scan --jar-manifest --jobs --jwt --kv --last --ls --ls-s --lsblk --lsmod --lsof --lspci --lsusb --m3u --mdadm --mount --mpstat --mpstat-s --netstat --nmcli --ntpq --openvpn --os-prober --passwd --pci-ids --pgpass --pidstat --pidstat-s --ping --ping-s --pip-list --pip-show --plist --postconf --proc --proc-buddyinfo --proc-consoles --proc-cpuinfo --proc-crypto --proc-devices --proc-diskstats --proc-filesystems --proc-interrupts --proc-iomem --proc-ioports --proc-loadavg --proc-locks --proc-meminfo --proc-modules --proc-mtrr --proc-pagetypeinfo --proc-partitions --proc-slabinfo --proc-softirqs --proc-stat --proc-swaps --proc-uptime --proc-version --proc-vmallocinfo --proc-vmstat --proc-zoneinfo --proc-driver-rtc --proc-net-arp --proc-net-dev --proc-net-dev-mcast --proc-net-if-inet6 --proc-net-igmp --proc-net-igmp6 --proc-net-ipv6-route --proc-net-netlink --proc-net-netstat --proc-net-packet --proc-net-protocols --proc-net-route --proc-net-unix --proc-pid-fdinfo --proc-pid-io --proc-pid-maps --proc-pid-mountinfo --proc-pid-numa-maps --proc-pid-smaps --proc-pid-stat --proc-pid-statm --proc-pid-status --ps --route --rpm-qi --rsync --rsync-s --semver --sfdisk --shadow --ss --sshd-conf --stat --stat-s --sysctl --syslog --syslog-s --syslog-bsd --syslog-bsd-s --systemctl --systemctl-lj --systemctl-ls --systemctl-luf --systeminfo --time --timedatectl --timestamp --top --top-s --tracepath --traceroute --udevadm --ufw --ufw-appinfo --uname --update-alt-gs --update-alt-q --upower --uptime --url --vmstat --vmstat-s --w --wc --who --x509-cert --xml --xrandr --yaml --zipinfo)
jc_parsers_describe=(
'--acpi:`acpi` command parser'
'--airport:`airport -I` command parser'
@@ -104,15 +110,19 @@ _jc() {
'--asciitable:ASCII and Unicode table parser'
'--asciitable-m:multi-line ASCII and Unicode table parser'
'--blkid:`blkid` command parser'
'--cbt:`cbt` (Google Bigtable) command parser'
'--cef:CEF string parser'
'--cef-s:CEF string streaming parser'
'--chage:`chage --list` command parser'
'--cksum:`cksum` and `sum` command parser'
'--clf:Common and Combined Log Format file parser'
'--clf-s:Common and Combined Log Format file streaming parser'
'--crontab:`crontab` command and file parser'
'--crontab-u:`crontab` file parser with user support'
'--csv:CSV file parser'
'--csv-s:CSV file streaming parser'
'--date:`date` command parser'
'--datetime-iso:ISO 8601 Datetime string parser'
'--df:`df` command parser'
'--dig:`dig` command parser'
'--dir:`dir` command parser'
@@ -122,11 +132,13 @@ _jc() {
'--email-address:Email Address string parser'
'--env:`env` command parser'
'--file:`file` command parser'
'--findmnt:`findmnt` command parser'
'--finger:`finger` command parser'
'--free:`free` command parser'
'--fstab:`/etc/fstab` file parser'
'--git-log:`git log` command parser'
'--git-log-s:`git log` command streaming parser'
'--git-ls-remote:`git ls-remote` command parser'
'--gpg:`gpg --with-colons` command parser'
'--group:`/etc/group` file parser'
'--gshadow:`/etc/gshadow` file parser'
@@ -142,7 +154,6 @@ _jc() {
'--iostat-s:`iostat` command streaming parser'
'--ip-address:IPv4 and IPv6 Address string parser'
'--iptables:`iptables` command parser'
'--iso-datetime:ISO 8601 Datetime string parser'
'--iw-scan:`iw dev [device] scan` command parser'
'--jar-manifest:Java MANIFEST.MF file parser'
'--jobs:`jobs` command parser'
@@ -154,6 +165,7 @@ _jc() {
'--lsblk:`lsblk` command parser'
'--lsmod:`lsmod` command parser'
'--lsof:`lsof` command parser'
'--lspci:`lspci -mmv` command parser'
'--lsusb:`lsusb` command parser'
'--m3u:M3U and M3U8 file parser'
'--mdadm:`mdadm` command parser'
@@ -163,7 +175,11 @@ _jc() {
'--netstat:`netstat` command parser'
'--nmcli:`nmcli` command parser'
'--ntpq:`ntpq -p` command parser'
'--openvpn:openvpn-status.log file parser'
'--os-prober:`os-prober` command parser'
'--passwd:`/etc/passwd` file parser'
'--pci-ids:`pci.ids` file parser'
'--pgpass:PostgreSQL password file parser'
'--pidstat:`pidstat -H` command parser'
'--pidstat-s:`pidstat -H` command streaming parser'
'--ping:`ping` and `ping6` command parser'
@@ -227,9 +243,11 @@ _jc() {
'--rpm-qi:`rpm -qi` command parser'
'--rsync:`rsync` command parser'
'--rsync-s:`rsync` command streaming parser'
'--semver:Semantic Version string parser'
'--sfdisk:`sfdisk` command parser'
'--shadow:`/etc/shadow` file parser'
'--ss:`ss` command parser'
'--sshd-conf:sshd config file and `sshd -T` command parser'
'--stat:`stat` command parser'
'--stat-s:`stat` command streaming parser'
'--sysctl:`sysctl` command parser'
@@ -249,6 +267,7 @@ _jc() {
'--top-s:`top -b` command streaming parser'
'--tracepath:`tracepath` and `tracepath6` command parser'
'--traceroute:`traceroute` and `traceroute6` command parser'
'--udevadm:`udevadm info` command parser'
'--ufw:`ufw status` command parser'
'--ufw-appinfo:`ufw app info [application]` command parser'
'--uname:`uname -a` command parser'

View File

@@ -21,15 +21,18 @@ jc - JSON Convert lib module
### parse
```python
def parse(parser_mod_name: str,
data: Union[str, bytes, Iterable[str]],
quiet: bool = False,
raw: bool = False,
ignore_exceptions: bool = None,
**kwargs) -> Union[Dict, List[Dict], Iterator[Dict]]
def parse(
parser_mod_name: Union[str, ModuleType],
data: Union[str, bytes, Iterable[str]],
quiet: bool = False,
raw: bool = False,
ignore_exceptions: bool = None,
**kwargs
) -> Union[JSONDictType, List[JSONDictType], Iterator[JSONDictType]]
```
Parse the string data using the supplied parser module.
Parse the data (string or bytes) using the supplied parser (string or
module object).
This function provides a high-level API to simplify parser use. This
function will call built-in parsers and custom plugin parsers.
@@ -53,6 +56,14 @@ Example (streaming parsers):
To get a list of available parser module names, use `parser_mod_list()`.
Alternatively, a parser module object can be supplied:
>>> import jc
>>> import jc.parsers.date as jc_date
>>> date_obj = jc.parse(jc_date, 'Tue Jan 18 10:23:07 PST 2022')
>>> print(f'The year is: {date_obj["year"]}')
The year is: 2022
You can also use the lower-level parser modules directly:
>>> import jc.parsers.date
@@ -73,11 +84,14 @@ parsers without this API:
Parameters:
parser_mod_name: (string) name of the parser module. This
function will accept module_name,
parser_mod_name: (string or name of the parser module. This
Module) function will accept module_name,
cli-name, and --argument-name
variants of the module name.
A Module object can also be passed
directly or via _get_parser()
data: (string or data to parse (string or bytes for
bytes or standard parsers, iterable of
iterable) strings for streaming parsers)
@@ -99,7 +113,8 @@ Returns:
### parser\_mod\_list
```python
def parser_mod_list() -> List[str]
def parser_mod_list(show_hidden: bool = False,
show_deprecated: bool = False) -> List[str]
```
Returns a list of all available parser module names.
@@ -109,7 +124,8 @@ Returns a list of all available parser module names.
### plugin\_parser\_mod\_list
```python
def plugin_parser_mod_list() -> List[str]
def plugin_parser_mod_list(show_hidden: bool = False,
show_deprecated: bool = False) -> List[str]
```
Returns a list of plugin parser module names. This function is a
@@ -120,7 +136,8 @@ subset of `parser_mod_list()`.
### standard\_parser\_mod\_list
```python
def standard_parser_mod_list() -> List[str]
def standard_parser_mod_list(show_hidden: bool = False,
show_deprecated: bool = False) -> List[str]
```
Returns a list of standard parser module names. This function is a
@@ -132,7 +149,8 @@ parsers.
### streaming\_parser\_mod\_list
```python
def streaming_parser_mod_list() -> List[str]
def streaming_parser_mod_list(show_hidden: bool = False,
show_deprecated: bool = False) -> List[str]
```
Returns a list of streaming parser module names. This function is a
@@ -143,17 +161,19 @@ subset of `parser_mod_list()`.
### parser\_info
```python
def parser_info(parser_mod_name: str, documentation: bool = False) -> Dict
def parser_info(parser_mod_name: Union[str, ModuleType],
documentation: bool = False) -> ParserInfoType
```
Returns a dictionary that includes the parser module metadata.
Parameters:
parser_mod_name: (string) name of the parser module. This
function will accept module_name,
parser_mod_name: (string or name of the parser module. This
Module) function will accept module_name,
cli-name, and --argument-name
variants of the module name.
variants of the module name as well
as a parser module object.
documentation: (boolean) include parser docstring if True
@@ -163,28 +183,33 @@ Parameters:
```python
def all_parser_info(documentation: bool = False,
show_hidden: bool = False) -> List[Dict]
show_hidden: bool = False,
show_deprecated: bool = False) -> List[ParserInfoType]
```
Returns a list of dictionaries that includes metadata for all parser
modules.
modules. By default only non-hidden, non-deprecated parsers are
returned.
Parameters:
documentation: (boolean) include parser docstrings if True
show_hidden: (boolean) also show parsers marked as hidden
in their info metadata.
show_deprecated: (boolean) also show parsers marked as
deprecated in their info metadata.
<a id="jc.lib.get_help"></a>
### get\_help
```python
def get_help(parser_mod_name: str) -> None
def get_help(parser_mod_name: Union[str, ModuleType]) -> None
```
Show help screen for the selected parser.
This function will accept **module_name**, **cli-name**, and
**--argument-name** variants of the module name string.
**--argument-name** variants of the module name string as well as a
parser module object.

125
docs/parsers/cbt.md Normal file
View File

@@ -0,0 +1,125 @@
[Home](https://kellyjonbrazil.github.io/jc/)
<a id="jc.parsers.cbt"></a>
# jc.parsers.cbt
jc - JSON Convert `cbt` command output parser (Google Bigtable)
Parses the human-, but not machine-, friendly output of the cbt command (for
Google's Bigtable).
No effort is made to convert the data types of the values in the cells.
The `timestamp_epoch` calculated timestamp field is naive. (i.e. based on
the local time of the system the parser is run on)
The `timestamp_epoch_utc` calculated timestamp field is timezone-aware and
is only available if the timestamp has a UTC timezone.
The `timestamp_iso` calculated timestamp field will only include UTC
timezone information if the timestamp has a UTC timezone.
Raw output contains all cells for each column (including timestamps), while
the normal output contains only the latest value for each column.
Usage (cli):
$ cbt | jc --cbt
or
$ jc cbt
Usage (module):
import jc
result = jc.parse('cbt', cbt_command_output)
Schema:
[
{
"key": string,
"cells": {
<string>: { # column family
<string>: string # column: value
}
}
}
]
Schema (raw):
[
{
"key": string,
"cells": [
{
"column_family": string,
"column": string,
"value": string,
"timestamp_iso": string,
"timestamp_epoch": integer,
"timestamp_epoch_utc": integer
}
]
}
]
Examples:
$ cbt -project=$PROJECT -instance=$INSTANCE lookup $TABLE foo | jc --cbt -p
[
{
"key": "foo",
"cells": {
"foo": {
"bar": "baz"
}
}
}
]
$ cbt -project=$PROJECT -instance=$INSTANCE lookup $TABLE foo | jc --cbt -p -r
[
{
"key": "foo",
"cells": [
{
"column_family": "foo",
"column": "bar",
"value": "baz1",
"timestamp_iso": "1970-01-01T01:00:00",
"timestamp_epoch": 32400,
"timestamp_epoch_utc": null
}
]
}
]
<a id="jc.parsers.cbt.parse"></a>
### parse
```python
def parse(data: str,
raw: bool = False,
quiet: bool = False) -> List[JSONDictType]
```
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
Returns:
List of Dictionaries. Raw or processed structured data.
### Parser Information
Compatibility: linux, darwin, cygwin, win32, aix, freebsd
Version 1.0 by Andreas Weiden (andreas.weiden@gmail.com)

199
docs/parsers/clf.md Normal file
View File

@@ -0,0 +1,199 @@
[Home](https://kellyjonbrazil.github.io/jc/)
<a id="jc.parsers.clf"></a>
# jc.parsers.clf
jc - JSON Convert Common Log Format file parser
This parser will handle the Common Log Format standard as specified at
https://www.w3.org/Daemon/User/Config/Logging.html#common-logfile-format.
Combined Log Format is also supported. (Referer and User Agent fields added)
Extra fields may be present and will be enclosed in the `extra` field as
a single string.
If a log line cannot be parsed, an object with an `unparsable` field will
be present with a value of the original line.
The `epoch` calculated timestamp field is naive. (i.e. based on the
local time of the system the parser is run on)
The `epoch_utc` calculated timestamp field is timezone-aware and is
only available if the timezone field is UTC.
Usage (cli):
$ cat file.log | jc --clf
Usage (module):
import jc
result = jc.parse('clf', common_log_file_output)
Schema:
Empty strings and `-` values are converted to `null`/`None`.
[
{
"host": string,
"ident": string,
"authuser": string,
"date": string,
"day": integer,
"month": string,
"year": integer,
"hour": integer,
"minute": integer,
"second": integer,
"tz": string,
"request": string,
"request_method": string,
"request_url": string,
"request_version": string,
"status": integer,
"bytes": integer,
"referer": string,
"user_agent": string,
"extra": string,
"epoch": integer, # [0]
"epoch_utc": integer, # [1]
"unparsable": string # [2]
}
]
[0] naive timestamp
[1] timezone-aware timestamp. Only available if timezone field is UTC
[2] exists if the line was not able to be parsed
Examples:
$ cat file.log | jc --clf -p
[
{
"host": "127.0.0.1",
"ident": "user-identifier",
"authuser": "frank",
"date": "10/Oct/2000:13:55:36 -0700",
"day": 10,
"month": "Oct",
"year": 2000,
"hour": 13,
"minute": 55,
"second": 36,
"tz": "-0700",
"request": "GET /apache_pb.gif HTTPS/1.0",
"status": 200,
"bytes": 2326,
"referer": null,
"user_agent": null,
"extra": null,
"request_method": "GET",
"request_url": "/apache_pb.gif",
"request_version": "HTTPS/1.0",
"epoch": 971211336,
"epoch_utc": null
},
{
"host": "1.1.1.2",
"ident": null,
"authuser": null,
"date": "11/Nov/2016:03:04:55 +0100",
"day": 11,
"month": "Nov",
"year": 2016,
"hour": 3,
"minute": 4,
"second": 55,
"tz": "+0100",
"request": "GET /",
"status": 200,
"bytes": 83,
"referer": null,
"user_agent": null,
"extra": "- 9221 1.1.1.1",
"request_method": "GET",
"request_url": "/",
"request_version": null,
"epoch": 1478862295,
"epoch_utc": null
},
...
]
$ cat file.log | jc --clf -p -r
[
{
"host": "127.0.0.1",
"ident": "user-identifier",
"authuser": "frank",
"date": "10/Oct/2000:13:55:36 -0700",
"day": "10",
"month": "Oct",
"year": "2000",
"hour": "13",
"minute": "55",
"second": "36",
"tz": "-0700",
"request": "GET /apache_pb.gif HTTPS/1.0",
"status": "200",
"bytes": "2326",
"referer": null,
"user_agent": null,
"extra": "",
"request_method": "GET",
"request_url": "/apache_pb.gif",
"request_version": "HTTPS/1.0"
},
{
"host": "1.1.1.2",
"ident": "-",
"authuser": "-",
"date": "11/Nov/2016:03:04:55 +0100",
"day": "11",
"month": "Nov",
"year": "2016",
"hour": "03",
"minute": "04",
"second": "55",
"tz": "+0100",
"request": "GET /",
"status": "200",
"bytes": "83",
"referer": "-",
"user_agent": "-",
"extra": "- 9221 1.1.1.1",
"request_method": "GET",
"request_url": "/",
"request_version": null
},
...
]
<a id="jc.parsers.clf.parse"></a>
### parse
```python
def parse(data: str,
raw: bool = False,
quiet: bool = False) -> List[JSONDictType]
```
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
Returns:
List of Dictionaries. Raw or processed structured data.
### Parser Information
Compatibility: linux, darwin, cygwin, win32, aix, freebsd
Version 1.0 by Kelly Brazil (kellyjonbrazil@gmail.com)

117
docs/parsers/clf_s.md Normal file
View File

@@ -0,0 +1,117 @@
[Home](https://kellyjonbrazil.github.io/jc/)
<a id="jc.parsers.clf_s"></a>
# jc.parsers.clf\_s
jc - JSON Convert Common Log Format file streaming parser
> This streaming parser outputs JSON Lines (cli) or returns an Iterable of
> Dictionaries (module)
This parser will handle the Common Log Format standard as specified at
https://www.w3.org/Daemon/User/Config/Logging.html#common-logfile-format.
Combined Log Format is also supported. (Referer and User Agent fields added)
Extra fields may be present and will be enclosed in the `extra` field as
a single string.
If a log line cannot be parsed, an object with an `unparsable` field will
be present with a value of the original line.
The `epoch` calculated timestamp field is naive. (i.e. based on the
local time of the system the parser is run on)
The `epoch_utc` calculated timestamp field is timezone-aware and is
only available if the timezone field is UTC.
Usage (cli):
$ cat file.log | jc --clf-s
Usage (module):
import jc
result = jc.parse('clf_s', common_log_file_output.splitlines())
for item in result:
# do something
Schema:
Empty strings and `-` values are converted to `null`/`None`.
{
"host": string,
"ident": string,
"authuser": string,
"date": string,
"day": integer,
"month": string,
"year": integer,
"hour": integer,
"minute": integer,
"second": integer,
"tz": string,
"request": string,
"request_method": string,
"request_url": string,
"request_version": string,
"status": integer,
"bytes": integer,
"referer": string,
"user_agent": string,
"extra": string,
"epoch": integer, # [0]
"epoch_utc": integer, # [1]
"unparsable": string # [2]
}
[0] naive timestamp
[1] timezone-aware timestamp. Only available if timezone field is UTC
[2] exists if the line was not able to be parsed
Examples:
$ cat file.log | jc --clf-s
{"host":"127.0.0.1","ident":"user-identifier","authuser":"frank","...}
{"host":"1.1.1.2","ident":null,"authuser":null,"date":"11/Nov/2016...}
...
$ cat file.log | jc --clf-s -r
{"host":"127.0.0.1","ident":"user-identifier","authuser":"frank","...}
{"host":"1.1.1.2","ident":"-","authuser":"-","date":"11/Nov/2016:0...}
...
<a id="jc.parsers.clf_s.parse"></a>
### parse
```python
@add_jc_meta
def parse(data: Iterable[str],
raw: bool = False,
quiet: bool = False,
ignore_exceptions: bool = False) -> StreamingOutputType
```
Main text parsing generator function. Returns an iterable object.
Parameters:
data: (iterable) line-based text data to parse
(e.g. sys.stdin or str.splitlines())
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
ignore_exceptions: (boolean) ignore parsing exceptions if True
Returns:
Iterable of Dictionaries
### Parser Information
Compatibility: linux, darwin, cygwin, win32, aix, freebsd
Version 1.0 by Kelly Brazil (kellyjonbrazil@gmail.com)

View File

@@ -82,7 +82,9 @@ Examples:
### parse
```python
def parse(data, raw=False, quiet=False)
def parse(data: Union[str, bytes],
raw: bool = False,
quiet: bool = False) -> List[JSONDictType]
```
Main text parsing function
@@ -100,4 +102,4 @@ Returns:
### Parser Information
Compatibility: linux, darwin, cygwin, win32, aix, freebsd
Version 1.4 by Kelly Brazil (kellyjonbrazil@gmail.com)
Version 1.5 by Kelly Brazil (kellyjonbrazil@gmail.com)

View File

@@ -86,4 +86,4 @@ Returns:
### Parser Information
Compatibility: linux, darwin, cygwin, win32, aix, freebsd
Version 1.3 by Kelly Brazil (kellyjonbrazil@gmail.com)
Version 1.4 by Kelly Brazil (kellyjonbrazil@gmail.com)

View File

@@ -0,0 +1,91 @@
[Home](https://kellyjonbrazil.github.io/jc/)
<a id="jc.parsers.datetime_iso"></a>
# jc.parsers.datetime\_iso
jc - JSON Convert ISO 8601 Datetime string parser
This parser supports standard ISO 8601 strings that include both date and
time. If no timezone or offset information is available in the string, then
UTC timezone is used.
Usage (cli):
$ echo "2022-07-20T14:52:45Z" | jc --iso-datetime
Usage (module):
import jc
result = jc.parse('iso_datetime', iso_8601_string)
Schema:
{
"year": integer,
"month": string,
"month_num": integer,
"day": integer,
"weekday": string,
"weekday_num": integer,
"hour": integer,
"hour_24": integer,
"minute": integer,
"second": integer,
"microsecond": integer,
"period": string,
"utc_offset": string,
"day_of_year": integer,
"week_of_year": integer,
"iso": string,
"timestamp": integer # [0]
}
[0] timezone aware UNIX timestamp expressed in UTC
Examples:
$ echo "2022-07-20T14:52:45Z" | jc --iso-datetime -p
{
"year": 2022,
"month": "Jul",
"month_num": 7,
"day": 20,
"weekday": "Wed",
"weekday_num": 3,
"hour": 2,
"hour_24": 14,
"minute": 52,
"second": 45,
"microsecond": 0,
"period": "PM",
"utc_offset": "+0000",
"day_of_year": 201,
"week_of_year": 29,
"iso": "2022-07-20T14:52:45+00:00",
"timestamp": 1658328765
}
<a id="jc.parsers.datetime_iso.parse"></a>
### parse
```python
def parse(data, raw=False, quiet=False)
```
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
Returns:
Dictionary. Raw or processed structured data.
### Parser Information
Compatibility: linux, aix, freebsd, darwin, win32, cygwin
Version 1.0 by Kelly Brazil (kellyjonbrazil@gmail.com)

View File

@@ -106,7 +106,7 @@ Schema:
]
[0] naive timestamp if "when" field is parsable, else null
[1] timezone aware timestamp availabe for UTC, else null
[1] timezone aware timestamp available for UTC, else null
Examples:

View File

@@ -5,6 +5,10 @@
jc - JSON Convert `du` command output parser
The `du -h` option is not supported with the default output. If you
would like to use `du -h` or other options that change the output, be sure
to use `jc --raw` (cli) or `raw=True` (module).
Usage (cli):
$ du | jc --du

117
docs/parsers/findmnt.md Normal file
View File

@@ -0,0 +1,117 @@
[Home](https://kellyjonbrazil.github.io/jc/)
<a id="jc.parsers.findmnt"></a>
# jc.parsers.findmnt
jc - JSON Convert `findmnt` command output parser
Supports `-a`, `-l`, or no `findmnt` options.
> Note: Newer versions of `findmnt` have a JSON output option.
Usage (cli):
$ findmnt | jc --findmnt
or
$ jc findmnt
Usage (module):
import jc
result = jc.parse('findmnt', findmnt_command_output)
Schema:
[
{
"target": string,
"source": string,
"fstype": string,
"options": [
string
],
"kv_options": {
"<key_name>": string
}
]
Examples:
$ findmnt | jc --findmnt -p
[
{
"target": "/",
"source": "/dev/mapper/centos-root",
"fstype": "xfs",
"options": [
"rw",
"relatime",
"seclabel",
"attr2",
"inode64",
"noquota"
]
},
{
"target": "/sys/fs/cgroup",
"source": "tmpfs",
"fstype": "tmpfs",
"options": [
"ro",
"nosuid",
"nodev",
"noexec",
"seclabel"
],
"kv_options": {
"mode": "755"
}
},
...
]
$ findmnt | jc --findmnt -p -r
[
{
"target": "/",
"source": "/dev/mapper/centos-root",
"fstype": "xfs",
"options": "rw,relatime,seclabel,attr2,inode64,noquota"
},
{
"target": "/sys/fs/cgroup",
"source": "tmpfs",
"fstype": "tmpfs",
"options": "ro,nosuid,nodev,noexec,seclabel,mode=755"
},
...
]
<a id="jc.parsers.findmnt.parse"></a>
### parse
```python
def parse(data: str,
raw: bool = False,
quiet: bool = False) -> List[JSONDictType]
```
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
Returns:
List of Dictionaries. Raw or processed structured data.
### Parser Information
Compatibility: linux
Version 1.0 by Kelly Brazil (kellyjonbrazil@gmail.com)

View File

@@ -40,13 +40,13 @@ Schema:
[
{
"commit": string,
"author": string,
"author_email": string,
"author": string/null,
"author_email": string/null,
"date": string,
"epoch": integer, # [0]
"epoch_utc": integer, # [1]
"commit_by": string,
"commit_by_email": string,
"commit_by": string/null,
"commit_by_email": string/null,
"commit_by_date": string,
"message": string,
"stats" : {
@@ -61,7 +61,7 @@ Schema:
]
[0] naive timestamp if "date" field is parsable, else null
[1] timezone aware timestamp availabe for UTC, else null
[1] timezone aware timestamp available for UTC, else null
Examples:
@@ -172,4 +172,4 @@ Returns:
### Parser Information
Compatibility: linux, darwin, cygwin, win32, aix, freebsd
Version 1.2 by Kelly Brazil (kellyjonbrazil@gmail.com)
Version 1.3 by Kelly Brazil (kellyjonbrazil@gmail.com)

View File

@@ -41,13 +41,13 @@ Schema:
{
"commit": string,
"author": string,
"author_email": string,
"author": string/null,
"author_email": string/null,
"date": string,
"epoch": integer, # [0]
"epoch_utc": integer, # [1]
"commit_by": string,
"commit_by_email": string,
"commit_by": string/null,
"commit_by_email": string/null,
"commit_by_date": string,
"message": string,
"stats" : {
@@ -68,7 +68,7 @@ Schema:
}
[0] naive timestamp if "date" field is parsable, else null
[1] timezone aware timestamp availabe for UTC, else null
[1] timezone aware timestamp available for UTC, else null
Examples:
@@ -108,4 +108,4 @@ Returns:
### Parser Information
Compatibility: linux, darwin, cygwin, win32, aix, freebsd
Version 1.2 by Kelly Brazil (kellyjonbrazil@gmail.com)
Version 1.3 by Kelly Brazil (kellyjonbrazil@gmail.com)

View File

@@ -0,0 +1,92 @@
[Home](https://kellyjonbrazil.github.io/jc/)
<a id="jc.parsers.git_ls_remote"></a>
# jc.parsers.git\_ls\_remote
jc - JSON Convert `git ls-remote` command output parser
This parser outputs two schemas:
- Default: A single object with key/value pairs
- Raw: An array of objects (`--raw` (cli) or `raw=True (module))
See the Schema section for more details
Usage (cli):
$ git ls-remote | jc --git-ls-remote
or
$ jc git ls-remote
Usage (module):
import jc
result = jc.parse('git_ls_remote', git_ls_remote_command_output)
Schema:
Default:
{
<reference>: string
}
Raw:
[
{
"reference": string,
"commit": string
}
]
Examples:
$ git ls-remote | jc --git-ls-remote -p
{
"HEAD": "214cd6b9e09603b3c4fa02203b24fb2bc3d4e338",
"refs/heads/dev": "b884f6aacca39e05994596d8fdfa7e7c4f1e0389",
"refs/heads/master": "214cd6b9e09603b3c4fa02203b24fb2bc3d4e338",
"refs/pull/1/head": "e416c77bed1267254da972b0f95b7ff1d43fccef",
...
}
$ git ls-remote | jc --git-ls-remote -p -r
[
{
"reference": "HEAD",
"commit": "214cd6b9e09603b3c4fa02203b24fb2bc3d4e338"
},
{
"reference": "refs/heads/dev",
"commit": "b884f6aacca39e05994596d8fdfa7e7c4f1e0389"
},
...
]
<a id="jc.parsers.git_ls_remote.parse"></a>
### parse
```python
def parse(data: str,
raw: bool = False,
quiet: bool = False) -> Union[JSONDictType, List[JSONDictType]]
```
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
Returns:
Dictionary (default) or List of Dictionaries (raw)
### Parser Information
Compatibility: linux, darwin, cygwin, win32, aix, freebsd
Version 1.0 by Kelly Brazil (kellyjonbrazil@gmail.com)

View File

@@ -5,7 +5,10 @@
jc - JSON Convert `ifconfig` command output parser
> Note: No `ifconfig` options are supported.
No `ifconfig` options are supported.
Consider using the `ip` command instead of `ifconfig` as it supports native
JSON output.
Usage (cli):
@@ -24,40 +27,92 @@ Schema:
[
{
"name": string,
"flags": integer,
"name": string,
"type": string,
"metric": integer
"flags": integer,
"state": [
string
string
],
"mtu": integer,
"ipv4_addr": string,
"ipv4_mask": string,
"ipv4_bcast": string,
"ipv6_addr": string,
"ipv6_mask": integer,
"ipv6_scope": string,
"mac_addr": string,
"type": string,
"rx_packets": integer,
"rx_bytes": integer,
"rx_errors": integer,
"rx_dropped": integer,
"rx_overruns": integer,
"rx_frame": integer,
"tx_packets": integer,
"tx_bytes": integer,
"tx_errors": integer,
"tx_dropped": integer,
"tx_overruns": integer,
"tx_carrier": integer,
"tx_collisions": integer,
"metric": integer
"mtu": integer,
"mac_addr": string,
"ipv4_addr": string, # [0]
"ipv4_mask": string, # [0]
"ipv4_bcast": string, # [0]
"ipv6_addr": string, # [0]
"ipv6_mask": integer, # [0]
"ipv6_scope": string, # [0]
"ipv6_scope_id": string, # [0]
"ipv6_type": string, # [0]
"rx_packets": integer,
"rx_bytes": integer,
"rx_errors": integer,
"rx_dropped": integer,
"rx_overruns": integer,
"rx_frame": integer,
"tx_packets": integer,
"tx_bytes": integer,
"tx_errors": integer,
"tx_dropped": integer,
"tx_overruns": integer,
"tx_carrier": integer,
"tx_collisions": integer,
"options": string,
"options_flags": [
string
],
"status": string,
"hw_address": string,
"media": string,
"media_flags": [
string
],
"nd6_options": integer,
"nd6_flags": [
string
],
"plugged": string,
"vendor": string,
"vendor_pn": string,
"vendor_sn": string,
"vendor_date": string,
"module_temperature": string,
"module_voltage": string
"ipv4": [
{
"address": string,
"mask": string,
"broadcast": string
}
],
"ipv6: [
{
"address": string,
"scope_id": string,
"mask": integer,
"scope": string,
"type": string
}
],
"lanes": [
{
"lane": integer,
"rx_power_mw": float,
"rx_power_dbm": float,
"tx_bias_ma": float
}
]
}
]
[0] these fields only pick up the last IP address in the interface
output and are here for backwards compatibility. For information on
all IP addresses, use the `ipv4` and `ipv6` objects which contain an
array of IP address objects.
Examples:
$ ifconfig | jc --ifconfig -p
$ ifconfig ens33 | jc --ifconfig -p
[
{
"name": "ens33",
@@ -69,120 +124,94 @@ Examples:
"MULTICAST"
],
"mtu": 1500,
"type": "Ethernet",
"mac_addr": "00:0c:29:3b:58:0e",
"ipv4_addr": "192.168.71.137",
"ipv4_mask": "255.255.255.0",
"ipv4_bcast": "192.168.71.255",
"ipv6_addr": "fe80::c1cb:715d:bc3e:b8a0",
"ipv6_mask": 64,
"ipv6_scope": "0x20",
"mac_addr": "00:0c:29:3b:58:0e",
"type": "Ethernet",
"ipv6_type": "link",
"metric": null,
"rx_packets": 8061,
"rx_bytes": 1514413,
"rx_errors": 0,
"rx_dropped": 0,
"rx_overruns": 0,
"rx_frame": 0,
"tx_packets": 4502,
"tx_errors": 0,
"tx_dropped": 0,
"tx_overruns": 0,
"tx_carrier": 0,
"tx_collisions": 0,
"rx_bytes": 1514413,
"tx_bytes": 866622,
"tx_errors": 0,
"tx_dropped": 0,
"tx_overruns": 0,
"tx_carrier": 0,
"tx_collisions": 0,
"metric": null
},
{
"name": "lo",
"flags": 73,
"state": [
"UP",
"LOOPBACK",
"RUNNING"
"ipv4": [
{
"address": "192.168.71.137",
"mask": "255.255.255.0",
"broadcast": "192.168.71.255"
}
],
"mtu": 65536,
"ipv4_addr": "127.0.0.1",
"ipv4_mask": "255.0.0.0",
"ipv4_bcast": null,
"ipv6_addr": "::1",
"ipv6_mask": 128,
"ipv6_scope": "0x10",
"mac_addr": null,
"type": "Local Loopback",
"rx_packets": 73,
"rx_bytes": 6009,
"rx_errors": 0,
"rx_dropped": 0,
"rx_overruns": 0,
"rx_frame": 0,
"tx_packets": 73,
"tx_bytes": 6009,
"tx_errors": 0,
"tx_dropped": 0,
"tx_overruns": 0,
"tx_carrier": 0,
"tx_collisions": 0,
"metric": null
"ipv6": [
{
"address": "fe80::c1cb:715d:bc3e:b8a0",
"scope_id": null,
"mask": 64,
"scope": "0x20",
"type": "link"
}
]
}
]
$ ifconfig | jc --ifconfig -p -r
$ ifconfig ens33 | jc --ifconfig -p -r
[
{
"name": "ens33",
"flags": "4163",
"state": "UP,BROADCAST,RUNNING,MULTICAST",
"mtu": "1500",
"type": "Ethernet",
"mac_addr": "00:0c:29:3b:58:0e",
"ipv4_addr": "192.168.71.137",
"ipv4_mask": "255.255.255.0",
"ipv4_bcast": "192.168.71.255",
"ipv6_addr": "fe80::c1cb:715d:bc3e:b8a0",
"ipv6_mask": "64",
"ipv6_scope": "0x20",
"mac_addr": "00:0c:29:3b:58:0e",
"type": "Ethernet",
"ipv6_type": "link",
"metric": null,
"rx_packets": "8061",
"rx_bytes": "1514413",
"rx_errors": "0",
"rx_dropped": "0",
"rx_overruns": "0",
"rx_frame": "0",
"tx_packets": "4502",
"tx_errors": "0",
"tx_dropped": "0",
"tx_overruns": "0",
"tx_carrier": "0",
"tx_collisions": "0",
"rx_bytes": "1514413",
"tx_bytes": "866622",
"tx_errors": "0",
"tx_dropped": "0",
"tx_overruns": "0",
"tx_carrier": "0",
"tx_collisions": "0",
"metric": null
},
{
"name": "lo",
"flags": "73",
"state": "UP,LOOPBACK,RUNNING",
"mtu": "65536",
"ipv4_addr": "127.0.0.1",
"ipv4_mask": "255.0.0.0",
"ipv4_bcast": null,
"ipv6_addr": "::1",
"ipv6_mask": "128",
"ipv6_scope": "0x10",
"mac_addr": null,
"type": "Local Loopback",
"rx_packets": "73",
"rx_bytes": "6009",
"rx_errors": "0",
"rx_dropped": "0",
"rx_overruns": "0",
"rx_frame": "0",
"tx_packets": "73",
"tx_bytes": "6009",
"tx_errors": "0",
"tx_dropped": "0",
"tx_overruns": "0",
"tx_carrier": "0",
"tx_collisions": "0",
"metric": null
"ipv4": [
{
"address": "192.168.71.137",
"mask": "255.255.255.0",
"broadcast": "192.168.71.255"
}
],
"ipv6": [
{
"address": "fe80::c1cb:715d:bc3e:b8a0",
"scope_id": null,
"mask": "64",
"scope": "0x20",
"type": "link"
}
]
}
]
@@ -191,7 +220,9 @@ Examples:
### parse
```python
def parse(data, raw=False, quiet=False)
def parse(data: str,
raw: bool = False,
quiet: bool = False) -> List[JSONDictType]
```
Main text parsing function
@@ -209,4 +240,4 @@ Returns:
### Parser Information
Compatibility: linux, aix, freebsd, darwin
Version 1.12 by Kelly Brazil (kellyjonbrazil@gmail.com)
Version 2.1 by Kelly Brazil (kellyjonbrazil@gmail.com)

View File

@@ -130,4 +130,4 @@ Returns:
### Parser Information
Compatibility: linux
Version 1.2 by Kelly Brazil (kellyjonbrazil@gmail.com)
Version 1.3 by Kelly Brazil (kellyjonbrazil@gmail.com)

View File

@@ -26,6 +26,9 @@ Schema:
"ip": string,
"ip_compressed": string,
"ip_exploded": string,
"ip_split": [
string
],
"scope_id": string/null,
"ipv4_mapped": string/null,
"six_to_four": string/null,
@@ -83,6 +86,12 @@ Examples:
"ip": "192.168.2.10",
"ip_compressed": "192.168.2.10",
"ip_exploded": "192.168.2.10",
"ip_split": [
"192",
"168",
"2",
"10"
],
"scope_id": null,
"ipv4_mapped": null,
"six_to_four": null,
@@ -138,6 +147,12 @@ Examples:
"ip": "192.168.2.10",
"ip_compressed": "192.168.2.10",
"ip_exploded": "192.168.2.10",
"ip_split": [
"192",
"168",
"2",
"10"
],
"scope_id": null,
"ipv4_mapped": null,
"six_to_four": null,
@@ -191,14 +206,24 @@ Examples:
"version": 6,
"max_prefix_length": 128,
"ip": "127:0:de::1",
"ip_compressed": "127:0:de::1%128",
"ip_compressed": "127:0:de::1",
"ip_exploded": "0127:0000:00de:0000:0000:0000:0000:0001",
"ip_split": [
"0127",
"0000",
"00de",
"0000",
"0000",
"0000",
"0000",
"0001"
],
"scope_id": "128",
"ipv4_mapped": null,
"six_to_four": null,
"teredo_client": null,
"teredo_server": null,
"dns_ptr": "1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.....0.7.2.1.0.ip6.arpa",
"dns_ptr": "1.0.0.0.0.0...0.0.0.e.d.0.0.0.0.0.0.7.2.1.0.ip6.arpa",
"network": "127:0:de::",
"broadcast": "127:0:de::ffff:ffff",
"hostmask": "::ffff:ffff",
@@ -231,13 +256,13 @@ Examples:
"last_host": "01:27:00:00:00:de:00:00:00:00:00:00:ff:ff:ff:fe"
},
"bin": {
"ip": "000000010010011100000000000000000000000011011110000000...",
"network": "0000000100100111000000000000000000000000110111100...",
"broadcast": "00000001001001110000000000000000000000001101111...",
"hostmask": "000000000000000000000000000000000000000000000000...",
"netmask": "1111111111111111111111111111111111111111111111111...",
"first_host": "0000000100100111000000000000000000000000110111...",
"last_host": "00000001001001110000000000000000000000001101111..."
"ip": "0000000100100111000000000000000000000000110...000000000001",
"network": "00000001001001110000000000000000000000...000000000000",
"broadcast": "000000010010011100000000000000000000...111111111111",
"hostmask": "0000000000000000000000000000000000000...111111111111",
"netmask": "11111111111111111111111111111111111111...000000000000",
"first_host": "00000001001001110000000000000000000...000000000001",
"last_host": "000000010010011100000000000000000000...1111111111110"
}
}
@@ -248,12 +273,22 @@ Examples:
"ip": "127:0:de::1",
"ip_compressed": "127:0:de::1",
"ip_exploded": "0127:0000:00de:0000:0000:0000:0000:0001",
"ip_split": [
"0127",
"0000",
"00de",
"0000",
"0000",
"0000",
"0000",
"0001"
],
"scope_id": null,
"ipv4_mapped": null,
"six_to_four": null,
"teredo_client": null,
"teredo_server": null,
"dns_ptr": "1.0.0.0.0.0.0....0.0.0.e.d.0.0.0.0.0.0.7.2.1.0.ip6.arpa",
"dns_ptr": "1.0.0.0.0.0....0.0.0.0.e.d.0.0.0.0.0.0.7.2.1.0.ip6.arpa",
"network": "127:0:de::1",
"broadcast": "127:0:de::1",
"hostmask": "::",
@@ -286,13 +321,13 @@ Examples:
"last_host": "01:27:00:00:00:de:00:00:00:00:00:00:00:00:00:01"
},
"bin": {
"ip": "0000000100100111000000000000000000000000110111100000000...",
"network": "00000001001001110000000000000000000000001101111000...",
"broadcast": "000000010010011100000000000000000000000011011110...",
"hostmask": "0000000000000000000000000000000000000000000000000...",
"netmask": "11111111111111111111111111111111111111111111111111...",
"first_host": "00000001001001110000000000000000000000001101111...",
"last_host": "000000010010011100000000000000000000000011011110..."
"ip": "0000000100100111000000000000000000000000110111100...000001",
"network": "00000001001001110000000000000000000000001101...000001",
"broadcast": "000000010010011100000000000000000000000011...000001",
"hostmask": "0000000000000000000000000000000000000000000...000000",
"netmask": "11111111111111111111111111111111111111111111...111111",
"first_host": "00000001001001110000000000000000000000001...000001",
"last_host": "000000010010011100000000000000000000000011...0000001"
}
}
@@ -304,12 +339,22 @@ Examples:
"ip": "::ffff:c0a8:123",
"ip_compressed": "::ffff:c0a8:123",
"ip_exploded": "0000:0000:0000:0000:0000:ffff:c0a8:0123",
"ip_split": [
"0000",
"0000",
"0000",
"0000",
"0000",
"ffff",
"c0a8",
"0123"
],
"scope_id": null,
"ipv4_mapped": "192.168.1.35",
"six_to_four": null,
"teredo_client": null,
"teredo_server": null,
"dns_ptr": "3.2.1.0.8.a.0.c.f.f.f.f.0.0.0....0.0.0.0.0.0.0.ip6.arpa",
"dns_ptr": "3.2.1.0.8.a.0.c.f.f.f.f.0.0....0.0.0.0.0.0.ip6.arpa",
"network": "::ffff:c0a8:123",
"broadcast": "::ffff:c0a8:123",
"hostmask": "::",
@@ -342,13 +387,13 @@ Examples:
"last_host": "00:00:00:00:00:00:00:00:00:00:ff:ff:c0:a8:01:23"
},
"bin": {
"ip": "0000000000000000000000000000000000000000000000000000000...",
"network": "00000000000000000000000000000000000000000000000000...",
"broadcast": "000000000000000000000000000000000000000000000000...",
"hostmask": "0000000000000000000000000000000000000000000000000...",
"netmask": "11111111111111111111111111111111111111111111111111...",
"first_host": "00000000000000000000000000000000000000000000000...",
"last_host": "000000000000000000000000000000000000000000000000..."
"ip": "000000000000000000000000000000000000000000000...100100011",
"network": "0000000000000000000000000000000000000000...000100011",
"broadcast": "00000000000000000000000000000000000000...000100011",
"hostmask": "000000000000000000000000000000000000000...000000000",
"netmask": "1111111111111111111111111111111111111111...111111111",
"first_host": "0000000000000000000000000000000000000...100100011",
"last_host": "00000000000000000000000000000000000000...0100100011"
}
}
@@ -360,12 +405,22 @@ Examples:
"ip": "2002:c000:204::",
"ip_compressed": "2002:c000:204::",
"ip_exploded": "2002:c000:0204:0000:0000:0000:0000:0000",
"ip_split": [
"2002",
"c000",
"0204",
"0000",
"0000",
"0000",
"0000",
"0000"
],
"scope_id": null,
"ipv4_mapped": null,
"six_to_four": "192.0.2.4",
"teredo_client": null,
"teredo_server": null,
"dns_ptr": "0.0.0.0.0.0.0.0......0.4.0.2.0.0.0.0.c.2.0.0.2.ip6.arpa",
"dns_ptr": "0.0.0.0.0.0.0...0.0.0.4.0.2.0.0.0.0.c.2.0.0.2.ip6.arpa",
"network": "2002:c000:204::",
"broadcast": "2002:c000:204:ffff:ffff:ffff:ffff:ffff",
"hostmask": "::ffff:ffff:ffff:ffff:ffff",
@@ -398,13 +453,13 @@ Examples:
"last_host": "20:02:c0:00:02:04:ff:ff:ff:ff:ff:ff:ff:ff:ff:fe"
},
"bin": {
"ip": "0010000000000010110000000000000000000010000001000000000...",
"network": "00100000000000101100000000000000000000100000010000...",
"broadcast": "001000000000001011000000000000000000001000000100...",
"hostmask": "0000000000000000000000000000000000000000000000001...",
"netmask": "11111111111111111111111111111111111111111111111100...",
"first_host": "00100000000000101100000000000000000000100000010...",
"last_host": "001000000000001011000000000000000000001000000100..."
"ip": "00100000000000101100000000000000000000100000010...00000000",
"network": "001000000000001011000000000000000000001000...00000000",
"broadcast": "0010000000000010110000000000000000000010...11111111",
"hostmask": "00000000000000000000000000000000000000000...11111111",
"netmask": "111111111111111111111111111111111111111111...00000000",
"first_host": "001000000000001011000000000000000000001...00000001",
"last_host": "0010000000000010110000000000000000000010...111111110"
}
}
@@ -416,12 +471,22 @@ Examples:
"ip": "2001:0:4136:e378:8000:63bf:3fff:fdd2",
"ip_compressed": "2001:0:4136:e378:8000:63bf:3fff:fdd2",
"ip_exploded": "2001:0000:4136:e378:8000:63bf:3fff:fdd2",
"ip_split": [
"2001",
"0000",
"4136",
"e378",
"8000",
"63bf",
"3fff",
"fdd2"
],
"scope_id": null,
"ipv4_mapped": null,
"six_to_four": null,
"teredo_client": "192.0.2.45",
"teredo_server": "65.54.227.120",
"dns_ptr": "2.d.d.f.f.f.f.3.f.b.3.6.0.0.0....0.0.0.1.0.0.2.ip6.arpa",
"dns_ptr": "2.d.d.f.f.f.f.3.f.b.3.6.0.0.0.8.8....0.1.0.0.2.ip6.arpa",
"network": "2001:0:4136:e378:8000:63bf:3fff:fdd2",
"broadcast": "2001:0:4136:e378:8000:63bf:3fff:fdd2",
"hostmask": "::",
@@ -454,13 +519,13 @@ Examples:
"last_host": "20:01:00:00:41:36:e3:78:80:00:63:bf:3f:ff:fd:d2"
},
"bin": {
"ip": "001000000000000100000000000000000100000100110110111000...",
"network": "0010000000000001000000000000000001000001001101101...",
"broadcast": "00100000000000010000000000000000010000010011011...",
"hostmask": "000000000000000000000000000000000000000000000000...",
"netmask": "1111111111111111111111111111111111111111111111111...",
"first_host": "0010000000000001000000000000000001000001001101...",
"last_host": "00100000000000010000000000000000010000010011011..."
"ip": "0010000000000001000000000000000001000001001...110111010010",
"network": "00100000000000010000000000000000010000...110111010010",
"broadcast": "001000000000000100000000000000000100...110111010010",
"hostmask": "0000000000000000000000000000000000000...000000000000",
"netmask": "11111111111111111111111111111111111111...111111111111",
"first_host": "00100000000000010000000000000000010...110111010010",
"last_host": "001000000000000100000000000000000100...110111010010"
}
}
@@ -487,4 +552,4 @@ Returns:
### Parser Information
Compatibility: linux, darwin, cygwin, win32, aix, freebsd
Version 1.2 by Kelly Brazil (kellyjonbrazil@gmail.com)
Version 1.3 by Kelly Brazil (kellyjonbrazil@gmail.com)

View File

@@ -5,65 +5,10 @@
jc - JSON Convert ISO 8601 Datetime string parser
This parser supports standard ISO 8601 strings that include both date and
time. If no timezone or offset information is available in the sring, then
UTC timezone is used.
This parser has been renamed to datetime-iso (cli) or datetime_iso (module).
Usage (cli):
$ echo "2022-07-20T14:52:45Z" | jc --iso-datetime
Usage (module):
import jc
result = jc.parse('iso_datetime', iso_8601_string)
Schema:
{
"year": integer,
"month": string,
"month_num": integer,
"day": integer,
"weekday": string,
"weekday_num": integer,
"hour": integer,
"hour_24": integer,
"minute": integer,
"second": integer,
"microsecond": integer,
"period": string,
"utc_offset": string,
"day_of_year": integer,
"week_of_year": integer,
"iso": string,
"timestamp": integer # [0]
}
[0] timezone aware UNIX timestamp expressed in UTC
Examples:
$ echo "2022-07-20T14:52:45Z" | jc --iso-datetime -p
{
"year": 2022,
"month": "Jul",
"month_num": 7,
"day": 20,
"weekday": "Wed",
"weekday_num": 3,
"hour": 2,
"hour_24": 14,
"minute": 52,
"second": 45,
"microsecond": 0,
"period": "PM",
"utc_offset": "+0000",
"day_of_year": 201,
"week_of_year": 29,
"iso": "2022-07-20T14:52:45+00:00",
"timestamp": 1658328765
}
This parser will be removed in a future version, so please start using
the new parser name.
<a id="jc.parsers.iso_datetime.parse"></a>
@@ -73,7 +18,8 @@ Examples:
def parse(data, raw=False, quiet=False)
```
Main text parsing function
This parser is deprecated and calls datetime_iso. Please use datetime_iso
directly. This parser will be removed in the future.
Parameters:
@@ -88,4 +34,4 @@ Returns:
### Parser Information
Compatibility: linux, aix, freebsd, darwin, win32, cygwin
Version 1.0 by Kelly Brazil (kellyjonbrazil@gmail.com)
Version 1.1 by Kelly Brazil (kellyjonbrazil@gmail.com)

View File

@@ -99,4 +99,4 @@ Returns:
### Parser Information
Compatibility: linux, darwin, cygwin, aix, freebsd
Version 1.1 by Kelly Brazil (kellyjonbrazil@gmail.com)
Version 1.2 by Kelly Brazil (kellyjonbrazil@gmail.com)

148
docs/parsers/lspci.md Normal file
View File

@@ -0,0 +1,148 @@
[Home](https://kellyjonbrazil.github.io/jc/)
<a id="jc.parsers.lspci"></a>
# jc.parsers.lspci
jc - JSON Convert `lspci -mmv` command output parser
This parser supports the following `lspci` options:
- `-mmv`
- `-nmmv`
- `-nnmmv`
Usage (cli):
$ lspci -nnmmv | jc --lspci
or
$ jc lspci -nnmmv
Usage (module):
import jc
result = jc.parse('lspci', lspci_command_output)
Schema:
[
{
"slot": string,
"domain": string,
"domain_int": integer,
"bus": string,
"bus_int": integer,
"dev": string,
"dev_int": integer,
"function": string,
"function_int": integer,
"class": string,
"class_id": string,
"class_id_int": integer,
"vendor": string,
"vendor_id": string,
"vendor_id_int": integer,
"device": string,
"device_id": string,
"device_id_int": integer,
"svendor": string,
"svendor_id": string,
"svendor_id_int": integer,
"sdevice": string,
"sdevice_id": string,
"sdevice_id_int": integer,
"rev": string,
"physlot": string,
"physlot_int": integer,
"progif": string,
"progif_int": integer
}
]
Examples:
$ lspci -nnmmv | jc --lspci -p
[
{
"slot": "ff:02:05.0",
"domain": "ff",
"domain_int": 255,
"bus": "02",
"bus_int": 2,
"dev": "05",
"dev_int": 5,
"function": "0",
"function_int": 0,
"class": "SATA controller",
"class_id": "0106",
"class_id_int": 262,
"vendor": "VMware",
"vendor_id": "15ad",
"vendor_id_int": 5549,
"device": "SATA AHCI controller",
"device_id": "07e0",
"device_id_int": 2016,
"svendor": "VMware",
"svendor_id": "15ad",
"svendor_id_int": 5549,
"sdevice": "SATA AHCI controller",
"sdevice_id": "07e0",
"sdevice_id_int": 2016,
"physlot": "37",
"physlot_int": 55,
"progif": "01",
"progif_int": 1
},
...
]
$ lspci -nnmmv | jc --lspci -p -r
[
{
"slot": "ff:02:05.0",
"domain": "ff",
"bus": "02",
"dev": "05",
"function": "0",
"class": "SATA controller",
"class_id": "0106",
"vendor": "VMware",
"vendor_id": "15ad",
"device": "SATA AHCI controller",
"device_id": "07e0",
"svendor": "VMware",
"svendor_id": "15ad",
"sdevice": "SATA AHCI controller",
"sdevice_id": "07e0",
"physlot": "37",
"progif": "01"
},
...
]
<a id="jc.parsers.lspci.parse"></a>
### parse
```python
def parse(data: str,
raw: bool = False,
quiet: bool = False) -> List[JSONDictType]
```
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
Returns:
List of Dictionaries. Raw or processed structured data.
### Parser Information
Compatibility: linux
Version 1.0 by Kelly Brazil (kellyjonbrazil@gmail.com)

178
docs/parsers/openvpn.md Normal file
View File

@@ -0,0 +1,178 @@
[Home](https://kellyjonbrazil.github.io/jc/)
<a id="jc.parsers.openvpn"></a>
# jc.parsers.openvpn
jc - JSON Convert openvpn-status.log file parser
The `*_epoch` calculated timestamp fields are naive. (i.e. based on
the local time of the system the parser is run on)
Usage (cli):
$ cat openvpn-status.log | jc --openvpn
Usage (module):
import jc
result = jc.parse('openvpn', openvpn_status_log_file_output)
Schema:
{
"clients": [
{
"common_name": string,
"real_address": string,
"real_address_prefix": integer, # [0]
"real_address_port": integer, # [0]
"bytes_received": integer,
"bytes_sent": integer,
"connected_since": string,
"connected_since_epoch": integer,
"updated": string,
"updated_epoch": integer,
}
],
"routing_table": [
{
"virtual_address": string,
"virtual_address_prefix": integer, # [0]
"virtual_address_port": integer, # [0]
"common_name": string,
"real_address": string,
"real_address_prefix": integer, # [0]
"real_address_port": integer, # [0]
"last_reference": string,
"last_reference_epoch": integer,
}
],
"global_stats": {
"max_bcast_mcast_queue_len": integer
}
}
[0] null/None if not found
Examples:
$ cat openvpn-status.log | jc --openvpn -p
{
"clients": [
{
"common_name": "foo@example.com",
"real_address": "10.10.10.10",
"bytes_received": 334948,
"bytes_sent": 1973012,
"connected_since": "Thu Jun 18 04:23:03 2015",
"updated": "Thu Jun 18 08:12:15 2015",
"real_address_prefix": null,
"real_address_port": 49502,
"connected_since_epoch": 1434626583,
"updated_epoch": 1434640335
},
{
"common_name": "foo@example.com",
"real_address": "10.10.10.10",
"bytes_received": 334948,
"bytes_sent": 1973012,
"connected_since": "Thu Jun 18 04:23:03 2015",
"updated": "Thu Jun 18 08:12:15 2015",
"real_address_prefix": null,
"real_address_port": 49503,
"connected_since_epoch": 1434626583,
"updated_epoch": 1434640335
}
],
"routing_table": [
{
"virtual_address": "192.168.255.118",
"common_name": "baz@example.com",
"real_address": "10.10.10.10",
"last_reference": "Thu Jun 18 08:12:09 2015",
"virtual_address_prefix": null,
"virtual_address_port": null,
"real_address_prefix": null,
"real_address_port": 63414,
"last_reference_epoch": 1434640329
},
{
"virtual_address": "10.200.0.0",
"common_name": "baz@example.com",
"real_address": "10.10.10.10",
"last_reference": "Thu Jun 18 08:12:09 2015",
"virtual_address_prefix": 16,
"virtual_address_port": null,
"real_address_prefix": null,
"real_address_port": 63414,
"last_reference_epoch": 1434640329
}
],
"global_stats": {
"max_bcast_mcast_queue_len": 0
}
}
$ cat openvpn-status.log | jc --openvpn -p -r
{
"clients": [
{
"common_name": "foo@example.com",
"real_address": "10.10.10.10:49502",
"bytes_received": "334948",
"bytes_sent": "1973012",
"connected_since": "Thu Jun 18 04:23:03 2015",
"updated": "Thu Jun 18 08:12:15 2015"
},
{
"common_name": "foo@example.com",
"real_address": "10.10.10.10:49503",
"bytes_received": "334948",
"bytes_sent": "1973012",
"connected_since": "Thu Jun 18 04:23:03 2015",
"updated": "Thu Jun 18 08:12:15 2015"
}
],
"routing_table": [
{
"virtual_address": "192.168.255.118",
"common_name": "baz@example.com",
"real_address": "10.10.10.10:63414",
"last_reference": "Thu Jun 18 08:12:09 2015"
},
{
"virtual_address": "10.200.0.0/16",
"common_name": "baz@example.com",
"real_address": "10.10.10.10:63414",
"last_reference": "Thu Jun 18 08:12:09 2015"
}
],
"global_stats": {
"max_bcast_mcast_queue_len": "0"
}
}
<a id="jc.parsers.openvpn.parse"></a>
### parse
```python
def parse(data: str, raw: bool = False, quiet: bool = False) -> JSONDictType
```
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
Returns:
Dictionary. Raw or processed structured data.
### Parser Information
Compatibility: linux, darwin, cygwin, win32, aix, freebsd
Version 1.0 by Kelly Brazil (kellyjonbrazil@gmail.com)

66
docs/parsers/os_prober.md Normal file
View File

@@ -0,0 +1,66 @@
[Home](https://kellyjonbrazil.github.io/jc/)
<a id="jc.parsers.os_prober"></a>
# jc.parsers.os\_prober
jc - JSON Convert `os-prober` command output parser
Usage (cli):
$ os-prober | jc --os-prober
or
$ jc os-prober
Usage (module):
import jc
result = jc.parse('os_prober', os_prober_command_output)
Schema:
{
"partition": string,
"efi_bootmgr": string, # [0]
"name": string,
"short_name": string,
"type": string
}
[0] only exists if an EFI boot manager is detected
Examples:
$ os-prober | jc --os-prober -p
{
"partition": "/dev/sda1",
"name": "Windows 10",
"short_name": "Windows",
"type": "chain"
}
<a id="jc.parsers.os_prober.parse"></a>
### parse
```python
def parse(data: str, raw: bool = False, quiet: bool = False) -> JSONDictType
```
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
Returns:
Dictionary. Raw or processed structured data.
### Parser Information
Compatibility: linux
Version 1.1 by Kelly Brazil (kellyjonbrazil@gmail.com)

99
docs/parsers/pci_ids.md Normal file
View File

@@ -0,0 +1,99 @@
[Home](https://kellyjonbrazil.github.io/jc/)
<a id="jc.parsers.pci_ids"></a>
# jc.parsers.pci\_ids
jc - JSON Convert `pci.ids` file parser
This parser converts the pci.ids database file.
https://raw.githubusercontent.com/pciutils/pciids/master/pci.ids
A nested schema allows straightforward queries with tools like `jq`. Hex id
numbers are prefixed with an underscore (`_`) so bracket notation is not
necessary when referencing. For example:
$ cat pci.ids | jc --pci-ids | jq '.vendors._9005._0053._9005._ffff.subsystem_name'
"AIC-7896 SCSI Controller mainboard implementation"
Here are the vendor and class mappings:
jq '.vendors._001c._0001._001c._0005.subsystem_name'
| | | |
| | | subdevice
| | subvendor
| device
vendor
jq '.classes._0c._03._40'
| | |
| | prog_if
| subclass
class
Usage (cli):
$ cat pci.ids | jc --pci-ids
Usage (module):
import jc
result = jc.parse('pci_ids', pci_ids_file_output)
Schema:
{
"vendors": {
"_<vendor_id>": {
"vendor_name": string,
"_<device_id>": {
"device_name": string,
"_<subvendor_id>": {
"_<subdevice_id": string
}
}
}
},
"classes": {
"_<class_id>": {
"class_name": string,
"_<subclass_id>": {
"subclass_name": string,
"_<prog_if>": string
}
}
}
}
Examples:
$ cat pci.ids | jc --pci-ids | jq '.vendors._001c._0001._001c._0005.subsystem_name'
"2 Channel CAN Bus SJC1000 (Optically Isolated)"
$ cat pci.ids | jc --pci-ids | jq '.classes._0c._03._40'
"USB4 Host Interface"
<a id="jc.parsers.pci_ids.parse"></a>
### parse
```python
def parse(data: str, raw: bool = False, quiet: bool = False) -> JSONDictType
```
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
Returns:
Dictionary. Raw or processed structured data.
### Parser Information
Compatibility: linux, darwin, cygwin, win32, aix, freebsd
Version 1.0 by Kelly Brazil (kellyjonbrazil@gmail.com)

75
docs/parsers/pgpass.md Normal file
View File

@@ -0,0 +1,75 @@
[Home](https://kellyjonbrazil.github.io/jc/)
<a id="jc.parsers.pgpass"></a>
# jc.parsers.pgpass
jc - JSON Convert PostgreSQL password file parser
Usage (cli):
$ cat /var/lib/postgresql/.pgpass | jc --pgpass
Usage (module):
import jc
result = jc.parse('pgpass', postgres_password_file)
Schema:
[
{
"hostname": string,
"port": string,
"database": string,
"username": string,
"password": string
}
]
Examples:
$ cat /var/lib/postgresql/.pgpass | jc --pgpass -p
[
{
"hostname": "dbserver",
"port": "*",
"database": "db1",
"username": "dbuser",
"password": "pwd123"
},
{
"hostname": "dbserver2",
"port": "8888",
"database": "inventory",
"username": "joe:user",
"password": "abc123"
},
...
]
<a id="jc.parsers.pgpass.parse"></a>
### parse
```python
def parse(data: str,
raw: bool = False,
quiet: bool = False) -> List[JSONDictType]
```
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
Returns:
List of Dictionaries. Raw or processed structured data.
### Parser Information
Compatibility: linux, darwin, cygwin, win32, aix, freebsd
Version 1.0 by Kelly Brazil (kellyjonbrazil@gmail.com)

View File

@@ -106,4 +106,4 @@ Returns:
### Parser Information
Compatibility: linux, darwin, freebsd
Version 1.1 by Kelly Brazil (kellyjonbrazil@gmail.com)
Version 1.2 by Kelly Brazil (kellyjonbrazil@gmail.com)

View File

@@ -6,7 +6,7 @@
jc - JSON Convert Proc file output parser
This parser automatically identifies the Proc file and calls the
corresponding parser to peform the parsing.
corresponding parser to perform the parsing.
Magic syntax for converting `/proc` files is also supported by running
`jc /proc/<path to file>`. Any `jc` options must be specified before the
@@ -85,7 +85,7 @@ Examples:
...
]
$ proc_modules | jc --proc_modules -p -r
$ cat /proc/modules | jc --proc-modules -p -r
[
{
"module": "binfmt_misc",

View File

@@ -223,4 +223,4 @@ Returns:
### Parser Information
Compatibility: linux
Version 1.0 by Kelly Brazil (kellyjonbrazil@gmail.com)
Version 1.1 by Kelly Brazil (kellyjonbrazil@gmail.com)

View File

@@ -114,4 +114,4 @@ Returns:
### Parser Information
Compatibility: linux, darwin, freebsd
Version 1.1 by Kelly Brazil (kellyjonbrazil@gmail.com)
Version 1.2 by Kelly Brazil (kellyjonbrazil@gmail.com)

75
docs/parsers/semver.md Normal file
View File

@@ -0,0 +1,75 @@
[Home](https://kellyjonbrazil.github.io/jc/)
<a id="jc.parsers.semver"></a>
# jc.parsers.semver
jc - JSON Convert Semantic Version string parser
This parser conforms to the specification at https://semver.org/
Usage (cli):
$ echo 1.2.3-rc.1+44837 | jc --semver
Usage (module):
import jc
result = jc.parse('semver', semver_string)
Schema:
Strings that do not strictly conform to the specification will return an
empty object.
{
"major": integer,
"minor": integer,
"patch": integer,
"prerelease": string/null,
"build": string/null
}
Examples:
$ echo 1.2.3-rc.1+44837 | jc --semver -p
{
"major": 1,
"minor": 2,
"patch": 3,
"prerelease": "rc.1",
"build": "44837"
}
$ echo 1.2.3-rc.1+44837 | jc --semver -p -r
{
"major": "1",
"minor": "2",
"patch": "3",
"prerelease": "rc.1",
"build": "44837"
}
<a id="jc.parsers.semver.parse"></a>
### parse
```python
def parse(data: str, raw: bool = False, quiet: bool = False) -> JSONDictType
```
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
Returns:
Dictionary. Raw or processed structured data.
### Parser Information
Compatibility: linux, darwin, cygwin, win32, aix, freebsd
Version 1.0 by Kelly Brazil (kellyjonbrazil@gmail.com)

507
docs/parsers/sshd_conf.md Normal file
View File

@@ -0,0 +1,507 @@
[Home](https://kellyjonbrazil.github.io/jc/)
<a id="jc.parsers.sshd_conf"></a>
# jc.parsers.sshd\_conf
jc - JSON Convert sshd configuration file and `sshd -T` command output parser
This parser will work with `sshd` configuration files or the output of
`sshd -T`. Any `Match` blocks in the `sshd` configuration file will be
ignored.
Usage (cli):
$ sshd -T | jc --sshd-conf
or
$ jc sshd -T
or
$ cat sshd_conf | jc --sshd-conf
Usage (module):
import jc
result = jc.parse('sshd_conf', sshd_conf_output)
Schema:
{
"acceptenv": [
string
],
"addressfamily": string,
"allowagentforwarding": string,
"allowstreamlocalforwarding": string,
"allowtcpforwarding": string,
"authenticationmethods": string,
"authorizedkeyscommand": string,
"authorizedkeyscommanduser": string,
"authorizedkeysfile": [
string
],
"authorizedprincipalscommand": string,
"authorizedprincipalscommanduser": string,
"authorizedprincipalsfile": string,
"banner": string,
"casignaturealgorithms": [
string
],
"chrootdirectory": string,
"ciphers": [
string
],
"ciphers_strategy": string,
"clientalivecountmax": integer,
"clientaliveinterval": integer,
"compression": string,
"disableforwarding": string,
"exposeauthinfo": string,
"fingerprinthash": string,
"forcecommand": string,
"gatewayports": string,
"gssapiauthentication": string,
"gssapicleanupcredentials": string,
"gssapikexalgorithms": [
string
],
"gssapikeyexchange": string,
"gssapistorecredentialsonrekey": string,
"gssapistrictacceptorcheck": string,
"hostbasedacceptedalgorithms": [
string
],
"hostbasedauthentication": string,
"hostbasedusesnamefrompacketonly": string,
"hostkeyagent": string,
"hostkeyalgorithms": [
string
],
"hostkey": [
string
],
"ignorerhosts": string,
"ignoreuserknownhosts": string,
"include": [
string
],
"ipqos": [
string
],
"kbdinteractiveauthentication": string,
"kerberosauthentication": string,
"kerberosorlocalpasswd": string,
"kerberosticketcleanup": sttring,
"kexalgorithms": [
string
],
"listenaddress": [
string
],
"logingracetime": integer,
"loglevel": string,
"macs": [
string
],
"macs_strategy": string,
"maxauthtries": integer,
"maxsessions": integer,
"maxstartups": integer,
"maxstartups_rate": integer,
"maxstartups_full": integer,
"modulifile": string,
"passwordauthentication": string,
"permitemptypasswords": string,
"permitlisten": [
string
],
"permitopen": [
string
],
"permitrootlogin": string,
"permittty": string,
"permittunnel": string,
"permituserenvironment": string,
"permituserrc": string,
"persourcemaxstartups": string,
"persourcenetblocksize": string,
"pidfile": string,
"port": [
integer
],
"printlastlog": string,
"printmotd": string,
"pubkeyacceptedalgorithms": [
string
],
"pubkeyauthentication": string,
"pubkeyauthoptions": string,
"rekeylimit": integer,
"rekeylimit_time": integer,
"revokedkeys": string,
"securitykeyprovider": string,
"streamlocalbindmask": string,
"streamlocalbindunlink": string,
"strictmodes": string,
"subsystem": string,
"subsystem_command": string
"syslogfacility": string,
"tcpkeepalive": string,
"trustedusercakeys": string,
"usedns": string,
"usepam": string,
"versionaddendum": string,
"x11displayoffset": integer,
"x11forwarding": string,
"x11uselocalhost": string,
"xauthlocation": string
}
Examples:
$ sshd -T | jc --sshd-conf -p
{
"acceptenv": [
"LANG",
"LC_*"
],
"addressfamily": "any",
"allowagentforwarding": "yes",
"allowstreamlocalforwarding": "yes",
"allowtcpforwarding": "yes",
"authenticationmethods": "any",
"authorizedkeyscommand": "none",
"authorizedkeyscommanduser": "none",
"authorizedkeysfile": [
".ssh/authorized_keys",
".ssh/authorized_keys2"
],
"authorizedprincipalscommand": "none",
"authorizedprincipalscommanduser": "none",
"authorizedprincipalsfile": "none",
"banner": "none",
"casignaturealgorithms": [
"ssh-ed25519",
"ecdsa-sha2-nistp256",
"ecdsa-sha2-nistp384",
"ecdsa-sha2-nistp521",
"sk-ssh-ed25519@openssh.com",
"sk-ecdsa-sha2-nistp256@openssh.com",
"rsa-sha2-512",
"rsa-sha2-256"
],
"chrootdirectory": "none",
"ciphers": [
"chacha20-poly1305@openssh.com",
"aes128-ctr",
"aes192-ctr",
"aes256-ctr",
"aes128-gcm@openssh.com",
"aes256-gcm@openssh.com"
],
"ciphers_strategy": "+",
"clientalivecountmax": 3,
"clientaliveinterval": 0,
"compression": "yes",
"disableforwarding": "no",
"exposeauthinfo": "no",
"fingerprinthash": "SHA256",
"forcecommand": "none",
"gatewayports": "no",
"gssapiauthentication": "no",
"gssapicleanupcredentials": "yes",
"gssapikexalgorithms": [
"gss-group14-sha256-",
"gss-group16-sha512-",
"gss-nistp256-sha256-",
"gss-curve25519-sha256-",
"gss-group14-sha1-",
"gss-gex-sha1-"
],
"gssapikeyexchange": "no",
"gssapistorecredentialsonrekey": "no",
"gssapistrictacceptorcheck": "yes",
"hostbasedacceptedalgorithms": [
"ssh-ed25519-cert-v01@openssh.com",
"ecdsa-sha2-nistp256-cert-v01@openssh.com",
"ecdsa-sha2-nistp384-cert-v01@openssh.com",
"ecdsa-sha2-nistp521-cert-v01@openssh.com",
"sk-ssh-ed25519-cert-v01@openssh.com",
"sk-ecdsa-sha2-nistp256-cert-v01@openssh.com",
"rsa-sha2-512-cert-v01@openssh.com",
"rsa-sha2-256-cert-v01@openssh.com",
"ssh-ed25519",
"ecdsa-sha2-nistp256",
"ecdsa-sha2-nistp384",
"ecdsa-sha2-nistp521",
"sk-ssh-ed25519@openssh.com",
"sk-ecdsa-sha2-nistp256@openssh.com",
"rsa-sha2-512",
"rsa-sha2-256"
],
"hostbasedauthentication": "no",
"hostbasedusesnamefrompacketonly": "no",
"hostkeyagent": "none",
"hostkeyalgorithms": [
"ssh-ed25519-cert-v01@openssh.com",
"ecdsa-sha2-nistp256-cert-v01@openssh.com",
"ecdsa-sha2-nistp384-cert-v01@openssh.com",
"ecdsa-sha2-nistp521-cert-v01@openssh.com",
"sk-ssh-ed25519-cert-v01@openssh.com",
"sk-ecdsa-sha2-nistp256-cert-v01@openssh.com",
"rsa-sha2-512-cert-v01@openssh.com",
"rsa-sha2-256-cert-v01@openssh.com",
"ssh-ed25519",
"ecdsa-sha2-nistp256",
"ecdsa-sha2-nistp384",
"ecdsa-sha2-nistp521",
"sk-ssh-ed25519@openssh.com",
"sk-ecdsa-sha2-nistp256@openssh.com",
"rsa-sha2-512",
"rsa-sha2-256"
],
"hostkey": [
"/etc/ssh/ssh_host_ecdsa_key",
"/etc/ssh/ssh_host_ed25519_key",
"/etc/ssh/ssh_host_rsa_key"
],
"ignorerhosts": "yes",
"ignoreuserknownhosts": "no",
"ipqos": [
"lowdelay",
"throughput"
],
"kbdinteractiveauthentication": "no",
"kerberosauthentication": "no",
"kerberosorlocalpasswd": "yes",
"kerberosticketcleanup": "yes",
"kexalgorithms": [
"sntrup761x25519-sha512@openssh.com",
"curve25519-sha256",
"curve25519-sha256@libssh.org",
"ecdh-sha2-nistp256",
"ecdh-sha2-nistp384",
"ecdh-sha2-nistp521",
"diffie-hellman-group-exchange-sha256",
"diffie-hellman-group16-sha512",
"diffie-hellman-group18-sha512",
"diffie-hellman-group14-sha256"
],
"listenaddress": [
"0.0.0.0:22",
"[::]:22"
],
"logingracetime": 120,
"loglevel": "INFO",
"macs": [
"umac-64-etm@openssh.com",
"umac-128-etm@openssh.com",
"hmac-sha2-256-etm@openssh.com",
"hmac-sha2-512-etm@openssh.com",
"hmac-sha1-etm@openssh.com",
"umac-64@openssh.com",
"umac-128@openssh.com",
"hmac-sha2-256",
"hmac-sha2-512",
"hmac-sha1"
],
"macs_strategy": "^",
"maxauthtries": 6,
"maxsessions": 10,
"maxstartups": 10,
"modulifile": "/etc/ssh/moduli",
"passwordauthentication": "yes",
"permitemptypasswords": "no",
"permitlisten": [
"any"
],
"permitopen": [
"any"
],
"permitrootlogin": "without-password",
"permittty": "yes",
"permittunnel": "no",
"permituserenvironment": "no",
"permituserrc": "yes",
"persourcemaxstartups": "none",
"persourcenetblocksize": "32:128",
"pidfile": "/run/sshd.pid",
"port": [
22
],
"printlastlog": "yes",
"printmotd": "no",
"pubkeyacceptedalgorithms": [
"ssh-ed25519-cert-v01@openssh.com",
"ecdsa-sha2-nistp256-cert-v01@openssh.com",
"ecdsa-sha2-nistp384-cert-v01@openssh.com",
"ecdsa-sha2-nistp521-cert-v01@openssh.com",
"sk-ssh-ed25519-cert-v01@openssh.com",
"sk-ecdsa-sha2-nistp256-cert-v01@openssh.com",
"rsa-sha2-512-cert-v01@openssh.com",
"rsa-sha2-256-cert-v01@openssh.com",
"ssh-ed25519",
"ecdsa-sha2-nistp256",
"ecdsa-sha2-nistp384",
"ecdsa-sha2-nistp521",
"sk-ssh-ed25519@openssh.com",
"sk-ecdsa-sha2-nistp256@openssh.com",
"rsa-sha2-512",
"rsa-sha2-256"
],
"pubkeyauthentication": "yes",
"pubkeyauthoptions": "none",
"rekeylimit": 0,
"revokedkeys": "none",
"securitykeyprovider": "internal",
"streamlocalbindmask": "0177",
"streamlocalbindunlink": "no",
"strictmodes": "yes",
"subsystem": "sftp",
"syslogfacility": "AUTH",
"tcpkeepalive": "yes",
"trustedusercakeys": "none",
"usedns": "no",
"usepam": "yes",
"versionaddendum": "none",
"x11displayoffset": 10,
"x11forwarding": "yes",
"x11uselocalhost": "yes",
"xauthlocation": "/usr/bin/xauth",
"maxstartups_rate": 30,
"maxstartups_full": 100,
"rekeylimit_time": 0,
"subsystem_command": "/usr/lib/openssh/sftp-server"
}
$ sshd -T | jc --sshd-conf -p -r
{
"acceptenv": [
"LANG",
"LC_*"
],
"addressfamily": "any",
"allowagentforwarding": "yes",
"allowstreamlocalforwarding": "yes",
"allowtcpforwarding": "yes",
"authenticationmethods": "any",
"authorizedkeyscommand": "none",
"authorizedkeyscommanduser": "none",
"authorizedkeysfile": ".ssh/authorized_keys .ssh/authorized_keys2",
"authorizedprincipalscommand": "none",
"authorizedprincipalscommanduser": "none",
"authorizedprincipalsfile": "none",
"banner": "none",
"casignaturealgorithms": "ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-s...",
"chrootdirectory": "none",
"ciphers": "chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,...",
"ciphers_strategy": "+",
"clientalivecountmax": "3",
"clientaliveinterval": "0",
"compression": "yes",
"disableforwarding": "no",
"exposeauthinfo": "no",
"fingerprinthash": "SHA256",
"forcecommand": "none",
"gatewayports": "no",
"gssapiauthentication": "no",
"gssapicleanupcredentials": "yes",
"gssapikexalgorithms": "gss-group14-sha256-,gss-group16-sha512-,...",
"gssapikeyexchange": "no",
"gssapistorecredentialsonrekey": "no",
"gssapistrictacceptorcheck": "yes",
"hostbasedacceptedalgorithms": "ssh-ed25519-cert-v01@openssh.co...",
"hostbasedauthentication": "no",
"hostbasedusesnamefrompacketonly": "no",
"hostkeyagent": "none",
"hostkeyalgorithms": "ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2...",
"hostkey": [
"/etc/ssh/ssh_host_ecdsa_key",
"/etc/ssh/ssh_host_ed25519_key",
"/etc/ssh/ssh_host_rsa_key"
],
"ignorerhosts": "yes",
"ignoreuserknownhosts": "no",
"ipqos": "lowdelay throughput",
"kbdinteractiveauthentication": "no",
"kerberosauthentication": "no",
"kerberosorlocalpasswd": "yes",
"kerberosticketcleanup": "yes",
"kexalgorithms": "sntrup761x25519-sha512@openssh.com,curve25519...",
"listenaddress": [
"0.0.0.0:22",
"[::]:22"
],
"logingracetime": "120",
"loglevel": "INFO",
"macs": "umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac...",
"macs_strategy": "^",
"maxauthtries": "6",
"maxsessions": "10",
"maxstartups": "10:30:100",
"modulifile": "/etc/ssh/moduli",
"passwordauthentication": "yes",
"permitemptypasswords": "no",
"permitlisten": "any",
"permitopen": "any",
"permitrootlogin": "without-password",
"permittty": "yes",
"permittunnel": "no",
"permituserenvironment": "no",
"permituserrc": "yes",
"persourcemaxstartups": "none",
"persourcenetblocksize": "32:128",
"pidfile": "/run/sshd.pid",
"port": [
"22"
],
"printlastlog": "yes",
"printmotd": "no",
"pubkeyacceptedalgorithms": "ssh-ed25519-cert-v01@openssh.com,...",
"pubkeyauthentication": "yes",
"pubkeyauthoptions": "none",
"rekeylimit": "0 0",
"revokedkeys": "none",
"securitykeyprovider": "internal",
"streamlocalbindmask": "0177",
"streamlocalbindunlink": "no",
"strictmodes": "yes",
"subsystem": "sftp /usr/lib/openssh/sftp-server",
"syslogfacility": "AUTH",
"tcpkeepalive": "yes",
"trustedusercakeys": "none",
"usedns": "no",
"usepam": "yes",
"versionaddendum": "none",
"x11displayoffset": "10",
"x11forwarding": "yes",
"x11uselocalhost": "yes",
"xauthlocation": "/usr/bin/xauth"
}
<a id="jc.parsers.sshd_conf.parse"></a>
### parse
```python
def parse(data: str, raw: bool = False, quiet: bool = False) -> JSONDictType
```
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
Returns:
Dictionary. Raw or processed structured data.
### Parser Information
Compatibility: linux, darwin, freebsd
Version 1.0 by Kelly Brazil (kellyjonbrazil@gmail.com)

View File

@@ -83,7 +83,10 @@ Examples:
```python
@add_jc_meta
def parse(data, raw=False, quiet=False, ignore_exceptions=False)
def parse(data: Iterable[str],
raw: bool = False,
quiet: bool = False,
ignore_exceptions: bool = False) -> StreamingOutputType
```
Main text parsing generator function. Returns an iterable object.
@@ -104,4 +107,4 @@ Returns:
### Parser Information
Compatibility: linux, darwin, freebsd
Version 1.1 by Kelly Brazil (kellyjonbrazil@gmail.com)
Version 1.3 by Kelly Brazil (kellyjonbrazil@gmail.com)

View File

@@ -53,7 +53,7 @@ Blank values converted to `null`/`None`.
]
[0] naive timestamp if "timestamp" field is parsable, else null
[1] timezone aware timestamp availabe for UTC, else null
[1] timezone aware timestamp available for UTC, else null
[2] this field exists if the syslog line is not parsable. The value
is the original syslog line.

View File

@@ -64,7 +64,7 @@ Blank values converted to `null`/`None`.
}
[0] naive timestamp if "timestamp" field is parsable, else null
[1] timezone aware timestamp availabe for UTC, else null
[1] timezone aware timestamp available for UTC, else null
[2] this field exists if the syslog line is not parsable. The value
is the original syslog line.

143
docs/parsers/udevadm.md Normal file
View File

@@ -0,0 +1,143 @@
[Home](https://kellyjonbrazil.github.io/jc/)
<a id="jc.parsers.udevadm"></a>
# jc.parsers.udevadm
jc - JSON Convert `udevadm info` command output parser
Usage (cli):
$ udevadm info --query=all /dev/sda | jc --udevadm
or
$ jc udevadm info --query=all /dev/sda
Usage (module):
import jc
result = jc.parse('udevadm', udevadm_command_output)
Schema:
{
"P": string,
"N": string,
"L": integer,
"S": [
string
],
"E": {
"<key>": string
}
}
Examples:
$ udevadm info --query=all /dev/sda | jc --udevadm -p
{
"P": "/devices/pci0000:00/0000:00:10.0/host32/target32:0:0/32:0:0:0/block/sda",
"N": "sda",
"L": 0,
"S": [
"disk/by-path/pci-0000:00:10.0-scsi-0:0:0:0"
],
"E": {
"DEVPATH": "/devices/pci0000:00/0000:00:10.0/host32/target32:0:0/32:0:0:0/block/sda",
"DEVNAME": "/dev/sda",
"DEVTYPE": "disk",
"MAJOR": "8",
"MINOR": "0",
"SUBSYSTEM": "block",
"USEC_INITIALIZED": "6100111",
"SCSI_TPGS": "0",
"SCSI_TYPE": "disk",
"SCSI_VENDOR": "VMware,",
"SCSI_VENDOR_ENC": "VMware,\\x20",
"SCSI_MODEL": "VMware_Virtual_S",
"SCSI_MODEL_ENC": "VMware\\x20Virtual\\x20S",
"SCSI_REVISION": "1.0",
"ID_SCSI": "1",
"ID_VENDOR": "VMware_",
"ID_VENDOR_ENC": "VMware\\x2c\\x20",
"ID_MODEL": "VMware_Virtual_S",
"ID_MODEL_ENC": "VMware\\x20Virtual\\x20S",
"ID_REVISION": "1.0",
"ID_TYPE": "disk",
"MPATH_SBIN_PATH": "/sbin",
"ID_BUS": "scsi",
"ID_PATH": "pci-0000:00:10.0-scsi-0:0:0:0",
"ID_PATH_TAG": "pci-0000_00_10_0-scsi-0_0_0_0",
"ID_PART_TABLE_UUID": "a5bd0c01-4210-46f2-b558-5c11c209a8f7",
"ID_PART_TABLE_TYPE": "gpt",
"DEVLINKS": "/dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:0:0",
"TAGS": ":systemd:"
}
}
$ udevadm info --query=all /dev/sda | jc --udevadm -p -r
{
"P": "/devices/pci0000:00/0000:00:10.0/host32/target32:0:0/32:0:0:0/block/sda",
"N": "sda",
"L": "0",
"S": [
"disk/by-path/pci-0000:00:10.0-scsi-0:0:0:0"
],
"E": {
"DEVPATH": "/devices/pci0000:00/0000:00:10.0/host32/target32:0:0/32:0:0:0/block/sda",
"DEVNAME": "/dev/sda",
"DEVTYPE": "disk",
"MAJOR": "8",
"MINOR": "0",
"SUBSYSTEM": "block",
"USEC_INITIALIZED": "6100111",
"SCSI_TPGS": "0",
"SCSI_TYPE": "disk",
"SCSI_VENDOR": "VMware,",
"SCSI_VENDOR_ENC": "VMware,\\x20",
"SCSI_MODEL": "VMware_Virtual_S",
"SCSI_MODEL_ENC": "VMware\\x20Virtual\\x20S",
"SCSI_REVISION": "1.0",
"ID_SCSI": "1",
"ID_VENDOR": "VMware_",
"ID_VENDOR_ENC": "VMware\\x2c\\x20",
"ID_MODEL": "VMware_Virtual_S",
"ID_MODEL_ENC": "VMware\\x20Virtual\\x20S",
"ID_REVISION": "1.0",
"ID_TYPE": "disk",
"MPATH_SBIN_PATH": "/sbin",
"ID_BUS": "scsi",
"ID_PATH": "pci-0000:00:10.0-scsi-0:0:0:0",
"ID_PATH_TAG": "pci-0000_00_10_0-scsi-0_0_0_0",
"ID_PART_TABLE_UUID": "a5bd0c01-4210-46f2-b558-5c11c209a8f7",
"ID_PART_TABLE_TYPE": "gpt",
"DEVLINKS": "/dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:0:0",
"TAGS": ":systemd:"
}
}
<a id="jc.parsers.udevadm.parse"></a>
### parse
```python
def parse(data: str, raw: bool = False, quiet: bool = False) -> JSONDictType
```
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
Returns:
Dictionary. Raw or processed structured data.
### Parser Information
Compatibility: linux
Version 1.0 by Kelly Brazil (kellyjonbrazil@gmail.com)

View File

@@ -123,4 +123,4 @@ Returns:
### Parser Information
Compatibility: linux
Version 1.1 by Kelly Brazil (kellyjonbrazil@gmail.com)
Version 1.2 by Kelly Brazil (kellyjonbrazil@gmail.com)

View File

@@ -5,6 +5,11 @@
jc - JSON Convert `XML` file parser
This parser adds a `@` prefix to attributes by default. This can be changed
to a `_` prefix by using the `-r` (cli) or `raw=True` (module) option.
Text values for nodes will have the key-name of `#text`.
Usage (cli):
$ cat foo.xml | jc --xml
@@ -93,4 +98,4 @@ Returns:
### Parser Information
Compatibility: linux, darwin, cygwin, win32, aix, freebsd
Version 1.6 by Kelly Brazil (kellyjonbrazil@gmail.com)
Version 1.7 by Kelly Brazil (kellyjonbrazil@gmail.com)

View File

@@ -19,7 +19,7 @@ jc - JSON Convert streaming utils
### streaming\_input\_type\_check
```python
def streaming_input_type_check(data: Iterable) -> None
def streaming_input_type_check(data: Iterable[Union[str, bytes]]) -> None
```
Ensure input data is an iterable, but not a string or bytes. Raises
@@ -40,7 +40,8 @@ Ensure each line is a string. Raises `TypeError` if not.
### stream\_success
```python
def stream_success(output_line: Dict, ignore_exceptions: bool) -> Dict
def stream_success(output_line: JSONDictType,
ignore_exceptions: bool) -> JSONDictType
```
Add `_jc_meta` object to output line if `ignore_exceptions=True`
@@ -50,7 +51,7 @@ Add `_jc_meta` object to output line if `ignore_exceptions=True`
### stream\_error
```python
def stream_error(e: BaseException, line: str) -> Dict
def stream_error(e: BaseException, line: str) -> JSONDictType
```
Return an error `_jc_meta` field.
@@ -60,7 +61,7 @@ Return an error `_jc_meta` field.
### add\_jc\_meta
```python
def add_jc_meta(func)
def add_jc_meta(func: F) -> F
```
Decorator for streaming parsers to add stream_success and stream_error
@@ -106,7 +107,7 @@ In all cases above:
```python
def raise_or_yield(ignore_exceptions: bool, e: BaseException,
line: str) -> tuple
line: str) -> Tuple[BaseException, str]
```
Return the exception object and line string if ignore_exceptions is

View File

@@ -64,7 +64,7 @@ Returns:
### is\_compatible
```python
def is_compatible(compatible: List) -> bool
def is_compatible(compatible: List[str]) -> bool
```
Returns True if the parser is compatible with the running OS platform.
@@ -75,7 +75,7 @@ Returns True if the parser is compatible with the running OS platform.
```python
def compatibility(mod_name: str,
compatible: List,
compatible: List[str],
quiet: bool = False) -> None
```
@@ -91,7 +91,7 @@ Parameters:
the parser. compatible options:
linux, darwin, cygwin, win32, aix, freebsd
quiet: (bool) supress compatibility message if True
quiet: (bool) suppress compatibility message if True
Returns:
@@ -125,7 +125,7 @@ Returns:
### convert\_to\_int
```python
def convert_to_int(value: Union[str, float]) -> Optional[int]
def convert_to_int(value: object) -> Optional[int]
```
Converts string and float input to int. Strips all non-numeric
@@ -144,7 +144,7 @@ Returns:
### convert\_to\_float
```python
def convert_to_float(value: Union[str, int]) -> Optional[float]
def convert_to_float(value: object) -> Optional[float]
```
Converts string and int input to float. Strips all non-numeric
@@ -163,7 +163,7 @@ Returns:
### convert\_to\_bool
```python
def convert_to_bool(value: Union[str, int, float]) -> bool
def convert_to_bool(value: object) -> bool
```
Converts string, integer, or float input to boolean by checking
@@ -183,7 +183,7 @@ Returns:
### input\_type\_check
```python
def input_type_check(data: str) -> None
def input_type_check(data: object) -> None
```
Ensure input data is a string. Raises `TypeError` if not.
@@ -201,8 +201,8 @@ class timestamp()
### \_\_init\_\_
```python
def __init__(datetime_string: str,
format_hint: Optional[Iterable] = None) -> None
def __init__(datetime_string: Optional[str],
format_hint: Optional[Iterable[int]] = None) -> None
```
Input a datetime text string of several formats and convert to a
@@ -233,3 +233,6 @@ Returns a timestamp object with the following attributes:
utc (int | None): aware timestamp only if UTC timezone
detected in datetime string. None if conversion fails.
iso (str | None): ISO string - timezone information is output
only if UTC timezone is detected in the datetime string.

View File

@@ -1,4 +1,5 @@
# -*- coding: utf-8 -*-
# type: ignore
# Copyright (c) 2005-2010 ActiveState Software Inc.
# Copyright (c) 2013 Eddy Petrișor

1243
jc/cli.py

File diff suppressed because it is too large Load Diff

View File

@@ -18,7 +18,7 @@ long_options_map: Dict[str, List[str]] = {
'--zsh-comp': ['Z', 'gen Zsh completion: jc -Z > "${fpath[1]}/_jc"']
}
new_pygments_colors = {
new_pygments_colors: Dict[str, str] = {
'black': 'ansiblack',
'red': 'ansired',
'green': 'ansigreen',
@@ -37,7 +37,7 @@ new_pygments_colors = {
'white': 'ansiwhite',
}
old_pygments_colors = {
old_pygments_colors: Dict[str, str] = {
'black': '#ansiblack',
'red': '#ansidarkred',
'green': '#ansidarkgreen',
@@ -55,3 +55,42 @@ old_pygments_colors = {
'brightcyan': '#ansiturquoise',
'white': '#ansiwhite',
}
helptext_preamble_string: str = f'''\
jc converts the output of many commands, file-types, and strings to JSON or YAML
Usage:
Standard syntax:
COMMAND | jc [OPTIONS] PARSER
cat FILE | jc [OPTIONS] PARSER
echo STRING | jc [OPTIONS] PARSER
Magic syntax:
jc [OPTIONS] COMMAND
jc [OPTIONS] /proc/<path-to-procfile>
Parsers:
'''
helptext_end_string: str = '''\
Examples:
Standard Syntax:
$ dig www.google.com | jc --pretty --dig
$ cat /proc/meminfo | jc --pretty --proc
Magic Syntax:
$ jc --pretty dig www.google.com
$ jc --pretty /proc/meminfo
Parser Documentation:
$ jc --help --dig
Show Hidden Parsers:
$ jc -hh
'''

51
jc/jc_types.py Normal file
View File

@@ -0,0 +1,51 @@
"""jc - JSON Convert lib module"""
import sys
from typing import Any, Dict, List, Tuple, Iterator, Optional, Union
JSONDictType = Dict[str, Any]
StreamingOutputType = Iterator[Union[JSONDictType, Tuple[BaseException, str]]]
if sys.version_info >= (3, 8):
from typing import TypedDict
ParserInfoType = TypedDict(
'ParserInfoType',
{
"name": str,
"argument": str,
"version": str,
"description": str,
"author": str,
"author_email": str,
"compatible": List[str],
"magic_commands": List[str],
"documentation": str,
"streaming": bool,
"plugin": bool,
"hidden": bool,
"deprecated": bool
},
total=False
)
TimeStampFormatType = TypedDict(
'TimeStampFormatType',
{
'id': int,
'format': str,
'locale': Optional[str]
}
)
else:
ParserInfoType = Dict
TimeStampFormatType = Dict
try:
from pygments.token import (Name, Number, String, Keyword)
CustomColorType = Dict[Union[Name.Tag, Number, String, Keyword], str]
except Exception:
CustomColorType = Dict # type: ignore

240
jc/lib.py
View File

@@ -3,12 +3,15 @@ import sys
import os
import re
import importlib
from typing import Dict, List, Iterable, Union, Iterator
from typing import List, Iterable, Union, Iterator
from types import ModuleType
from .jc_types import ParserInfoType, JSONDictType
from jc import appdirs
__version__ = '1.22.0'
parsers = [
__version__ = '1.22.3'
parsers: List[str] = [
'acpi',
'airport',
'airport-s',
@@ -16,15 +19,19 @@ parsers = [
'asciitable',
'asciitable-m',
'blkid',
'cbt',
'cef',
'cef-s',
'chage',
'cksum',
'clf',
'clf-s',
'crontab',
'crontab-u',
'csv',
'csv-s',
'date',
'datetime-iso',
'df',
'dig',
'dir',
@@ -34,11 +41,13 @@ parsers = [
'email-address',
'env',
'file',
'findmnt',
'finger',
'free',
'fstab',
'git-log',
'git-log-s',
'git-ls-remote',
'gpg',
'group',
'gshadow',
@@ -66,6 +75,7 @@ parsers = [
'lsblk',
'lsmod',
'lsof',
'lspci',
'lsusb',
'm3u',
'mdadm',
@@ -75,7 +85,11 @@ parsers = [
'netstat',
'nmcli',
'ntpq',
'openvpn',
'os-prober',
'passwd',
'pci-ids',
'pgpass',
'pidstat',
'pidstat-s',
'ping',
@@ -139,9 +153,11 @@ parsers = [
'rpm-qi',
'rsync',
'rsync-s',
'semver',
'sfdisk',
'shadow',
'ss',
'sshd-conf',
'stat',
'stat-s',
'sysctl',
@@ -161,6 +177,7 @@ parsers = [
'top-s',
'tracepath',
'traceroute',
'udevadm',
'ufw',
'ufw-appinfo',
'uname',
@@ -181,19 +198,19 @@ parsers = [
'zipinfo'
]
def _cliname_to_modname(parser_cli_name):
def _cliname_to_modname(parser_cli_name: str) -> str:
"""Return real module name (dashes converted to underscores)"""
return parser_cli_name.replace('--', '').replace('-', '_')
def _modname_to_cliname(parser_mod_name):
def _modname_to_cliname(parser_mod_name: str) -> str:
"""Return module's cli name (underscores converted to dashes)"""
return parser_mod_name.replace('_', '-')
# Create the local_parsers list. This is a list of custom or
# override parsers from <user_data_dir>/jc/jcparsers/*.py.
# Once this list is created, extend the parsers list with it.
local_parsers = []
data_dir = appdirs.user_data_dir('jc', 'jc')
local_parsers: List[str] = []
data_dir = appdirs.user_data_dir('jc', 'jc') # type: ignore
local_parsers_dir = os.path.join(data_dir, 'jcparsers')
if os.path.isdir(local_parsers_dir):
sys.path.append(data_dir)
@@ -208,21 +225,20 @@ if os.path.isdir(local_parsers_dir):
except Exception:
pass
def _parser_argument(parser_mod_name):
def _parser_argument(parser_mod_name: str) -> str:
"""Return short name of the parser with dashes and with -- prefix"""
parser = _modname_to_cliname(parser_mod_name)
return f'--{parser}'
def _get_parser(parser_mod_name):
def _get_parser(parser_mod_name: str) -> ModuleType:
"""Return the parser module object"""
# ensure parser_mod_name is a true module name and not a cli name
parser_mod_name = _cliname_to_modname(parser_mod_name)
parser_cli_name = _modname_to_cliname(parser_mod_name)
modpath = 'jcparsers.' if parser_cli_name in local_parsers else 'jc.parsers.'
modpath: str = 'jcparsers.' if parser_cli_name in local_parsers else 'jc.parsers.'
return importlib.import_module(f'{modpath}{parser_mod_name}')
def _parser_is_streaming(parser):
def _parser_is_streaming(parser: ModuleType) -> bool:
"""
Returns True if this is a streaming parser, else False
@@ -233,16 +249,39 @@ def _parser_is_streaming(parser):
return False
def _parser_is_hidden(parser: ModuleType) -> bool:
"""
Returns True if this is a hidden parser, else False
parser is a parser module object.
"""
if getattr(parser.info, 'hidden', None):
return True
return False
def _parser_is_deprecated(parser: ModuleType) -> bool:
"""
Returns True if this is a deprecated parser, else False
parser is a parser module object.
"""
if getattr(parser.info, 'deprecated', None):
return True
return False
def parse(
parser_mod_name: str,
parser_mod_name: Union[str, ModuleType],
data: Union[str, bytes, Iterable[str]],
quiet: bool = False,
raw: bool = False,
ignore_exceptions: bool = None,
**kwargs
) -> Union[Dict, List[Dict], Iterator[Dict]]:
) -> Union[JSONDictType, List[JSONDictType], Iterator[JSONDictType]]:
"""
Parse the string data using the supplied parser module.
Parse the data (string or bytes) using the supplied parser (string or
module object).
This function provides a high-level API to simplify parser use. This
function will call built-in parsers and custom plugin parsers.
@@ -266,6 +305,14 @@ def parse(
To get a list of available parser module names, use `parser_mod_list()`.
Alternatively, a parser module object can be supplied:
>>> import jc
>>> import jc.parsers.date as jc_date
>>> date_obj = jc.parse(jc_date, 'Tue Jan 18 10:23:07 PST 2022')
>>> print(f'The year is: {date_obj["year"]}')
The year is: 2022
You can also use the lower-level parser modules directly:
>>> import jc.parsers.date
@@ -286,11 +333,14 @@ def parse(
Parameters:
parser_mod_name: (string) name of the parser module. This
function will accept module_name,
parser_mod_name: (string or name of the parser module. This
Module) function will accept module_name,
cli-name, and --argument-name
variants of the module name.
A Module object can also be passed
directly or via _get_parser()
data: (string or data to parse (string or bytes for
bytes or standard parsers, iterable of
iterable) strings for streaming parsers)
@@ -307,67 +357,138 @@ def parse(
Standard Parsers: Dictionary or List of Dictionaries
Streaming Parsers: Generator Object containing Dictionaries
"""
jc_parser = _get_parser(parser_mod_name)
if isinstance(parser_mod_name, ModuleType):
jc_parser = parser_mod_name
else:
jc_parser = _get_parser(parser_mod_name)
if ignore_exceptions is not None:
return jc_parser.parse(data, quiet=quiet, raw=raw,
ignore_exceptions=ignore_exceptions, **kwargs)
return jc_parser.parse(
data,
quiet=quiet,
raw=raw,
ignore_exceptions=ignore_exceptions,
**kwargs
)
return jc_parser.parse(data, quiet=quiet, raw=raw, **kwargs)
def parser_mod_list() -> List[str]:
def parser_mod_list(
show_hidden: bool = False,
show_deprecated: bool = False
) -> List[str]:
"""Returns a list of all available parser module names."""
return [_cliname_to_modname(p) for p in parsers]
plist: List[str] = []
for p in parsers:
parser = _get_parser(p)
def plugin_parser_mod_list() -> List[str]:
if not show_hidden and _parser_is_hidden(parser):
continue
if not show_deprecated and _parser_is_deprecated(parser):
continue
plist.append(_cliname_to_modname(p))
return plist
def plugin_parser_mod_list(
show_hidden: bool = False,
show_deprecated: bool = False
) -> List[str]:
"""
Returns a list of plugin parser module names. This function is a
subset of `parser_mod_list()`.
"""
return [_cliname_to_modname(p) for p in local_parsers]
plist: List[str] = []
for p in local_parsers:
parser = _get_parser(p)
def standard_parser_mod_list() -> List[str]:
if not show_hidden and _parser_is_hidden(parser):
continue
if not show_deprecated and _parser_is_deprecated(parser):
continue
plist.append(_cliname_to_modname(p))
return plist
def standard_parser_mod_list(
show_hidden: bool = False,
show_deprecated: bool = False
) -> List[str]:
"""
Returns a list of standard parser module names. This function is a
subset of `parser_mod_list()` and does not contain any streaming
parsers.
"""
plist = []
plist: List[str] = []
for p in parsers:
parser = _get_parser(p)
if not _parser_is_streaming(parser):
if not show_hidden and _parser_is_hidden(parser):
continue
if not show_deprecated and _parser_is_deprecated(parser):
continue
plist.append(_cliname_to_modname(p))
return plist
def streaming_parser_mod_list() -> List[str]:
def streaming_parser_mod_list(
show_hidden: bool = False,
show_deprecated: bool = False
) -> List[str]:
"""
Returns a list of streaming parser module names. This function is a
subset of `parser_mod_list()`.
"""
plist = []
plist: List[str] = []
for p in parsers:
parser = _get_parser(p)
if _parser_is_streaming(parser):
if not show_hidden and _parser_is_hidden(parser):
continue
if not show_deprecated and _parser_is_deprecated(parser):
continue
plist.append(_cliname_to_modname(p))
return plist
def parser_info(parser_mod_name: str, documentation: bool = False) -> Dict:
def parser_info(
parser_mod_name: Union[str, ModuleType],
documentation: bool = False
) -> ParserInfoType:
"""
Returns a dictionary that includes the parser module metadata.
Parameters:
parser_mod_name: (string) name of the parser module. This
function will accept module_name,
parser_mod_name: (string or name of the parser module. This
Module) function will accept module_name,
cli-name, and --argument-name
variants of the module name.
variants of the module name as well
as a parser module object.
documentation: (boolean) include parser docstring if True
"""
# ensure parser_mod_name is a true module name and not a cli name
parser_mod_name = _cliname_to_modname(parser_mod_name)
parser_mod = _get_parser(parser_mod_name)
info_dict: Dict = {}
if isinstance(parser_mod_name, ModuleType):
parser_mod = parser_mod_name
parser_mod_name = parser_mod.__name__.split('.')[-1]
else:
# ensure parser_mod_name is a true module name and not a cli name
parser_mod_name = _cliname_to_modname(parser_mod_name)
parser_mod = _get_parser(parser_mod_name)
info_dict: ParserInfoType = {}
if hasattr(parser_mod, 'info'):
info_dict['name'] = parser_mod_name
@@ -376,7 +497,7 @@ def parser_info(parser_mod_name: str, documentation: bool = False) -> Dict:
for k, v in parser_entry.items():
if not k.startswith('__'):
info_dict[k] = v
info_dict[k] = v # type: ignore
if _modname_to_cliname(parser_mod_name) in local_parsers:
info_dict['plugin'] = True
@@ -389,38 +510,51 @@ def parser_info(parser_mod_name: str, documentation: bool = False) -> Dict:
return info_dict
def all_parser_info(documentation: bool = False,
show_hidden: bool = False
) -> List[Dict]:
def all_parser_info(
documentation: bool = False,
show_hidden: bool = False,
show_deprecated: bool = False
) -> List[ParserInfoType]:
"""
Returns a list of dictionaries that includes metadata for all parser
modules.
modules. By default only non-hidden, non-deprecated parsers are
returned.
Parameters:
documentation: (boolean) include parser docstrings if True
show_hidden: (boolean) also show parsers marked as hidden
in their info metadata.
show_deprecated: (boolean) also show parsers marked as
deprecated in their info metadata.
"""
temp_list = [parser_info(p, documentation=documentation) for p in parsers]
plist: List[str] = []
for p in parsers:
parser = _get_parser(p)
p_list = []
if show_hidden:
p_list = temp_list
if not show_hidden and _parser_is_hidden(parser):
continue
else:
for item in temp_list:
if not item.get('hidden', None):
p_list.append(item)
if not show_deprecated and _parser_is_deprecated(parser):
continue
return p_list
plist.append(_cliname_to_modname(p))
p_info_list: List[ParserInfoType] = [parser_info(p, documentation=documentation) for p in plist]
def get_help(parser_mod_name: str) -> None:
return p_info_list
def get_help(parser_mod_name: Union[str, ModuleType]) -> None:
"""
Show help screen for the selected parser.
This function will accept **module_name**, **cli-name**, and
**--argument-name** variants of the module name string.
**--argument-name** variants of the module name string as well as a
parser module object.
"""
help(_get_parser(parser_mod_name))
if isinstance(parser_mod_name, ModuleType):
jc_parser = parser_mod_name
else:
jc_parser = _get_parser(parser_mod_name)
help(jc_parser)

View File

@@ -567,7 +567,7 @@ class DSASignature(Sequence):
@classmethod
def from_p1363(cls, data):
"""
Reads a signature from a byte string encoding accordint to IEEE P1363,
Reads a signature from a byte string encoding according to IEEE P1363,
which is used by Microsoft's BCryptSignHash() function.
:param data:

View File

@@ -247,7 +247,7 @@ class Asn1Value(object):
:param no_explicit:
If explicit tagging info should be removed from this instance.
Used internally to allow contructing the underlying value that
Used internally to allow constructing the underlying value that
has been wrapped in an explicit tag.
:param tag_type:
@@ -697,7 +697,7 @@ class Castable(object):
if other_class.tag != self.__class__.tag:
raise TypeError(unwrap(
'''
Can not covert a value from %s object to %s object since they
Can not convert a value from %s object to %s object since they
use different tags: %d versus %d
''',
type_name(other_class),
@@ -1349,7 +1349,7 @@ class Choice(Asn1Value):
class Concat(object):
"""
A class that contains two or more encoded child values concatentated
A class that contains two or more encoded child values concatenated
together. THIS IS NOT PART OF THE ASN.1 SPECIFICATION! This exists to handle
the x509.TrustedCertificate() class for OpenSSL certificates containing
extra information.
@@ -3757,7 +3757,7 @@ class Sequence(Asn1Value):
def _make_value(self, field_name, field_spec, value_spec, field_params, value):
"""
Contructs an appropriate Asn1Value object for a field
Constructs an appropriate Asn1Value object for a field
:param field_name:
A unicode string of the field name
@@ -3766,7 +3766,7 @@ class Sequence(Asn1Value):
An Asn1Value class that is the field spec
:param value_spec:
An Asn1Value class that is the vaue spec
An Asn1Value class that is the value spec
:param field_params:
None or a dict of params for the field spec

194
jc/parsers/cbt.py Normal file
View File

@@ -0,0 +1,194 @@
"""jc - JSON Convert `cbt` command output parser (Google Bigtable)
Parses the human-, but not machine-, friendly output of the cbt command (for
Google's Bigtable).
No effort is made to convert the data types of the values in the cells.
The `timestamp_epoch` calculated timestamp field is naive. (i.e. based on
the local time of the system the parser is run on)
The `timestamp_epoch_utc` calculated timestamp field is timezone-aware and
is only available if the timestamp has a UTC timezone.
The `timestamp_iso` calculated timestamp field will only include UTC
timezone information if the timestamp has a UTC timezone.
Raw output contains all cells for each column (including timestamps), while
the normal output contains only the latest value for each column.
Usage (cli):
$ cbt | jc --cbt
or
$ jc cbt
Usage (module):
import jc
result = jc.parse('cbt', cbt_command_output)
Schema:
[
{
"key": string,
"cells": {
<string>: { # column family
<string>: string # column: value
}
}
}
]
Schema (raw):
[
{
"key": string,
"cells": [
{
"column_family": string,
"column": string,
"value": string,
"timestamp_iso": string,
"timestamp_epoch": integer,
"timestamp_epoch_utc": integer
}
]
}
]
Examples:
$ cbt -project=$PROJECT -instance=$INSTANCE lookup $TABLE foo | jc --cbt -p
[
{
"key": "foo",
"cells": {
"foo": {
"bar": "baz"
}
}
}
]
$ cbt -project=$PROJECT -instance=$INSTANCE lookup $TABLE foo | jc --cbt -p -r
[
{
"key": "foo",
"cells": [
{
"column_family": "foo",
"column": "bar",
"value": "baz1",
"timestamp_iso": "1970-01-01T01:00:00",
"timestamp_epoch": 32400,
"timestamp_epoch_utc": null
}
]
}
]
"""
from itertools import groupby
from typing import List, Dict
from jc.jc_types import JSONDictType
import jc.utils
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.0'
description = '`cbt` (Google Bigtable) command parser'
author = 'Andreas Weiden'
author_email = 'andreas.weiden@gmail.com'
compatible = ['linux', 'darwin', 'cygwin', 'win32', 'aix', 'freebsd']
magic_commands = ['cbt']
__version__ = info.version
def _process(proc_data: List[JSONDictType]) -> List[JSONDictType]:
"""
Final processing to conform to the schema.
Parameters:
proc_data: (List of Dictionaries) raw structured data to process
Returns:
List of Dictionaries. Structured to conform to the schema.
"""
out_data = []
for row in proc_data:
cells: Dict = {}
key_func = lambda cell: (cell["column_family"], cell["column"])
all_cells = sorted(row["cells"], key=key_func)
for (column_family, column), group in groupby(all_cells, key=key_func):
group_list = sorted(group, key=lambda cell: cell["timestamp_iso"], reverse=True)
if column_family not in cells:
cells[column_family] = {}
cells[column_family][column] = group_list[0]["value"]
row["cells"] = cells
out_data.append(row)
return out_data
def parse(
data: str,
raw: bool = False,
quiet: bool = False
) -> List[JSONDictType]:
"""
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
Returns:
List of Dictionaries. Raw or processed structured data.
"""
jc.utils.compatibility(__name__, info.compatible, quiet)
jc.utils.input_type_check(data)
raw_output: List[Dict] = []
if jc.utils.has_data(data):
for line in filter(None, data.split("-" * 40)):
key = None
cells = []
column_name = ""
timestamp = ""
value_next = False
for field in line.splitlines():
if not field.strip():
continue
if field.startswith(" " * 4):
value = field.strip(' "')
if value_next:
dt = jc.utils.timestamp(timestamp, format_hint=(1750, 1755))
cells.append({
"column_family": column_name.split(":", 1)[0],
"column": column_name.split(":", 1)[1],
"value": value,
"timestamp_iso": dt.iso,
"timestamp_epoch": dt.naive,
"timestamp_epoch_utc": dt.utc
})
elif field.startswith(" " * 2):
column_name, timestamp = map(str.strip, field.split("@"))
value_next = True
else:
key = field
if key is not None:
raw_output.append({"key": key, "cells": cells})
return raw_output if raw else _process(raw_output)

View File

@@ -185,7 +185,7 @@ def _pycef_parse(str_input):
# If the input entry had any blanks in the required headers, that's wrong
# and we should return. Note we explicitly don't check the last item in the
# split list becuase the header ends in a '|' which means the last item
# split list because the header ends in a '|' which means the last item
# will always be an empty string (it doesn't exist, but the delimiter does).
if "" in spl[0:-1]:
raise ParseError('Blank field(s) in CEF header. Is it valid CEF format?')

298
jc/parsers/clf.py Normal file
View File

@@ -0,0 +1,298 @@
"""jc - JSON Convert Common Log Format file parser
This parser will handle the Common Log Format standard as specified at
https://www.w3.org/Daemon/User/Config/Logging.html#common-logfile-format.
Combined Log Format is also supported. (Referer and User Agent fields added)
Extra fields may be present and will be enclosed in the `extra` field as
a single string.
If a log line cannot be parsed, an object with an `unparsable` field will
be present with a value of the original line.
The `epoch` calculated timestamp field is naive. (i.e. based on the
local time of the system the parser is run on)
The `epoch_utc` calculated timestamp field is timezone-aware and is
only available if the timezone field is UTC.
Usage (cli):
$ cat file.log | jc --clf
Usage (module):
import jc
result = jc.parse('clf', common_log_file_output)
Schema:
Empty strings and `-` values are converted to `null`/`None`.
[
{
"host": string,
"ident": string,
"authuser": string,
"date": string,
"day": integer,
"month": string,
"year": integer,
"hour": integer,
"minute": integer,
"second": integer,
"tz": string,
"request": string,
"request_method": string,
"request_url": string,
"request_version": string,
"status": integer,
"bytes": integer,
"referer": string,
"user_agent": string,
"extra": string,
"epoch": integer, # [0]
"epoch_utc": integer, # [1]
"unparsable": string # [2]
}
]
[0] naive timestamp
[1] timezone-aware timestamp. Only available if timezone field is UTC
[2] exists if the line was not able to be parsed
Examples:
$ cat file.log | jc --clf -p
[
{
"host": "127.0.0.1",
"ident": "user-identifier",
"authuser": "frank",
"date": "10/Oct/2000:13:55:36 -0700",
"day": 10,
"month": "Oct",
"year": 2000,
"hour": 13,
"minute": 55,
"second": 36,
"tz": "-0700",
"request": "GET /apache_pb.gif HTTPS/1.0",
"status": 200,
"bytes": 2326,
"referer": null,
"user_agent": null,
"extra": null,
"request_method": "GET",
"request_url": "/apache_pb.gif",
"request_version": "HTTPS/1.0",
"epoch": 971211336,
"epoch_utc": null
},
{
"host": "1.1.1.2",
"ident": null,
"authuser": null,
"date": "11/Nov/2016:03:04:55 +0100",
"day": 11,
"month": "Nov",
"year": 2016,
"hour": 3,
"minute": 4,
"second": 55,
"tz": "+0100",
"request": "GET /",
"status": 200,
"bytes": 83,
"referer": null,
"user_agent": null,
"extra": "- 9221 1.1.1.1",
"request_method": "GET",
"request_url": "/",
"request_version": null,
"epoch": 1478862295,
"epoch_utc": null
},
...
]
$ cat file.log | jc --clf -p -r
[
{
"host": "127.0.0.1",
"ident": "user-identifier",
"authuser": "frank",
"date": "10/Oct/2000:13:55:36 -0700",
"day": "10",
"month": "Oct",
"year": "2000",
"hour": "13",
"minute": "55",
"second": "36",
"tz": "-0700",
"request": "GET /apache_pb.gif HTTPS/1.0",
"status": "200",
"bytes": "2326",
"referer": null,
"user_agent": null,
"extra": "",
"request_method": "GET",
"request_url": "/apache_pb.gif",
"request_version": "HTTPS/1.0"
},
{
"host": "1.1.1.2",
"ident": "-",
"authuser": "-",
"date": "11/Nov/2016:03:04:55 +0100",
"day": "11",
"month": "Nov",
"year": "2016",
"hour": "03",
"minute": "04",
"second": "55",
"tz": "+0100",
"request": "GET /",
"status": "200",
"bytes": "83",
"referer": "-",
"user_agent": "-",
"extra": "- 9221 1.1.1.1",
"request_method": "GET",
"request_url": "/",
"request_version": null
},
...
]
"""
import re
from typing import List, Dict
from jc.jc_types import JSONDictType
import jc.utils
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.0'
description = 'Common and Combined Log Format file parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
compatible = ['linux', 'darwin', 'cygwin', 'win32', 'aix', 'freebsd']
__version__ = info.version
def _process(proc_data: List[JSONDictType]) -> List[JSONDictType]:
"""
Final processing to conform to the schema.
Parameters:
proc_data: (List of Dictionaries) raw structured data to process
Returns:
List of Dictionaries. Structured to conform to the schema.
"""
int_list = {'day', 'year', 'hour', 'minute', 'second', 'status', 'bytes'}
for log in proc_data:
for key, val in log.items():
# integer conversions
if key in int_list:
log[key] = jc.utils.convert_to_int(val)
# convert `-` and blank values to None
if val == '-' or val == '':
log[key] = None
# add unix timestamps
if 'date' in log:
ts = jc.utils.timestamp(log['date'], format_hint=(1800,))
log['epoch'] = ts.naive
log['epoch_utc'] = ts.utc
return proc_data
def parse(
data: str,
raw: bool = False,
quiet: bool = False
) -> List[JSONDictType]:
"""
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
Returns:
List of Dictionaries. Raw or processed structured data.
"""
jc.utils.compatibility(__name__, info.compatible, quiet)
jc.utils.input_type_check(data)
raw_output: List[Dict] = []
output_line: Dict = {}
clf_pattern = re.compile(r'''
^(?P<host>-|\S+)\s
(?P<ident>-|\S+)\s
(?P<authuser>-|\S+)\s
\[
(?P<date>
(?P<day>\d+)/
(?P<month>\S\S\S)/
(?P<year>\d\d\d\d):
(?P<hour>\d\d):
(?P<minute>\d\d):
(?P<second>\d\d)\s
(?P<tz>\S+)
)
\]\s
\"(?P<request>.*?)\"\s
(?P<status>-|\d\d\d)\s
(?P<bytes>-|\d+)\s?
(?:\"(?P<referer>.*?)\"\s?)?
(?:\"(?P<user_agent>.*?)\"\s?)?
(?P<extra>.*)
''', re.VERBOSE
)
request_pattern = re.compile(r'''
(?P<request_method>\S+)\s
(?P<request_url>.*?(?=\sHTTPS?/|$))\s? # positive lookahead for HTTP(S)/ or end of string
(?P<request_version>HTTPS?/[\d\.]+)?
''', re.VERBOSE
)
if jc.utils.has_data(data):
for line in filter(None, data.splitlines()):
output_line = {}
clf_match = re.match(clf_pattern, line)
if clf_match:
output_line = clf_match.groupdict()
if output_line.get('request', None):
request_string = output_line['request']
request_match = re.match(request_pattern, request_string)
if request_match:
output_line.update(request_match.groupdict())
raw_output.append(output_line)
else:
raw_output.append(
{"unparsable": line}
)
return raw_output if raw else _process(raw_output)

223
jc/parsers/clf_s.py Normal file
View File

@@ -0,0 +1,223 @@
"""jc - JSON Convert Common Log Format file streaming parser
> This streaming parser outputs JSON Lines (cli) or returns an Iterable of
> Dictionaries (module)
This parser will handle the Common Log Format standard as specified at
https://www.w3.org/Daemon/User/Config/Logging.html#common-logfile-format.
Combined Log Format is also supported. (Referer and User Agent fields added)
Extra fields may be present and will be enclosed in the `extra` field as
a single string.
If a log line cannot be parsed, an object with an `unparsable` field will
be present with a value of the original line.
The `epoch` calculated timestamp field is naive. (i.e. based on the
local time of the system the parser is run on)
The `epoch_utc` calculated timestamp field is timezone-aware and is
only available if the timezone field is UTC.
Usage (cli):
$ cat file.log | jc --clf-s
Usage (module):
import jc
result = jc.parse('clf_s', common_log_file_output.splitlines())
for item in result:
# do something
Schema:
Empty strings and `-` values are converted to `null`/`None`.
{
"host": string,
"ident": string,
"authuser": string,
"date": string,
"day": integer,
"month": string,
"year": integer,
"hour": integer,
"minute": integer,
"second": integer,
"tz": string,
"request": string,
"request_method": string,
"request_url": string,
"request_version": string,
"status": integer,
"bytes": integer,
"referer": string,
"user_agent": string,
"extra": string,
"epoch": integer, # [0]
"epoch_utc": integer, # [1]
"unparsable": string # [2]
}
[0] naive timestamp
[1] timezone-aware timestamp. Only available if timezone field is UTC
[2] exists if the line was not able to be parsed
Examples:
$ cat file.log | jc --clf-s
{"host":"127.0.0.1","ident":"user-identifier","authuser":"frank","...}
{"host":"1.1.1.2","ident":null,"authuser":null,"date":"11/Nov/2016...}
...
$ cat file.log | jc --clf-s -r
{"host":"127.0.0.1","ident":"user-identifier","authuser":"frank","...}
{"host":"1.1.1.2","ident":"-","authuser":"-","date":"11/Nov/2016:0...}
...
"""
import re
from typing import Dict, Iterable
import jc.utils
from jc.streaming import (
add_jc_meta, streaming_input_type_check, streaming_line_input_type_check, raise_or_yield
)
from jc.jc_types import JSONDictType, StreamingOutputType
from jc.exceptions import ParseError
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.0'
description = 'Common and Combined Log Format file streaming parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
compatible = ['linux', 'darwin', 'cygwin', 'win32', 'aix', 'freebsd']
streaming = True
__version__ = info.version
def _process(proc_data: JSONDictType) -> JSONDictType:
"""
Final processing to conform to the schema.
Parameters:
proc_data: (Dictionary) raw structured data to process
Returns:
Dictionary. Structured data to conform to the schema.
"""
int_list = {'day', 'year', 'hour', 'minute', 'second', 'status', 'bytes'}
for key, val in proc_data.items():
# integer conversions
if key in int_list:
proc_data[key] = jc.utils.convert_to_int(val)
# convert `-` and blank values to None
if val == '-' or val == '':
proc_data[key] = None
# add unix timestamps
if 'date' in proc_data:
ts = jc.utils.timestamp(proc_data['date'], format_hint=(1800,))
proc_data['epoch'] = ts.naive
proc_data['epoch_utc'] = ts.utc
return proc_data
@add_jc_meta
def parse(
data: Iterable[str],
raw: bool = False,
quiet: bool = False,
ignore_exceptions: bool = False
) -> StreamingOutputType:
"""
Main text parsing generator function. Returns an iterable object.
Parameters:
data: (iterable) line-based text data to parse
(e.g. sys.stdin or str.splitlines())
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
ignore_exceptions: (boolean) ignore parsing exceptions if True
Returns:
Iterable of Dictionaries
"""
jc.utils.compatibility(__name__, info.compatible, quiet)
streaming_input_type_check(data)
clf_pattern = re.compile(r'''
^(?P<host>-|\S+)\s
(?P<ident>-|\S+)\s
(?P<authuser>-|\S+)\s
\[
(?P<date>
(?P<day>\d+)/
(?P<month>\S\S\S)/
(?P<year>\d\d\d\d):
(?P<hour>\d\d):
(?P<minute>\d\d):
(?P<second>\d\d)\s
(?P<tz>\S+)
)
\]\s
\"(?P<request>.*?)\"\s
(?P<status>-|\d\d\d)\s
(?P<bytes>-|\d+)\s?
(?:\"(?P<referer>.*?)\"\s?)?
(?:\"(?P<user_agent>.*?)\"\s?)?
(?P<extra>.*)
''', re.VERBOSE
)
request_pattern = re.compile(r'''
(?P<request_method>\S+)\s
(?P<request_url>.*?(?=\sHTTPS?/|$))\s? # positive lookahead for HTTP(S)/ or end of string
(?P<request_version>HTTPS?/[\d\.]+)?
''', re.VERBOSE
)
for line in data:
try:
streaming_line_input_type_check(line)
output_line: Dict = {}
if not line.strip():
continue
clf_match = re.match(clf_pattern, line)
if clf_match:
output_line = clf_match.groupdict()
if output_line.get('request', None):
request_string = output_line['request']
request_match = re.match(request_pattern, request_string)
if request_match:
output_line.update(request_match.groupdict())
else:
output_line = {"unparsable": line.strip()}
if output_line:
yield output_line if raw else _process(output_line)
else:
raise ParseError('Not Common Log Format data')
except Exception as e:
yield raise_or_yield(ignore_exceptions, e, line)

View File

@@ -72,13 +72,15 @@ Examples:
...
]
"""
from typing import List, Union, Type
from jc.jc_types import JSONDictType
import jc.utils
import csv
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.4'
version = '1.5'
description = 'CSV file parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
@@ -89,7 +91,7 @@ class info():
__version__ = info.version
def _process(proc_data):
def _process(proc_data: List[JSONDictType]) -> List[JSONDictType]:
"""
Final processing to conform to the schema.
@@ -107,7 +109,11 @@ def _process(proc_data):
return proc_data
def parse(data, raw=False, quiet=False):
def parse(
data: Union[str, bytes],
raw: bool = False,
quiet: bool = False
) -> List[JSONDictType]:
"""
Main text parsing function
@@ -124,6 +130,12 @@ def parse(data, raw=False, quiet=False):
jc.utils.compatibility(__name__, info.compatible, quiet)
jc.utils.input_type_check(data)
# remove BOM bytes, if present
if isinstance(data, str):
data = data.encode('utf-8')
data = data.decode('utf-8-sig')
raw_output = []
cleandata = data.splitlines()
@@ -132,7 +144,7 @@ def parse(data, raw=False, quiet=False):
if jc.utils.has_data(data):
dialect = 'excel' # default in csv module
dialect: Union[str, Type[csv.Dialect]] = 'excel' # default in csv module
try:
dialect = csv.Sniffer().sniff(data[:1024])
if '""' in data:
@@ -145,7 +157,4 @@ def parse(data, raw=False, quiet=False):
for row in reader:
raw_output.append(row)
if raw:
return raw_output
else:
return _process(raw_output)
return raw_output if raw else _process(raw_output)

View File

@@ -63,7 +63,7 @@ from jc.exceptions import ParseError
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.3'
version = '1.4'
description = 'CSV file streaming parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
@@ -127,7 +127,14 @@ def parse(data, raw=False, quiet=False, ignore_exceptions=False):
if len(temp_list) == 1:
raise ParseError('Unable to detect line endings. Please try the non-streaming CSV parser instead.')
sniffdata = '\n'.join(temp_list)[:1024]
# remove BOM bytes from first row, if present
if temp_list:
if isinstance(temp_list[0], str):
temp_list[0] = temp_list[0].encode('utf-8')
temp_list[0] = temp_list[0].decode('utf-8-sig')
sniffdata = '\r\n'.join(temp_list)[:1024]
dialect = 'excel' # default in csv module
try:

313
jc/parsers/datetime_iso.py Normal file
View File

@@ -0,0 +1,313 @@
"""jc - JSON Convert ISO 8601 Datetime string parser
This parser supports standard ISO 8601 strings that include both date and
time. If no timezone or offset information is available in the string, then
UTC timezone is used.
Usage (cli):
$ echo "2022-07-20T14:52:45Z" | jc --iso-datetime
Usage (module):
import jc
result = jc.parse('iso_datetime', iso_8601_string)
Schema:
{
"year": integer,
"month": string,
"month_num": integer,
"day": integer,
"weekday": string,
"weekday_num": integer,
"hour": integer,
"hour_24": integer,
"minute": integer,
"second": integer,
"microsecond": integer,
"period": string,
"utc_offset": string,
"day_of_year": integer,
"week_of_year": integer,
"iso": string,
"timestamp": integer # [0]
}
[0] timezone aware UNIX timestamp expressed in UTC
Examples:
$ echo "2022-07-20T14:52:45Z" | jc --iso-datetime -p
{
"year": 2022,
"month": "Jul",
"month_num": 7,
"day": 20,
"weekday": "Wed",
"weekday_num": 3,
"hour": 2,
"hour_24": 14,
"minute": 52,
"second": 45,
"microsecond": 0,
"period": "PM",
"utc_offset": "+0000",
"day_of_year": 201,
"week_of_year": 29,
"iso": "2022-07-20T14:52:45+00:00",
"timestamp": 1658328765
}
"""
import datetime
import re
import typing
from decimal import Decimal
import jc.utils
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.0'
description = 'ISO 8601 Datetime string parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
details = 'Using the pyiso8601 library from https://github.com/micktwomey/pyiso8601/releases/tag/1.0.2'
compatible = ['linux', 'aix', 'freebsd', 'darwin', 'win32', 'cygwin']
__version__ = info.version
####################################################
"""
pyiso8601 library from https://github.com/micktwomey/pyiso8601/releases/tag/1.0.2
"""
"""
Copyright (c) 2007 - 2022 Michael Twomey
Permission is hereby granted, free of charge, to any person obtaining a
copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be included
in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
"""
"""ISO 8601 date time string parsing
Basic usage:
>>> import iso8601
>>> iso8601._parse_date("2007-01-25T12:00:00Z")
datetime.datetime(2007, 1, 25, 12, 0, tzinfo=<iso8601.Utc ...>)
>>>
"""
# __all__ = ["_parse_date", "_ParseError", "UTC", "_FixedOffset"]
# Adapted from http://delete.me.uk/2005/03/iso8601.html
ISO8601_REGEX = re.compile(
r"""
(?P<year>[0-9]{4})
(
(
(-(?P<monthdash>[0-9]{1,2}))
|
(?P<month>[0-9]{2})
(?!$) # Don't allow YYYYMM
)
(
(
(-(?P<daydash>[0-9]{1,2}))
|
(?P<day>[0-9]{2})
)
(
(
(?P<separator>[ T])
(?P<hour>[0-9]{2})
(:{0,1}(?P<minute>[0-9]{2})){0,1}
(
:{0,1}(?P<second>[0-9]{1,2})
([.,](?P<second_fraction>[0-9]+)){0,1}
){0,1}
(?P<timezone>
Z
|
(
(?P<tz_sign>[-+])
(?P<tz_hour>[0-9]{2})
:{0,1}
(?P<tz_minute>[0-9]{2}){0,1}
)
){0,1}
){0,1}
)
){0,1} # YYYY-MM
){0,1} # YYYY only
$
""",
re.VERBOSE,
)
class _ParseError(ValueError):
"""Raised when there is a problem parsing a date string"""
UTC = datetime.timezone.utc
def _FixedOffset(
offset_hours: float, offset_minutes: float, name: str
) -> datetime.timezone:
return datetime.timezone(
datetime.timedelta(hours=offset_hours, minutes=offset_minutes), name
)
def _parse_timezone(
matches: typing.Dict[str, str],
default_timezone: typing.Optional[datetime.timezone] = UTC,
) -> typing.Optional[datetime.timezone]:
"""Parses ISO 8601 time zone specs into tzinfo offsets"""
tz = matches.get("timezone", None)
if tz == "Z":
return UTC
# This isn't strictly correct, but it's common to encounter dates without
# timezones so I'll assume the default (which defaults to UTC).
# Addresses issue 4.
if tz is None:
return default_timezone
sign = matches.get("tz_sign", None)
hours = int(matches.get("tz_hour", 0))
minutes = int(matches.get("tz_minute", 0))
description = f"{sign}{hours:02d}:{minutes:02d}"
if sign == "-":
hours = -hours
minutes = -minutes
return _FixedOffset(hours, minutes, description)
def _parse_date(
datestring: str, default_timezone: typing.Optional[datetime.timezone] = UTC
) -> datetime.datetime:
"""Parses ISO 8601 dates into datetime objects
The timezone is parsed from the date string. However it is quite common to
have dates without a timezone (not strictly correct). In this case the
default timezone specified in default_timezone is used. This is UTC by
default.
:param datestring: The date to parse as a string
:param default_timezone: A datetime tzinfo instance to use when no timezone
is specified in the datestring. If this is set to
None then a naive datetime object is returned.
:returns: A datetime.datetime instance
:raises: _ParseError when there is a problem parsing the date or
constructing the datetime instance.
"""
try:
m = ISO8601_REGEX.match(datestring)
except Exception as e:
raise _ParseError(e)
if not m:
raise _ParseError(f"Unable to parse date string {datestring!r}")
# Drop any Nones from the regex matches
# TODO: check if there's a way to omit results in regexes
groups: typing.Dict[str, str] = {
k: v for k, v in m.groupdict().items() if v is not None
}
try:
return datetime.datetime(
year=int(groups.get("year", 0)),
month=int(groups.get("month", groups.get("monthdash", 1))),
day=int(groups.get("day", groups.get("daydash", 1))),
hour=int(groups.get("hour", 0)),
minute=int(groups.get("minute", 0)),
second=int(groups.get("second", 0)),
microsecond=int(
Decimal(f"0.{groups.get('second_fraction', 0)}") * Decimal("1000000.0")
),
tzinfo=_parse_timezone(groups, default_timezone=default_timezone),
)
except Exception as e:
raise _ParseError(e)
####################################################
def _process(proc_data):
"""
Final processing to conform to the schema.
Parameters:
proc_data: (Dictionary) raw structured data to process
Returns:
Dictionary. Structured data to conform to the schema.
"""
# no further processing
return proc_data
def parse(data, raw=False, quiet=False):
"""
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
Returns:
Dictionary. Raw or processed structured data.
"""
jc.utils.compatibility(__name__, info.compatible, quiet)
jc.utils.input_type_check(data)
raw_output = {}
if jc.utils.has_data(data):
dt = _parse_date(data)
raw_output = {
'year': dt.year,
'month': dt.strftime('%b'),
'month_num': dt.month,
'day': dt.day,
'weekday': dt.strftime('%a'),
'weekday_num': dt.isoweekday(),
'hour': int(dt.strftime('%I')),
'hour_24': dt.hour,
'minute': dt.minute,
'second': dt.second,
'microsecond': dt.microsecond,
'period': dt.strftime('%p').upper(),
'utc_offset': dt.strftime('%z') or None,
'day_of_year': int(dt.strftime('%j')),
'week_of_year': int(dt.strftime('%W')),
'iso': dt.isoformat(),
'timestamp': int(dt.timestamp())
}
return raw_output if raw else _process(raw_output)

View File

@@ -101,7 +101,7 @@ Schema:
]
[0] naive timestamp if "when" field is parsable, else null
[1] timezone aware timestamp availabe for UTC, else null
[1] timezone aware timestamp available for UTC, else null
Examples:

View File

@@ -1,5 +1,9 @@
"""jc - JSON Convert `du` command output parser
The `du -h` option is not supported with the default output. If you
would like to use `du -h` or other options that change the output, be sure
to use `jc --raw` (cli) or `raw=True` (module).
Usage (cli):
$ du | jc --du

184
jc/parsers/findmnt.py Normal file
View File

@@ -0,0 +1,184 @@
"""jc - JSON Convert `findmnt` command output parser
Supports `-a`, `-l`, or no `findmnt` options.
> Note: Newer versions of `findmnt` have a JSON output option.
Usage (cli):
$ findmnt | jc --findmnt
or
$ jc findmnt
Usage (module):
import jc
result = jc.parse('findmnt', findmnt_command_output)
Schema:
[
{
"target": string,
"source": string,
"fstype": string,
"options": [
string
],
"kv_options": {
"<key_name>": string
}
]
Examples:
$ findmnt | jc --findmnt -p
[
{
"target": "/",
"source": "/dev/mapper/centos-root",
"fstype": "xfs",
"options": [
"rw",
"relatime",
"seclabel",
"attr2",
"inode64",
"noquota"
]
},
{
"target": "/sys/fs/cgroup",
"source": "tmpfs",
"fstype": "tmpfs",
"options": [
"ro",
"nosuid",
"nodev",
"noexec",
"seclabel"
],
"kv_options": {
"mode": "755"
}
},
...
]
$ findmnt | jc --findmnt -p -r
[
{
"target": "/",
"source": "/dev/mapper/centos-root",
"fstype": "xfs",
"options": "rw,relatime,seclabel,attr2,inode64,noquota"
},
{
"target": "/sys/fs/cgroup",
"source": "tmpfs",
"fstype": "tmpfs",
"options": "ro,nosuid,nodev,noexec,seclabel,mode=755"
},
...
]
"""
import re
from typing import List, Dict
from jc.jc_types import JSONDictType
from jc.parsers.universal import simple_table_parse
import jc.utils
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.0'
description = '`findmnt` command parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
compatible = ['linux']
magic_commands = ['findmnt']
__version__ = info.version
def _process(proc_data: List[JSONDictType]) -> List[JSONDictType]:
"""
Final processing to conform to the schema.
Parameters:
proc_data: (List of Dictionaries) raw structured data to process
Returns:
List of Dictionaries. Structured to conform to the schema.
"""
# split normal options and k/v options
for item in proc_data:
reg_options = []
kv_options = {}
if 'options' in item:
opt_list = item['options'].split(',')
for option in opt_list:
if '=' in option:
k, v = option.split('=', maxsplit=1)
kv_options[k] = v
else:
reg_options.append(option)
if reg_options:
item['options'] = reg_options
if kv_options:
item['kv_options'] = kv_options
return proc_data
def _replace(matchobj: re.Match) -> str:
if matchobj:
matchlen = len(matchobj.group(1))
return ' ' * matchlen + '/'
return ''
def parse(
data: str,
raw: bool = False,
quiet: bool = False
) -> List[JSONDictType]:
"""
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
Returns:
List of Dictionaries. Raw or processed structured data.
"""
jc.utils.compatibility(__name__, info.compatible, quiet)
jc.utils.input_type_check(data)
raw_output: List[Dict] = []
table: List[str] = []
if jc.utils.has_data(data):
for line in filter(None, data.splitlines()):
# remove initial drawing characters
line = re.sub(r'^([│ ├─└─|`-]+)/', _replace, line, count=1)
table.append(line)
table[0] = table[0].lower()
raw_output = simple_table_parse(table)
return raw_output if raw else _process(raw_output)

View File

@@ -34,6 +34,7 @@ Examples:
[]
"""
from typing import List, Dict
from jc.jc_types import JSONDictType
import jc.utils
@@ -53,7 +54,7 @@ class info():
__version__ = info.version
def _process(proc_data: List[Dict]) -> List[Dict]:
def _process(proc_data: List[JSONDictType]) -> List[JSONDictType]:
"""
Final processing to conform to the schema.
@@ -78,7 +79,7 @@ def parse(
data: str,
raw: bool = False,
quiet: bool = False
) -> List[Dict]:
) -> List[JSONDictType]:
"""
Main text parsing function
@@ -95,7 +96,7 @@ def parse(
jc.utils.compatibility(__name__, info.compatible, quiet)
jc.utils.input_type_check(data)
raw_output: List = []
raw_output: List[Dict] = []
if jc.utils.has_data(data):

View File

@@ -40,11 +40,12 @@ Examples:
{example output}
...
"""
from typing import Dict, Iterable, Union
from typing import Dict, Iterable
import jc.utils
from jc.streaming import (
add_jc_meta, streaming_input_type_check, streaming_line_input_type_check, raise_or_yield
)
from jc.jc_types import JSONDictType, StreamingOutputType
from jc.exceptions import ParseError
@@ -63,7 +64,7 @@ class info():
__version__ = info.version
def _process(proc_data: Dict) -> Dict:
def _process(proc_data: JSONDictType) -> JSONDictType:
"""
Final processing to conform to the schema.
@@ -90,7 +91,7 @@ def parse(
raw: bool = False,
quiet: bool = False,
ignore_exceptions: bool = False
) -> Union[Iterable[Dict], tuple]:
) -> StreamingOutputType:
"""
Main text parsing generator function. Returns an iterable object.
@@ -116,6 +117,10 @@ def parse(
streaming_line_input_type_check(line)
output_line: Dict = {}
# skip blank lines
if not line.strip():
continue
# parse the content here
# check out helper functions in jc.utils
# and jc.parsers.universal

View File

@@ -35,13 +35,13 @@ Schema:
[
{
"commit": string,
"author": string,
"author_email": string,
"author": string/null,
"author_email": string/null,
"date": string,
"epoch": integer, # [0]
"epoch_utc": integer, # [1]
"commit_by": string,
"commit_by_email": string,
"commit_by": string/null,
"commit_by_email": string/null,
"commit_by_date": string,
"message": string,
"stats" : {
@@ -56,7 +56,7 @@ Schema:
]
[0] naive timestamp if "date" field is parsable, else null
[1] timezone aware timestamp availabe for UTC, else null
[1] timezone aware timestamp available for UTC, else null
Examples:
@@ -153,7 +153,7 @@ changes_pattern = re.compile(r'\s(?P<files>\d+)\s+(files? changed),\s+(?P<insert
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.2'
version = '1.3'
description = '`git log` command parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
@@ -202,6 +202,28 @@ def _is_commit_hash(hash_string: str) -> bool:
return False
def _parse_name_email(line):
values = line.rsplit(maxsplit=1)
name = None
email = None
if len(values) == 2:
name = values[0]
if values[1].startswith('<') and values[1].endswith('>'):
email = values[1][1:-1]
else:
if values[0].lstrip().startswith('<') and values[0].endswith('>'):
email = values[0].lstrip()[1:-1]
else:
name = values[0]
if not name:
name = None
if not email:
email = None # covers '<>' case turning into null, not ''
return name, email
def parse(
data: str,
@@ -271,9 +293,7 @@ def parse(
continue
if line.startswith('Author: '):
values = line_list[1].rsplit(maxsplit=1)
output_line['author'] = values[0]
output_line['author_email'] = values[1].strip('<').strip('>')
output_line['author'], output_line['author_email'] = _parse_name_email(line_list[1])
continue
if line.startswith('Date: '):
@@ -289,9 +309,7 @@ def parse(
continue
if line.startswith('Commit: '):
values = line_list[1].rsplit(maxsplit=1)
output_line['commit_by'] = values[0]
output_line['commit_by_email'] = values[1].strip('<').strip('>')
output_line['commit_by'], output_line['commit_by_email'] = _parse_name_email(line_list[1])
continue
if line.startswith(' '):

View File

@@ -36,13 +36,13 @@ Schema:
{
"commit": string,
"author": string,
"author_email": string,
"author": string/null,
"author_email": string/null,
"date": string,
"epoch": integer, # [0]
"epoch_utc": integer, # [1]
"commit_by": string,
"commit_by_email": string,
"commit_by": string/null,
"commit_by_email": string/null,
"commit_by_date": string,
"message": string,
"stats" : {
@@ -63,7 +63,7 @@ Schema:
}
[0] naive timestamp if "date" field is parsable, else null
[1] timezone aware timestamp availabe for UTC, else null
[1] timezone aware timestamp available for UTC, else null
Examples:
@@ -75,6 +75,7 @@ Examples:
import re
from typing import List, Dict, Iterable, Union
import jc.utils
from jc.parsers.git_log import _parse_name_email
from jc.streaming import (
add_jc_meta, streaming_input_type_check, streaming_line_input_type_check, raise_or_yield
)
@@ -87,7 +88,7 @@ changes_pattern = re.compile(r'\s(?P<files>\d+)\s+(files? changed),\s+(?P<insert
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.2'
version = '1.3'
description = '`git log` command streaming parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
@@ -215,9 +216,7 @@ def parse(
continue
if line.startswith('Author: '):
values = line_list[1].rsplit(maxsplit=1)
output_line['author'] = values[0]
output_line['author_email'] = values[1].strip('<').strip('>')
output_line['author'], output_line['author_email'] = _parse_name_email(line_list[1])
continue
if line.startswith('Date: '):
@@ -233,9 +232,7 @@ def parse(
continue
if line.startswith('Commit: '):
values = line_list[1].rsplit(maxsplit=1)
output_line['commit_by'] = values[0]
output_line['commit_by_email'] = values[1].strip('<').strip('>')
output_line['commit_by'], output_line['commit_by_email'] = _parse_name_email(line_list[1])
continue
if line.startswith(' '):

139
jc/parsers/git_ls_remote.py Normal file
View File

@@ -0,0 +1,139 @@
"""jc - JSON Convert `git ls-remote` command output parser
This parser outputs two schemas:
- Default: A single object with key/value pairs
- Raw: An array of objects (`--raw` (cli) or `raw=True (module))
See the Schema section for more details
Usage (cli):
$ git ls-remote | jc --git-ls-remote
or
$ jc git ls-remote
Usage (module):
import jc
result = jc.parse('git_ls_remote', git_ls_remote_command_output)
Schema:
Default:
{
<reference>: string
}
Raw:
[
{
"reference": string,
"commit": string
}
]
Examples:
$ git ls-remote | jc --git-ls-remote -p
{
"HEAD": "214cd6b9e09603b3c4fa02203b24fb2bc3d4e338",
"refs/heads/dev": "b884f6aacca39e05994596d8fdfa7e7c4f1e0389",
"refs/heads/master": "214cd6b9e09603b3c4fa02203b24fb2bc3d4e338",
"refs/pull/1/head": "e416c77bed1267254da972b0f95b7ff1d43fccef",
...
}
$ git ls-remote | jc --git-ls-remote -p -r
[
{
"reference": "HEAD",
"commit": "214cd6b9e09603b3c4fa02203b24fb2bc3d4e338"
},
{
"reference": "refs/heads/dev",
"commit": "b884f6aacca39e05994596d8fdfa7e7c4f1e0389"
},
...
]
"""
from typing import List, Union
from jc.jc_types import JSONDictType
import jc.utils
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.0'
description = '`git ls-remote` command parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
compatible = ['linux', 'darwin', 'cygwin', 'win32', 'aix', 'freebsd']
magic_commands = ['git ls-remote']
__version__ = info.version
def _process(proc_data: List[JSONDictType]) -> JSONDictType:
"""
Final processing to conform to the schema.
Parameters:
proc_data: (List of Dictionaries) raw structured data to process
Returns:
Dictionary. Structured to conform to the schema.
"""
new_dict: JSONDictType = {}
for item in proc_data:
new_dict.update(
{
item['reference']: item['commit']
}
)
return new_dict
def parse(
data: str,
raw: bool = False,
quiet: bool = False
) -> Union[JSONDictType, List[JSONDictType]]:
"""
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
Returns:
Dictionary (default) or List of Dictionaries (raw)
"""
jc.utils.compatibility(__name__, info.compatible, quiet)
jc.utils.input_type_check(data)
raw_output: List[JSONDictType] = []
output_line: JSONDictType = {}
if jc.utils.has_data(data):
for line in filter(None, data.splitlines()):
commit, reference = line.split()
output_line = {
'reference': reference,
'commit': commit
}
raw_output.append(output_line)
return raw_output if raw else _process(raw_output)

File diff suppressed because it is too large Load Diff

View File

@@ -108,7 +108,7 @@ import jc.parsers.universal
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.2'
version = '1.3'
description = '`iostat` command streaming parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
@@ -196,7 +196,7 @@ def parse(data, raw=False, quiet=False, ignore_exceptions=False):
output_line = {}
# ignore blank lines and header line
if line == '\n' or line == '' or line.startswith('Linux'):
if not line.strip() or line.startswith('Linux'):
continue
if line.startswith('avg-cpu:'):

View File

@@ -21,6 +21,9 @@ Schema:
"ip": string,
"ip_compressed": string,
"ip_exploded": string,
"ip_split": [
string
],
"scope_id": string/null,
"ipv4_mapped": string/null,
"six_to_four": string/null,
@@ -78,6 +81,12 @@ Examples:
"ip": "192.168.2.10",
"ip_compressed": "192.168.2.10",
"ip_exploded": "192.168.2.10",
"ip_split": [
"192",
"168",
"2",
"10"
],
"scope_id": null,
"ipv4_mapped": null,
"six_to_four": null,
@@ -133,6 +142,12 @@ Examples:
"ip": "192.168.2.10",
"ip_compressed": "192.168.2.10",
"ip_exploded": "192.168.2.10",
"ip_split": [
"192",
"168",
"2",
"10"
],
"scope_id": null,
"ipv4_mapped": null,
"six_to_four": null,
@@ -186,14 +201,24 @@ Examples:
"version": 6,
"max_prefix_length": 128,
"ip": "127:0:de::1",
"ip_compressed": "127:0:de::1%128",
"ip_compressed": "127:0:de::1",
"ip_exploded": "0127:0000:00de:0000:0000:0000:0000:0001",
"ip_split": [
"0127",
"0000",
"00de",
"0000",
"0000",
"0000",
"0000",
"0001"
],
"scope_id": "128",
"ipv4_mapped": null,
"six_to_four": null,
"teredo_client": null,
"teredo_server": null,
"dns_ptr": "1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.....0.7.2.1.0.ip6.arpa",
"dns_ptr": "1.0.0.0.0.0...0.0.0.e.d.0.0.0.0.0.0.7.2.1.0.ip6.arpa",
"network": "127:0:de::",
"broadcast": "127:0:de::ffff:ffff",
"hostmask": "::ffff:ffff",
@@ -226,13 +251,13 @@ Examples:
"last_host": "01:27:00:00:00:de:00:00:00:00:00:00:ff:ff:ff:fe"
},
"bin": {
"ip": "000000010010011100000000000000000000000011011110000000...",
"network": "0000000100100111000000000000000000000000110111100...",
"broadcast": "00000001001001110000000000000000000000001101111...",
"hostmask": "000000000000000000000000000000000000000000000000...",
"netmask": "1111111111111111111111111111111111111111111111111...",
"first_host": "0000000100100111000000000000000000000000110111...",
"last_host": "00000001001001110000000000000000000000001101111..."
"ip": "0000000100100111000000000000000000000000110...000000000001",
"network": "00000001001001110000000000000000000000...000000000000",
"broadcast": "000000010010011100000000000000000000...111111111111",
"hostmask": "0000000000000000000000000000000000000...111111111111",
"netmask": "11111111111111111111111111111111111111...000000000000",
"first_host": "00000001001001110000000000000000000...000000000001",
"last_host": "000000010010011100000000000000000000...1111111111110"
}
}
@@ -243,12 +268,22 @@ Examples:
"ip": "127:0:de::1",
"ip_compressed": "127:0:de::1",
"ip_exploded": "0127:0000:00de:0000:0000:0000:0000:0001",
"ip_split": [
"0127",
"0000",
"00de",
"0000",
"0000",
"0000",
"0000",
"0001"
],
"scope_id": null,
"ipv4_mapped": null,
"six_to_four": null,
"teredo_client": null,
"teredo_server": null,
"dns_ptr": "1.0.0.0.0.0.0....0.0.0.e.d.0.0.0.0.0.0.7.2.1.0.ip6.arpa",
"dns_ptr": "1.0.0.0.0.0....0.0.0.0.e.d.0.0.0.0.0.0.7.2.1.0.ip6.arpa",
"network": "127:0:de::1",
"broadcast": "127:0:de::1",
"hostmask": "::",
@@ -281,13 +316,13 @@ Examples:
"last_host": "01:27:00:00:00:de:00:00:00:00:00:00:00:00:00:01"
},
"bin": {
"ip": "0000000100100111000000000000000000000000110111100000000...",
"network": "00000001001001110000000000000000000000001101111000...",
"broadcast": "000000010010011100000000000000000000000011011110...",
"hostmask": "0000000000000000000000000000000000000000000000000...",
"netmask": "11111111111111111111111111111111111111111111111111...",
"first_host": "00000001001001110000000000000000000000001101111...",
"last_host": "000000010010011100000000000000000000000011011110..."
"ip": "0000000100100111000000000000000000000000110111100...000001",
"network": "00000001001001110000000000000000000000001101...000001",
"broadcast": "000000010010011100000000000000000000000011...000001",
"hostmask": "0000000000000000000000000000000000000000000...000000",
"netmask": "11111111111111111111111111111111111111111111...111111",
"first_host": "00000001001001110000000000000000000000001...000001",
"last_host": "000000010010011100000000000000000000000011...0000001"
}
}
@@ -299,12 +334,22 @@ Examples:
"ip": "::ffff:c0a8:123",
"ip_compressed": "::ffff:c0a8:123",
"ip_exploded": "0000:0000:0000:0000:0000:ffff:c0a8:0123",
"ip_split": [
"0000",
"0000",
"0000",
"0000",
"0000",
"ffff",
"c0a8",
"0123"
],
"scope_id": null,
"ipv4_mapped": "192.168.1.35",
"six_to_four": null,
"teredo_client": null,
"teredo_server": null,
"dns_ptr": "3.2.1.0.8.a.0.c.f.f.f.f.0.0.0....0.0.0.0.0.0.0.ip6.arpa",
"dns_ptr": "3.2.1.0.8.a.0.c.f.f.f.f.0.0....0.0.0.0.0.0.ip6.arpa",
"network": "::ffff:c0a8:123",
"broadcast": "::ffff:c0a8:123",
"hostmask": "::",
@@ -337,13 +382,13 @@ Examples:
"last_host": "00:00:00:00:00:00:00:00:00:00:ff:ff:c0:a8:01:23"
},
"bin": {
"ip": "0000000000000000000000000000000000000000000000000000000...",
"network": "00000000000000000000000000000000000000000000000000...",
"broadcast": "000000000000000000000000000000000000000000000000...",
"hostmask": "0000000000000000000000000000000000000000000000000...",
"netmask": "11111111111111111111111111111111111111111111111111...",
"first_host": "00000000000000000000000000000000000000000000000...",
"last_host": "000000000000000000000000000000000000000000000000..."
"ip": "000000000000000000000000000000000000000000000...100100011",
"network": "0000000000000000000000000000000000000000...000100011",
"broadcast": "00000000000000000000000000000000000000...000100011",
"hostmask": "000000000000000000000000000000000000000...000000000",
"netmask": "1111111111111111111111111111111111111111...111111111",
"first_host": "0000000000000000000000000000000000000...100100011",
"last_host": "00000000000000000000000000000000000000...0100100011"
}
}
@@ -355,12 +400,22 @@ Examples:
"ip": "2002:c000:204::",
"ip_compressed": "2002:c000:204::",
"ip_exploded": "2002:c000:0204:0000:0000:0000:0000:0000",
"ip_split": [
"2002",
"c000",
"0204",
"0000",
"0000",
"0000",
"0000",
"0000"
],
"scope_id": null,
"ipv4_mapped": null,
"six_to_four": "192.0.2.4",
"teredo_client": null,
"teredo_server": null,
"dns_ptr": "0.0.0.0.0.0.0.0......0.4.0.2.0.0.0.0.c.2.0.0.2.ip6.arpa",
"dns_ptr": "0.0.0.0.0.0.0...0.0.0.4.0.2.0.0.0.0.c.2.0.0.2.ip6.arpa",
"network": "2002:c000:204::",
"broadcast": "2002:c000:204:ffff:ffff:ffff:ffff:ffff",
"hostmask": "::ffff:ffff:ffff:ffff:ffff",
@@ -393,13 +448,13 @@ Examples:
"last_host": "20:02:c0:00:02:04:ff:ff:ff:ff:ff:ff:ff:ff:ff:fe"
},
"bin": {
"ip": "0010000000000010110000000000000000000010000001000000000...",
"network": "00100000000000101100000000000000000000100000010000...",
"broadcast": "001000000000001011000000000000000000001000000100...",
"hostmask": "0000000000000000000000000000000000000000000000001...",
"netmask": "11111111111111111111111111111111111111111111111100...",
"first_host": "00100000000000101100000000000000000000100000010...",
"last_host": "001000000000001011000000000000000000001000000100..."
"ip": "00100000000000101100000000000000000000100000010...00000000",
"network": "001000000000001011000000000000000000001000...00000000",
"broadcast": "0010000000000010110000000000000000000010...11111111",
"hostmask": "00000000000000000000000000000000000000000...11111111",
"netmask": "111111111111111111111111111111111111111111...00000000",
"first_host": "001000000000001011000000000000000000001...00000001",
"last_host": "0010000000000010110000000000000000000010...111111110"
}
}
@@ -411,12 +466,22 @@ Examples:
"ip": "2001:0:4136:e378:8000:63bf:3fff:fdd2",
"ip_compressed": "2001:0:4136:e378:8000:63bf:3fff:fdd2",
"ip_exploded": "2001:0000:4136:e378:8000:63bf:3fff:fdd2",
"ip_split": [
"2001",
"0000",
"4136",
"e378",
"8000",
"63bf",
"3fff",
"fdd2"
],
"scope_id": null,
"ipv4_mapped": null,
"six_to_four": null,
"teredo_client": "192.0.2.45",
"teredo_server": "65.54.227.120",
"dns_ptr": "2.d.d.f.f.f.f.3.f.b.3.6.0.0.0....0.0.0.1.0.0.2.ip6.arpa",
"dns_ptr": "2.d.d.f.f.f.f.3.f.b.3.6.0.0.0.8.8....0.1.0.0.2.ip6.arpa",
"network": "2001:0:4136:e378:8000:63bf:3fff:fdd2",
"broadcast": "2001:0:4136:e378:8000:63bf:3fff:fdd2",
"hostmask": "::",
@@ -449,13 +514,13 @@ Examples:
"last_host": "20:01:00:00:41:36:e3:78:80:00:63:bf:3f:ff:fd:d2"
},
"bin": {
"ip": "001000000000000100000000000000000100000100110110111000...",
"network": "0010000000000001000000000000000001000001001101101...",
"broadcast": "00100000000000010000000000000000010000010011011...",
"hostmask": "000000000000000000000000000000000000000000000000...",
"netmask": "1111111111111111111111111111111111111111111111111...",
"first_host": "0010000000000001000000000000000001000001001101...",
"last_host": "00100000000000010000000000000000010000010011011..."
"ip": "0010000000000001000000000000000001000001001...110111010010",
"network": "00100000000000010000000000000000010000...110111010010",
"broadcast": "001000000000000100000000000000000100...110111010010",
"hostmask": "0000000000000000000000000000000000000...000000000000",
"netmask": "11111111111111111111111111111111111111...111111111111",
"first_host": "00100000000000010000000000000000010...110111010010",
"last_host": "001000000000000100000000000000000100...110111010010"
}
}
"""
@@ -468,7 +533,7 @@ import jc.utils
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.2'
version = '1.3'
description = 'IPv4 and IPv6 Address string parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
@@ -495,7 +560,7 @@ def _process(proc_data: Dict) -> Dict:
def _b2a(byte_string: bytes) -> str:
"""Convert a byte string to a colon-delimited hex ascii string"""
# need try/except since seperator was only introduced in python 3.8.
# need try/except since separator was only introduced in python 3.8.
# provides compatibility for python 3.6 and 3.7.
try:
return binascii.hexlify(byte_string, ':').decode('utf-8')
@@ -590,6 +655,13 @@ def parse(
bare_ip_string = str(interface.ip)
bare_ip = ipaddress.ip_address(bare_ip_string)
ip_ptr = bare_ip.reverse_pointer
ip_compressed = bare_ip.compressed
ip_exploded = bare_ip.exploded
if interface.version == 4:
ip_split = ip_exploded.split('.')
else:
ip_split = ip_exploded.split(':')
# fix for ipv6-only attributes
scope_id = None
@@ -638,8 +710,9 @@ def parse(
'version': interface.version,
'max_prefix_length': interface.max_prefixlen,
'ip': bare_ip_string,
'ip_compressed': bare_ip.compressed,
'ip_exploded': bare_ip.exploded,
'ip_compressed': ip_compressed,
'ip_exploded': ip_exploded,
'ip_split': ip_split,
'scope_id': scope_id,
'ipv4_mapped': ipv4_mapped,
'six_to_four': sixtofour,

View File

@@ -1,275 +1,32 @@
"""jc - JSON Convert ISO 8601 Datetime string parser
This parser supports standard ISO 8601 strings that include both date and
time. If no timezone or offset information is available in the sring, then
UTC timezone is used.
This parser has been renamed to datetime-iso (cli) or datetime_iso (module).
Usage (cli):
$ echo "2022-07-20T14:52:45Z" | jc --iso-datetime
Usage (module):
import jc
result = jc.parse('iso_datetime', iso_8601_string)
Schema:
{
"year": integer,
"month": string,
"month_num": integer,
"day": integer,
"weekday": string,
"weekday_num": integer,
"hour": integer,
"hour_24": integer,
"minute": integer,
"second": integer,
"microsecond": integer,
"period": string,
"utc_offset": string,
"day_of_year": integer,
"week_of_year": integer,
"iso": string,
"timestamp": integer # [0]
}
[0] timezone aware UNIX timestamp expressed in UTC
Examples:
$ echo "2022-07-20T14:52:45Z" | jc --iso-datetime -p
{
"year": 2022,
"month": "Jul",
"month_num": 7,
"day": 20,
"weekday": "Wed",
"weekday_num": 3,
"hour": 2,
"hour_24": 14,
"minute": 52,
"second": 45,
"microsecond": 0,
"period": "PM",
"utc_offset": "+0000",
"day_of_year": 201,
"week_of_year": 29,
"iso": "2022-07-20T14:52:45+00:00",
"timestamp": 1658328765
}
This parser will be removed in a future version, so please start using
the new parser name.
"""
import datetime
import re
import typing
from decimal import Decimal
from jc.parsers import datetime_iso
import jc.utils
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.0'
description = 'ISO 8601 Datetime string parser'
version = '1.1'
description = 'Deprecated - please use datetime-iso'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
details = 'Using the pyiso8601 library from https://github.com/micktwomey/pyiso8601/releases/tag/1.0.2'
details = 'Deprecated - please use datetime-iso'
compatible = ['linux', 'aix', 'freebsd', 'darwin', 'win32', 'cygwin']
deprecated = True
__version__ = info.version
####################################################
"""
pyiso8601 library from https://github.com/micktwomey/pyiso8601/releases/tag/1.0.2
"""
"""
Copyright (c) 2007 - 2022 Michael Twomey
Permission is hereby granted, free of charge, to any person obtaining a
copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be included
in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
"""
"""ISO 8601 date time string parsing
Basic usage:
>>> import iso8601
>>> iso8601._parse_date("2007-01-25T12:00:00Z")
datetime.datetime(2007, 1, 25, 12, 0, tzinfo=<iso8601.Utc ...>)
>>>
"""
# __all__ = ["_parse_date", "_ParseError", "UTC", "_FixedOffset"]
# Adapted from http://delete.me.uk/2005/03/iso8601.html
ISO8601_REGEX = re.compile(
r"""
(?P<year>[0-9]{4})
(
(
(-(?P<monthdash>[0-9]{1,2}))
|
(?P<month>[0-9]{2})
(?!$) # Don't allow YYYYMM
)
(
(
(-(?P<daydash>[0-9]{1,2}))
|
(?P<day>[0-9]{2})
)
(
(
(?P<separator>[ T])
(?P<hour>[0-9]{2})
(:{0,1}(?P<minute>[0-9]{2})){0,1}
(
:{0,1}(?P<second>[0-9]{1,2})
([.,](?P<second_fraction>[0-9]+)){0,1}
){0,1}
(?P<timezone>
Z
|
(
(?P<tz_sign>[-+])
(?P<tz_hour>[0-9]{2})
:{0,1}
(?P<tz_minute>[0-9]{2}){0,1}
)
){0,1}
){0,1}
)
){0,1} # YYYY-MM
){0,1} # YYYY only
$
""",
re.VERBOSE,
)
class _ParseError(ValueError):
"""Raised when there is a problem parsing a date string"""
UTC = datetime.timezone.utc
def _FixedOffset(
offset_hours: float, offset_minutes: float, name: str
) -> datetime.timezone:
return datetime.timezone(
datetime.timedelta(hours=offset_hours, minutes=offset_minutes), name
)
def _parse_timezone(
matches: typing.Dict[str, str],
default_timezone: typing.Optional[datetime.timezone] = UTC,
) -> typing.Optional[datetime.timezone]:
"""Parses ISO 8601 time zone specs into tzinfo offsets"""
tz = matches.get("timezone", None)
if tz == "Z":
return UTC
# This isn't strictly correct, but it's common to encounter dates without
# timezones so I'll assume the default (which defaults to UTC).
# Addresses issue 4.
if tz is None:
return default_timezone
sign = matches.get("tz_sign", None)
hours = int(matches.get("tz_hour", 0))
minutes = int(matches.get("tz_minute", 0))
description = f"{sign}{hours:02d}:{minutes:02d}"
if sign == "-":
hours = -hours
minutes = -minutes
return _FixedOffset(hours, minutes, description)
def _parse_date(
datestring: str, default_timezone: typing.Optional[datetime.timezone] = UTC
) -> datetime.datetime:
"""Parses ISO 8601 dates into datetime objects
The timezone is parsed from the date string. However it is quite common to
have dates without a timezone (not strictly correct). In this case the
default timezone specified in default_timezone is used. This is UTC by
default.
:param datestring: The date to parse as a string
:param default_timezone: A datetime tzinfo instance to use when no timezone
is specified in the datestring. If this is set to
None then a naive datetime object is returned.
:returns: A datetime.datetime instance
:raises: _ParseError when there is a problem parsing the date or
constructing the datetime instance.
"""
try:
m = ISO8601_REGEX.match(datestring)
except Exception as e:
raise _ParseError(e)
if not m:
raise _ParseError(f"Unable to parse date string {datestring!r}")
# Drop any Nones from the regex matches
# TODO: check if there's a way to omit results in regexes
groups: typing.Dict[str, str] = {
k: v for k, v in m.groupdict().items() if v is not None
}
try:
return datetime.datetime(
year=int(groups.get("year", 0)),
month=int(groups.get("month", groups.get("monthdash", 1))),
day=int(groups.get("day", groups.get("daydash", 1))),
hour=int(groups.get("hour", 0)),
minute=int(groups.get("minute", 0)),
second=int(groups.get("second", 0)),
microsecond=int(
Decimal(f"0.{groups.get('second_fraction', 0)}") * Decimal("1000000.0")
),
tzinfo=_parse_timezone(groups, default_timezone=default_timezone),
)
except Exception as e:
raise _ParseError(e)
####################################################
def _process(proc_data):
"""
Final processing to conform to the schema.
Parameters:
proc_data: (Dictionary) raw structured data to process
Returns:
Dictionary. Structured data to conform to the schema.
"""
# no further processing
return proc_data
def parse(data, raw=False, quiet=False):
"""
Main text parsing function
This parser is deprecated and calls datetime_iso. Please use datetime_iso
directly. This parser will be removed in the future.
Parameters:
@@ -281,33 +38,8 @@ def parse(data, raw=False, quiet=False):
Dictionary. Raw or processed structured data.
"""
jc.utils.compatibility(__name__, info.compatible, quiet)
jc.utils.input_type_check(data)
jc.utils.warning_message([
'iso-datetime parser is deprecated. Please use datetime-iso instead.'
])
raw_output = {}
if jc.utils.has_data(data):
dt = _parse_date(data)
raw_output = {
'year': dt.year,
'month': dt.strftime('%b'),
'month_num': dt.month,
'day': dt.day,
'weekday': dt.strftime('%a'),
'weekday_num': dt.isoweekday(),
'hour': int(dt.strftime('%I')),
'hour_24': dt.hour,
'minute': dt.minute,
'second': dt.second,
'microsecond': dt.microsecond,
'period': dt.strftime('%p').upper(),
'utc_offset': dt.strftime('%z') or None,
'day_of_year': int(dt.strftime('%j')),
'week_of_year': int(dt.strftime('%W')),
'iso': dt.isoformat(),
'timestamp': int(dt.timestamp())
}
return raw_output if raw else _process(raw_output)
return datetime_iso.parse(data, raw=raw, quiet=quiet)

View File

@@ -180,7 +180,7 @@ def _post_parse(data):
ssid = {k: v for k, v in ssid.items() if v}
cleandata.append(ssid)
# remove asterisks from begining of values
# remove asterisks from beginning of values
for ssid in cleandata:
for key in ssid:
if ssid[key].startswith('*'):

View File

@@ -78,7 +78,7 @@ def _process(proc_data: Dict) -> Dict:
def _b2a(byte_string: bytes) -> str:
"""Convert a byte string to a colon-delimited hex ascii string"""
# need try/except since seperator was only introduced in python 3.8.
# need try/except since separator was only introduced in python 3.8.
# provides compatibility for python 3.6 and 3.7.
try:
return binascii.hexlify(byte_string, ':').decode('utf-8')

View File

@@ -77,7 +77,7 @@ from jc.exceptions import ParseError
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.1'
version = '1.2'
description = '`ls` command streaming parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
@@ -148,7 +148,7 @@ def parse(data, raw=False, quiet=False, ignore_exceptions=False):
continue
# skip blank lines
if line.strip() == '':
if not line.strip():
continue
# Look for parent line if glob or -R is used

241
jc/parsers/lspci.py Normal file
View File

@@ -0,0 +1,241 @@
"""jc - JSON Convert `lspci -mmv` command output parser
This parser supports the following `lspci` options:
- `-mmv`
- `-nmmv`
- `-nnmmv`
Usage (cli):
$ lspci -nnmmv | jc --lspci
or
$ jc lspci -nnmmv
Usage (module):
import jc
result = jc.parse('lspci', lspci_command_output)
Schema:
[
{
"slot": string,
"domain": string,
"domain_int": integer,
"bus": string,
"bus_int": integer,
"dev": string,
"dev_int": integer,
"function": string,
"function_int": integer,
"class": string,
"class_id": string,
"class_id_int": integer,
"vendor": string,
"vendor_id": string,
"vendor_id_int": integer,
"device": string,
"device_id": string,
"device_id_int": integer,
"svendor": string,
"svendor_id": string,
"svendor_id_int": integer,
"sdevice": string,
"sdevice_id": string,
"sdevice_id_int": integer,
"rev": string,
"physlot": string,
"physlot_int": integer,
"progif": string,
"progif_int": integer
}
]
Examples:
$ lspci -nnmmv | jc --lspci -p
[
{
"slot": "ff:02:05.0",
"domain": "ff",
"domain_int": 255,
"bus": "02",
"bus_int": 2,
"dev": "05",
"dev_int": 5,
"function": "0",
"function_int": 0,
"class": "SATA controller",
"class_id": "0106",
"class_id_int": 262,
"vendor": "VMware",
"vendor_id": "15ad",
"vendor_id_int": 5549,
"device": "SATA AHCI controller",
"device_id": "07e0",
"device_id_int": 2016,
"svendor": "VMware",
"svendor_id": "15ad",
"svendor_id_int": 5549,
"sdevice": "SATA AHCI controller",
"sdevice_id": "07e0",
"sdevice_id_int": 2016,
"physlot": "37",
"physlot_int": 55,
"progif": "01",
"progif_int": 1
},
...
]
$ lspci -nnmmv | jc --lspci -p -r
[
{
"slot": "ff:02:05.0",
"domain": "ff",
"bus": "02",
"dev": "05",
"function": "0",
"class": "SATA controller",
"class_id": "0106",
"vendor": "VMware",
"vendor_id": "15ad",
"device": "SATA AHCI controller",
"device_id": "07e0",
"svendor": "VMware",
"svendor_id": "15ad",
"sdevice": "SATA AHCI controller",
"sdevice_id": "07e0",
"physlot": "37",
"progif": "01"
},
...
]
"""
import re
from typing import List, Dict
from jc.jc_types import JSONDictType
import jc.utils
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.0'
description = '`lspci -mmv` command parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
compatible = ['linux']
magic_commands = ['lspci']
__version__ = info.version
def _process(proc_data: List[JSONDictType]) -> List[JSONDictType]:
"""
Final processing to conform to the schema.
Parameters:
proc_data: (List of Dictionaries) raw structured data to process
Returns:
List of Dictionaries. Structured to conform to the schema.
"""
int_list: set[str] = {
'domain', 'bus', 'dev', 'function', 'class_id', 'vendor_id', 'device_id',
'svendor_id', 'sdevice_id', 'physlot', 'progif'
}
new_list: List[JSONDictType] = []
for item in proc_data:
output: Dict = {}
for key, val in item.items():
output[key] = val
if key in int_list:
output[key + '_int'] = int(val, 16)
new_list.append(output)
return new_list
def parse(
data: str,
raw: bool = False,
quiet: bool = False
) -> List[JSONDictType]:
"""
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
Returns:
List of Dictionaries. Raw or processed structured data.
"""
jc.utils.compatibility(__name__, info.compatible, quiet)
jc.utils.input_type_check(data)
raw_output: List = []
device_output: Dict = {}
if jc.utils.has_data(data):
item_id_p = re.compile(r'(?P<id>^[0-9a-f]{4}$)')
item_id_bracket_p = re.compile(r' \[(?P<id>[0-9a-f]{4})\]$')
for line in filter(None, data.splitlines()):
if line.startswith('Slot:'):
if device_output:
raw_output.append(device_output)
device_output = {}
device_output['slot'] = line.split()[1]
slot_info = line.split()[1]
*domain, bus, dev_fun = slot_info.split(':')
if domain:
dom = domain[0]
else:
dom = "00"
dev, fun = dev_fun.split('.')
device_output['domain'] = dom
device_output['bus'] = bus
device_output['dev'] = dev
device_output['function'] = fun
continue
key, val = line.split(maxsplit=1)
key = key[:-1].lower()
# numeric only (-nmmv)
if item_id_p.match(val):
device_output[key + '_id'] = val
continue
# string and numeric (-nnmmv)
if item_id_bracket_p.search(val):
string, idnum = val.rsplit(maxsplit=1)
device_output[key] = string
device_output[key + '_id'] = idnum[1:-1]
continue
# string only (-mmv)
device_output[key] = val
continue
if device_output:
raw_output.append(device_output)
return raw_output if raw else _process(raw_output)

View File

@@ -228,7 +228,7 @@ def _normalize_header(keyname: str) -> str:
def _add_text_kv(key: str, value: Optional[str]) -> Optional[Dict]:
"""
Add keys with _text suffix if there is a text description inside
paranthesis at the end of a value. The value of the _text field will
parenthesis at the end of a value. The value of the _text field will
only be the text inside the parenthesis. This allows cleanup of the
original field (convert to int/float/etc) without losing information.
"""

351
jc/parsers/openvpn.py Normal file
View File

@@ -0,0 +1,351 @@
"""jc - JSON Convert openvpn-status.log file parser
The `*_epoch` calculated timestamp fields are naive. (i.e. based on
the local time of the system the parser is run on)
Usage (cli):
$ cat openvpn-status.log | jc --openvpn
Usage (module):
import jc
result = jc.parse('openvpn', openvpn_status_log_file_output)
Schema:
{
"clients": [
{
"common_name": string,
"real_address": string,
"real_address_prefix": integer, # [0]
"real_address_port": integer, # [0]
"bytes_received": integer,
"bytes_sent": integer,
"connected_since": string,
"connected_since_epoch": integer,
"updated": string,
"updated_epoch": integer,
}
],
"routing_table": [
{
"virtual_address": string,
"virtual_address_prefix": integer, # [0]
"virtual_address_port": integer, # [0]
"common_name": string,
"real_address": string,
"real_address_prefix": integer, # [0]
"real_address_port": integer, # [0]
"last_reference": string,
"last_reference_epoch": integer,
}
],
"global_stats": {
"max_bcast_mcast_queue_len": integer
}
}
[0] null/None if not found
Examples:
$ cat openvpn-status.log | jc --openvpn -p
{
"clients": [
{
"common_name": "foo@example.com",
"real_address": "10.10.10.10",
"bytes_received": 334948,
"bytes_sent": 1973012,
"connected_since": "Thu Jun 18 04:23:03 2015",
"updated": "Thu Jun 18 08:12:15 2015",
"real_address_prefix": null,
"real_address_port": 49502,
"connected_since_epoch": 1434626583,
"updated_epoch": 1434640335
},
{
"common_name": "foo@example.com",
"real_address": "10.10.10.10",
"bytes_received": 334948,
"bytes_sent": 1973012,
"connected_since": "Thu Jun 18 04:23:03 2015",
"updated": "Thu Jun 18 08:12:15 2015",
"real_address_prefix": null,
"real_address_port": 49503,
"connected_since_epoch": 1434626583,
"updated_epoch": 1434640335
}
],
"routing_table": [
{
"virtual_address": "192.168.255.118",
"common_name": "baz@example.com",
"real_address": "10.10.10.10",
"last_reference": "Thu Jun 18 08:12:09 2015",
"virtual_address_prefix": null,
"virtual_address_port": null,
"real_address_prefix": null,
"real_address_port": 63414,
"last_reference_epoch": 1434640329
},
{
"virtual_address": "10.200.0.0",
"common_name": "baz@example.com",
"real_address": "10.10.10.10",
"last_reference": "Thu Jun 18 08:12:09 2015",
"virtual_address_prefix": 16,
"virtual_address_port": null,
"real_address_prefix": null,
"real_address_port": 63414,
"last_reference_epoch": 1434640329
}
],
"global_stats": {
"max_bcast_mcast_queue_len": 0
}
}
$ cat openvpn-status.log | jc --openvpn -p -r
{
"clients": [
{
"common_name": "foo@example.com",
"real_address": "10.10.10.10:49502",
"bytes_received": "334948",
"bytes_sent": "1973012",
"connected_since": "Thu Jun 18 04:23:03 2015",
"updated": "Thu Jun 18 08:12:15 2015"
},
{
"common_name": "foo@example.com",
"real_address": "10.10.10.10:49503",
"bytes_received": "334948",
"bytes_sent": "1973012",
"connected_since": "Thu Jun 18 04:23:03 2015",
"updated": "Thu Jun 18 08:12:15 2015"
}
],
"routing_table": [
{
"virtual_address": "192.168.255.118",
"common_name": "baz@example.com",
"real_address": "10.10.10.10:63414",
"last_reference": "Thu Jun 18 08:12:09 2015"
},
{
"virtual_address": "10.200.0.0/16",
"common_name": "baz@example.com",
"real_address": "10.10.10.10:63414",
"last_reference": "Thu Jun 18 08:12:09 2015"
}
],
"global_stats": {
"max_bcast_mcast_queue_len": "0"
}
}
"""
import re
import ipaddress
from typing import List, Dict, Tuple
from jc.jc_types import JSONDictType
import jc.utils
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.0'
description = 'openvpn-status.log file parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
compatible = ['linux', 'darwin', 'cygwin', 'win32', 'aix', 'freebsd']
__version__ = info.version
def _split_addr(addr_str: str) -> Tuple:
"""Check the type of address (v4, v6, mac) and split out the address,
prefix, and port. Values are None if they don't exist."""
address = possible_addr = prefix = port = possible_port = None
try:
address, prefix = addr_str.rsplit('/', maxsplit=1)
except Exception:
address = addr_str
# is this a mac address? then stop
if re.match(r'(?:\S\S\:){5}\S\S', address):
return address, prefix, port
# is it an ipv4 with port or just ipv6?
if ':' in address:
try:
possible_addr, possible_port = address.rsplit(':', maxsplit=1)
_ = ipaddress.IPv4Address(possible_addr)
address = possible_addr
port = possible_port
# assume it was an IPv6 address
except Exception:
pass
return address, prefix, port
def _process(proc_data: JSONDictType) -> JSONDictType:
"""
Final processing to conform to the schema.
Parameters:
proc_data: (Dictionary) raw structured data to process
Returns:
Dictionary. Structured to conform to the schema.
"""
int_list = {'bytes_received', 'bytes_sent', 'max_bcast_mcast_queue_len'}
date_fields = {'connected_since', 'updated', 'last_reference'}
addr_fields = {'real_address', 'virtual_address'}
if 'clients' in proc_data:
for item in proc_data['clients']:
for k, v in item.copy().items():
if k in int_list:
item[k] = jc.utils.convert_to_int(v)
if k in date_fields:
dt = jc.utils.timestamp(item[k], format_hint=(1000,))
item[k + '_epoch'] = dt.naive
if k in addr_fields:
addr, prefix, port = _split_addr(v)
item[k] = addr
item[k + '_prefix'] = jc.utils.convert_to_int(prefix)
item[k + '_port'] = jc.utils.convert_to_int(port)
if 'routing_table' in proc_data:
for item in proc_data['routing_table']:
for k, v in item.copy(). items():
if k in date_fields:
dt = jc.utils.timestamp(item[k], format_hint=(1000,))
item[k + '_epoch'] = dt.naive
if k in addr_fields:
addr, prefix, port = _split_addr(v)
item[k] = addr
item[k + '_prefix'] = jc.utils.convert_to_int(prefix)
item[k + '_port'] = jc.utils.convert_to_int(port)
if 'global_stats' in proc_data:
for k, v in proc_data['global_stats'].items():
if k in int_list:
if k in int_list:
proc_data['global_stats'][k] = jc.utils.convert_to_int(v)
return proc_data
def parse(
data: str,
raw: bool = False,
quiet: bool = False
) -> JSONDictType:
"""
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
Returns:
Dictionary. Raw or processed structured data.
"""
jc.utils.compatibility(__name__, info.compatible, quiet)
jc.utils.input_type_check(data)
raw_output: Dict = {}
clients: List[Dict] = []
routing_table: List[Dict] = []
global_stats: Dict = {}
section: str = '' # clients, routing, stats
updated: str = ''
if jc.utils.has_data(data):
for line in filter(None, data.splitlines()):
if line.startswith('OpenVPN CLIENT LIST'):
section = 'clients'
continue
if line.startswith('ROUTING TABLE'):
section = 'routing'
continue
if line.startswith('GLOBAL STATS'):
section = 'stats'
continue
if line.startswith('END'):
break
if section == 'clients' and line.startswith('Updated,'):
_, updated = line.split(',', maxsplit=1)
continue
if section == 'clients' and line.startswith('Common Name,Real Address,'):
continue
if section == 'clients':
c_name, real_addr, r_bytes, s_bytes, connected = line.split(',', maxsplit=5)
clients.append(
{
'common_name': c_name,
'real_address': real_addr,
'bytes_received': r_bytes,
'bytes_sent': s_bytes,
'connected_since': connected,
'updated': updated
}
)
continue
if section == 'routing' and line.startswith('Virtual Address,Common Name,'):
continue
if section == 'routing':
# Virtual Address,Common Name,Real Address,Last Ref
# 192.168.255.118,baz@example.com,10.10.10.10:63414,Thu Jun 18 08:12:09 2015
virt_addr, c_name, real_addr, last_ref = line.split(',', maxsplit=4)
route = {
'virtual_address': virt_addr,
'common_name': c_name,
'real_address': real_addr,
'last_reference': last_ref
}
# fixup for virtual addresses ending in "C"
if 'virtual_address' in route:
if route['virtual_address'].endswith('C'):
route['virtual_address'] = route['virtual_address'][:-1]
routing_table.append(route)
continue
if section == "stats":
if line.startswith('Max bcast/mcast queue length'):
global_stats['max_bcast_mcast_queue_len'] = line.split(',', maxsplit=1)[1]
continue
raw_output['clients'] = clients
raw_output['routing_table'] = routing_table
raw_output['global_stats'] = {}
raw_output['global_stats'].update(global_stats)
return raw_output if raw else _process(raw_output)

118
jc/parsers/os_prober.py Normal file
View File

@@ -0,0 +1,118 @@
"""jc - JSON Convert `os-prober` command output parser
Usage (cli):
$ os-prober | jc --os-prober
or
$ jc os-prober
Usage (module):
import jc
result = jc.parse('os_prober', os_prober_command_output)
Schema:
{
"partition": string,
"efi_bootmgr": string, # [0]
"name": string,
"short_name": string,
"type": string
}
[0] only exists if an EFI boot manager is detected
Examples:
$ os-prober | jc --os-prober -p
{
"partition": "/dev/sda1",
"name": "Windows 10",
"short_name": "Windows",
"type": "chain"
}
"""
from typing import Dict
from jc.jc_types import JSONDictType
import jc.utils
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.1'
description = '`os-prober` command parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
compatible = ['linux']
magic_commands = ['os-prober']
__version__ = info.version
def _process(proc_data: JSONDictType) -> JSONDictType:
"""
Final processing to conform to the schema.
Parameters:
proc_data: (List of Dictionaries) raw structured data to process
Returns:
Dictionary. Structured to conform to the schema.
"""
# check for EFI partition@boot-manager and split/add fields
if 'partition' in proc_data and '@' in proc_data['partition']:
new_part, efi_bootmgr = proc_data['partition'].split('@', maxsplit=1)
proc_data['partition'] = new_part
proc_data['efi_bootmgr'] = efi_bootmgr
return proc_data
def parse(
data: str,
raw: bool = False,
quiet: bool = False
) -> JSONDictType:
"""
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
Returns:
Dictionary. Raw or processed structured data.
"""
jc.utils.compatibility(__name__, info.compatible, quiet)
jc.utils.input_type_check(data)
raw_output: Dict = {}
if jc.utils.has_data(data):
# /dev/sda1:Windows NT/2000/XP:WinNT:chain
# ^-------^ ^----------------^ ^---^ ^---^
# part. OS name for boot short May change: type of boot loader
# loader's pretty name required. Usually there is only
# output a 'linux' style bootloader or
# a chain one for other partitions
# with their own boot sectors.
partition, name, short_name, type_ = data.split(':')
raw_output = {
'partition': partition.strip(),
'name': name.strip(),
'short_name': short_name.strip(),
'type': type_.strip()
}
return raw_output if raw else _process(raw_output)

231
jc/parsers/pci_ids.py Normal file
View File

@@ -0,0 +1,231 @@
"""jc - JSON Convert `pci.ids` file parser
This parser converts the pci.ids database file.
https://raw.githubusercontent.com/pciutils/pciids/master/pci.ids
A nested schema allows straightforward queries with tools like `jq`. Hex id
numbers are prefixed with an underscore (`_`) so bracket notation is not
necessary when referencing. For example:
$ cat pci.ids | jc --pci-ids | jq '.vendors._9005._0053._9005._ffff.subsystem_name'
"AIC-7896 SCSI Controller mainboard implementation"
Here are the vendor and class mappings:
jq '.vendors._001c._0001._001c._0005.subsystem_name'
| | | |
| | | subdevice
| | subvendor
| device
vendor
jq '.classes._0c._03._40'
| | |
| | prog_if
| subclass
class
Usage (cli):
$ cat pci.ids | jc --pci-ids
Usage (module):
import jc
result = jc.parse('pci_ids', pci_ids_file_output)
Schema:
{
"vendors": {
"_<vendor_id>": {
"vendor_name": string,
"_<device_id>": {
"device_name": string,
"_<subvendor_id>": {
"_<subdevice_id": string
}
}
}
},
"classes": {
"_<class_id>": {
"class_name": string,
"_<subclass_id>": {
"subclass_name": string,
"_<prog_if>": string
}
}
}
}
Examples:
$ cat pci.ids | jc --pci-ids | jq '.vendors._001c._0001._001c._0005.subsystem_name'
"2 Channel CAN Bus SJC1000 (Optically Isolated)"
$ cat pci.ids | jc --pci-ids | jq '.classes._0c._03._40'
"USB4 Host Interface"
"""
import re
from typing import Dict
from jc.jc_types import JSONDictType
import jc.utils
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.0'
description = '`pci.ids` file parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
compatible = ['linux', 'darwin', 'cygwin', 'win32', 'aix', 'freebsd']
__version__ = info.version
def _process(proc_data: JSONDictType) -> JSONDictType:
"""
Final processing to conform to the schema.
Parameters:
proc_data: (List of Dictionaries) raw structured data to process
Returns:
Dictionary. Structured to conform to the schema.
"""
return proc_data
def parse(
data: str,
raw: bool = False,
quiet: bool = False
) -> JSONDictType:
"""
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
Returns:
Dictionary. Raw or processed structured data.
"""
jc.utils.compatibility(__name__, info.compatible, quiet)
jc.utils.input_type_check(data)
raw_output: Dict = {}
vdc_obj: Dict = {}
vendor_id: str = ''
device_id: str = ''
class_obj: Dict = {}
class_id: str = ''
subclass_id: str = ''
if jc.utils.has_data(data):
vdc_header_p = re.compile(r'^(?P<vendor_id>[0-9a-f]{4})\s+(?P<vendor_name>.+)')
vdc_device_p = re.compile(r'^\t(?P<device_id>[0-9a-f]{4})\s+(?P<device_name>.+)')
vdc_subvendor_p = re.compile(r'^\t\t(?P<subvendor>[0-9a-f]{4})\s+(?P<subdevice>[0-9a-f]{4})\s+(?P<subsystem_name>.+)')
class_header_p = re.compile(r'^C\s+(?P<class_id>[0-9a-f]{2})\s+(?P<class_name>.+)')
class_sub_p = re.compile(r'^\t(?P<subclass_id>[0-9a-f]{2})\s+(?P<subclass_name>.+)')
class_progif_p = re.compile(r'^\t\t(?P<prog_if_id>[0-9a-f]{2})\s+(?P<prog_if_name>.+)')
for line in filter(None, data.splitlines()):
vdc_header = vdc_header_p.match(line)
vdc_device = vdc_device_p.match(line)
vdc_subvendor = vdc_subvendor_p.match(line)
class_header = class_header_p.match(line)
class_sub = class_sub_p.match(line)
class_progif = class_progif_p.match(line)
# Vendors, devices and subsystems
# Syntax:
# vendor vendor_name
# device device_name <-- single tab
# subvendor subdevice subsystem_name <-- two tabs
# Example:
# 001c PEAK-System Technik GmbH
# 0001 PCAN-PCI CAN-Bus controller
# 001c 0004 2 Channel CAN Bus SJC1000
if vdc_header:
if vdc_obj:
if 'vendors' not in raw_output:
raw_output['vendors'] = {}
raw_output['vendors'][vendor_id] = vdc_obj[vendor_id]
vdc_obj = {}
vendor_id = '_' + vdc_header.groupdict()['vendor_id']
vdc_obj[vendor_id] = {}
vdc_obj[vendor_id]['vendor_name'] = vdc_header.groupdict()['vendor_name']
continue
if vdc_device:
device_id = '_' + vdc_device.groupdict()['device_id']
vdc_obj[vendor_id][device_id] = {}
vdc_obj[vendor_id][device_id]['device_name'] = vdc_device.groupdict()['device_name']
continue
if vdc_subvendor:
subvendor = '_' + vdc_subvendor.groupdict()['subvendor']
subdevice = '_' + vdc_subvendor.groupdict()['subdevice']
vdc_obj[vendor_id][device_id][subvendor] = {}
vdc_obj[vendor_id][device_id][subvendor][subdevice] = {}
vdc_obj[vendor_id][device_id][subvendor][subdevice]['subsystem_name'] = vdc_subvendor.groupdict()['subsystem_name']
continue
# List of known device classes, subclasses and programming interfaces
# Syntax:
# C class class_name
# subclass subclass_name <-- single tab
# prog-if prog-if_name <-- two tabs
# Example:
# C 01 Mass storage controller
# 01 IDE interface
# 00 ISA Compatibility mode-only controller
if class_header:
if class_obj:
if 'classes' not in raw_output:
raw_output['classes'] = {}
raw_output['classes'][class_id] = class_obj[class_id]
class_obj = {}
class_id = '_' + class_header.groupdict()['class_id']
class_obj[class_id] = {}
class_obj[class_id]['class_name'] = class_header.groupdict()['class_name']
continue
if class_sub:
subclass_id = '_' + class_sub.groupdict()['subclass_id']
class_obj[class_id][subclass_id] = {}
class_obj[class_id][subclass_id]['subclass_name'] = class_sub.groupdict()['subclass_name']
continue
if class_progif:
prog_if_id = '_' + class_progif.groupdict()['prog_if_id']
class_obj[class_id][subclass_id][prog_if_id] = class_progif.groupdict()['prog_if_name']
continue
if vdc_obj:
if 'vendors' not in raw_output:
raw_output['vendors'] = {}
raw_output['vendors'][vendor_id] = vdc_obj[vendor_id]
if class_obj:
if 'classes' not in raw_output:
raw_output['classes'] = {}
raw_output['classes'][class_id] = class_obj[class_id]
return raw_output if raw else _process(raw_output)

125
jc/parsers/pgpass.py Normal file
View File

@@ -0,0 +1,125 @@
"""jc - JSON Convert PostgreSQL password file parser
Usage (cli):
$ cat /var/lib/postgresql/.pgpass | jc --pgpass
Usage (module):
import jc
result = jc.parse('pgpass', postgres_password_file)
Schema:
[
{
"hostname": string,
"port": string,
"database": string,
"username": string,
"password": string
}
]
Examples:
$ cat /var/lib/postgresql/.pgpass | jc --pgpass -p
[
{
"hostname": "dbserver",
"port": "*",
"database": "db1",
"username": "dbuser",
"password": "pwd123"
},
{
"hostname": "dbserver2",
"port": "8888",
"database": "inventory",
"username": "joe:user",
"password": "abc123"
},
...
]
"""
from typing import List, Dict
from jc.jc_types import JSONDictType
import jc.utils
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.0'
description = 'PostgreSQL password file parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
compatible = ['linux', 'darwin', 'cygwin', 'win32', 'aix', 'freebsd']
__version__ = info.version
def _process(proc_data: List[JSONDictType]) -> List[JSONDictType]:
"""
Final processing to conform to the schema.
Parameters:
proc_data: (List of Dictionaries) raw structured data to process
Returns:
List of Dictionaries. Structured to conform to the schema.
"""
return proc_data
def parse(
data: str,
raw: bool = False,
quiet: bool = False
) -> List[JSONDictType]:
"""
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
Returns:
List of Dictionaries. Raw or processed structured data.
"""
jc.utils.compatibility(__name__, info.compatible, quiet)
jc.utils.input_type_check(data)
raw_output: List[Dict] = []
if jc.utils.has_data(data):
for line in filter(None, data.splitlines()):
# ignore comment lines
if line.strip().startswith('#'):
continue
# convert escaped characters (\ and :)
line = line.replace(':', '\u2063')
line = line.replace('\\\\', '\\')
line = line.replace('\\\u2063', ':')
hostname, port, database, username, password = line.split('\u2063')
raw_output.append(
{
'hostname': hostname,
'port': port,
'database': database,
'username': username,
'password': password
}
)
return raw_output if raw else _process(raw_output)

View File

@@ -161,7 +161,7 @@ def parse(
continue
if not line.startswith('#') and not found_first_hash:
# skip preample lines before header row
# skip preamble lines before header row
continue
if line.startswith('#') and not found_first_hash:

View File

@@ -85,7 +85,7 @@ from jc.exceptions import ParseError
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.1'
version = '1.2'
description = '`ping` and `ping6` command streaming parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
@@ -492,7 +492,7 @@ def parse(data, raw=False, quiet=False, ignore_exceptions=False):
output_line = {}
# skip blank lines
if line.strip() == '':
if not line.strip():
continue
# skip warning lines

View File

@@ -80,7 +80,7 @@ def _process(proc_data: Dict) -> Dict:
def _b2a(byte_string: bytes) -> str:
"""Convert a byte string to a colon-delimited hex ascii string"""
# need try/except since seperator was only introduced in python 3.8.
# need try/except since separator was only introduced in python 3.8.
# provides compatibility for python 3.6 and 3.7.
try:
return binascii.hexlify(byte_string, ':').decode('utf-8')

View File

@@ -1,7 +1,7 @@
"""jc - JSON Convert Proc file output parser
This parser automatically identifies the Proc file and calls the
corresponding parser to peform the parsing.
corresponding parser to perform the parsing.
Magic syntax for converting `/proc` files is also supported by running
`jc /proc/<path to file>`. Any `jc` options must be specified before the
@@ -80,7 +80,7 @@ Examples:
...
]
$ proc_modules | jc --proc_modules -p -r
$ cat /proc/modules | jc --proc-modules -p -r
[
{
"module": "binfmt_misc",
@@ -201,7 +201,7 @@ def parse(
pid_mountinfo_p = re.compile(r'^\d+ \d+ \d+:\d+ /.+\n')
pid_numa_maps_p = re.compile(r'^[a-f0-9]{12} default [^\n]+\n')
pid_smaps_p = re.compile(r'^[0-9a-f]{12}-[0-9a-f]{12} [rwxsp\-]{4} [0-9a-f]{8} [0-9a-f]{2}:[0-9a-f]{2} \d+ [^\n]+\nSize:\s+\d+ \S\S\n')
pid_stat_p = re.compile(r'^\d+ \(.{1,16}\) \w \d+ \d+ \d+ \d+ -?\d+ (?:\d+ ){43}\d+$')
pid_stat_p = re.compile(r'^\d+ \(.{1,15}\) \S \d+ \d+ \d+ \d+ -?\d+ (?:\d+ ){43}\d+$', re.DOTALL)
pid_statm_p = re.compile(r'^\d+ \d+ \d+\s\d+\s\d+\s\d+\s\d+$')
pid_status_p = re.compile(r'^Name:\t.+\nUmask:\t\d+\nState:\t.+\nTgid:\t\d+\n')

View File

@@ -195,13 +195,14 @@ Examples:
"exit_code": 0
}
"""
import re
from typing import Dict
import jc.utils
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.0'
version = '1.1'
description = '`/proc/<pid>/stat` file parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
@@ -242,6 +243,12 @@ def _process(proc_data: Dict) -> Dict:
if 'state' in proc_data:
proc_data['state_pretty'] = state_map[proc_data['state']]
for key, val in proc_data.items():
try:
proc_data[key] = int(val)
except Exception:
pass
return proc_data
@@ -270,74 +277,65 @@ def parse(
if jc.utils.has_data(data):
split_line = data.split()
raw_output = {
'pid': int(split_line[0]),
'comm': split_line[1].strip('()'),
'state': split_line[2],
'ppid': int(split_line[3]),
'pgrp': int(split_line[4]),
'session': int(split_line[5]),
'tty_nr': int(split_line[6]),
'tpg_id': int(split_line[7]),
'flags': int(split_line[8]),
'minflt': int(split_line[9]),
'cminflt': int(split_line[10]),
'majflt': int(split_line[11]),
'cmajflt': int(split_line[12]),
'utime': int(split_line[13]),
'stime': int(split_line[14]),
'cutime': int(split_line[15]),
'cstime': int(split_line[16]),
'priority': int(split_line[17]),
'nice': int(split_line[18]),
'num_threads': int(split_line[19]),
'itrealvalue': int(split_line[20]),
'starttime': int(split_line[21]),
'vsize': int(split_line[22]),
'rss': int(split_line[23]),
'rsslim': int(split_line[24]),
'startcode': int(split_line[25]),
'endcode': int(split_line[26]),
'startstack': int(split_line[27]),
'kstkeep': int(split_line[28]),
'kstkeip': int(split_line[29]),
'signal': int(split_line[30]),
'blocked': int(split_line[31]),
'sigignore': int(split_line[32]),
'sigcatch': int(split_line[33]),
'wchan': int(split_line[34]),
'nswap': int(split_line[35]),
'cnswap': int(split_line[36])
}
line_re = re.compile(r'''
^(?P<pid>\d+)\s
\((?P<comm>.+)\)\s
(?P<state>\S)\s
(?P<ppid>\d+)\s
(?P<pgrp>\d+)\s
(?P<session>\d+)\s
(?P<tty_nr>\d+)\s
(?P<tpg_id>-?\d+)\s
(?P<flags>\d+)\s
(?P<minflt>\d+)\s
(?P<cminflt>\d+)\s
(?P<majflt>\d+)\s
(?P<cmajflt>\d+)\s
(?P<utime>\d+)\s
(?P<stime>\d+)\s
(?P<cutime>\d+)\s
(?P<cstime>\d+)\s
(?P<priority>\d+)\s
(?P<nice>\d+)\s
(?P<num_threads>\d+)\s
(?P<itrealvalue>\d+)\s
(?P<starttime>\d+)\s
(?P<vsize>\d+)\s
(?P<rss>\d+)\s
(?P<rsslim>\d+)\s
(?P<startcode>\d+)\s
(?P<endcode>\d+)\s
(?P<startstack>\d+)\s
(?P<kstkeep>\d+)\s
(?P<kstkeip>\d+)\s
(?P<signal>\d+)\s
(?P<blocked>\d+)\s
(?P<sigignore>\d+)\s
(?P<sigcatch>\d+)\s
(?P<wchan>\d+)\s
(?P<nswap>\d+)\s
(?P<cnswap>\d+)\s
(?P<exit_signal>\d+)\s
(?P<processor>\d+)\s
(?P<rt_priority>\d+)\s
(?P<policy>\d+)\s
(?P<delayacct_blkio_ticks>\d+)\s
(?P<guest_time>\d+)\s
(?P<cguest_time>\d+)\s
(?P<start_data>\d+)\s
(?P<end_data>\d+)\s
(?P<start_brk>\d+)\s
(?P<arg_start>\d+)\s
(?P<arg_end>\d+)\s
(?P<env_start>\d+)\s
(?P<env_end>\d+)\s
(?P<exit_code>\d+)
''', re.VERBOSE | re.DOTALL
)
if len(split_line) > 37:
raw_output['exit_signal'] = int(split_line[37])
line_match = line_re.search(data)
if len(split_line) > 38:
raw_output['processor'] = int(split_line[38])
if len(split_line) > 39:
raw_output['rt_priority'] = int(split_line[39])
raw_output['policy'] = int(split_line[40])
if len(split_line) > 41:
raw_output['delayacct_blkio_ticks'] = int(split_line[41])
if len(split_line) > 42:
raw_output['guest_time'] = int(split_line[42])
raw_output['cguest_time'] = int(split_line[43])
if len(split_line) > 44:
raw_output['start_data'] = int(split_line[44])
raw_output['end_data'] = int(split_line[45])
raw_output['start_brk'] = int(split_line[46])
if len(split_line) > 47:
raw_output['arg_start'] = int(split_line[47])
raw_output['arg_end'] = int(split_line[48])
raw_output['env_start'] = int(split_line[49])
raw_output['env_end'] = int(split_line[50])
raw_output['exit_code'] = int(split_line[51])
if line_match:
raw_output = line_match.groupdict()
return raw_output if raw else _process(raw_output)

View File

@@ -88,7 +88,7 @@ from jc.streaming import (
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.1'
version = '1.2'
description = '`rsync` command streaming parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
@@ -267,7 +267,7 @@ def parse(
output_line: Dict = {}
# ignore blank lines
if line == '':
if not line.strip():
continue
file_line = file_line_re.match(line)

121
jc/parsers/semver.py Normal file
View File

@@ -0,0 +1,121 @@
"""jc - JSON Convert Semantic Version string parser
This parser conforms to the specification at https://semver.org/
Usage (cli):
$ echo 1.2.3-rc.1+44837 | jc --semver
Usage (module):
import jc
result = jc.parse('semver', semver_string)
Schema:
Strings that do not strictly conform to the specification will return an
empty object.
{
"major": integer,
"minor": integer,
"patch": integer,
"prerelease": string/null,
"build": string/null
}
Examples:
$ echo 1.2.3-rc.1+44837 | jc --semver -p
{
"major": 1,
"minor": 2,
"patch": 3,
"prerelease": "rc.1",
"build": "44837"
}
$ echo 1.2.3-rc.1+44837 | jc --semver -p -r
{
"major": "1",
"minor": "2",
"patch": "3",
"prerelease": "rc.1",
"build": "44837"
}
"""
import re
from typing import Set, Dict
from jc.jc_types import JSONDictType
import jc.utils
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.0'
description = 'Semantic Version string parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
compatible = ['linux', 'darwin', 'cygwin', 'win32', 'aix', 'freebsd']
__version__ = info.version
def _process(proc_data: JSONDictType) -> JSONDictType:
"""
Final processing to conform to the schema.
Parameters:
proc_data: (List of Dictionaries) raw structured data to process
Returns:
Dictionary. Structured to conform to the schema.
"""
int_list: Set[str] = {'major', 'minor', 'patch'}
for item in int_list:
if item in proc_data:
proc_data[item] = int(proc_data[item])
return proc_data
def parse(
data: str,
raw: bool = False,
quiet: bool = False
) -> JSONDictType:
"""
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
Returns:
Dictionary. Raw or processed structured data.
"""
jc.utils.compatibility(__name__, info.compatible, quiet)
jc.utils.input_type_check(data)
raw_output: Dict = {}
semver_pattern = re.compile(r'''
^(?P<major>0|[1-9]\d*)\.
(?P<minor>0|[1-9]\d*)\.
(?P<patch>0|[1-9]\d*)
(?:-(?P<prerelease>(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?
(?:\+(?P<build>[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$
''', re.VERBOSE)
if jc.utils.has_data(data):
semver_match = re.match(semver_pattern, data)
if semver_match:
raw_output = semver_match.groupdict()
return raw_output if raw else _process(raw_output)

660
jc/parsers/sshd_conf.py Normal file
View File

@@ -0,0 +1,660 @@
"""jc - JSON Convert sshd configuration file and `sshd -T` command output parser
This parser will work with `sshd` configuration files or the output of
`sshd -T`. Any `Match` blocks in the `sshd` configuration file will be
ignored.
Usage (cli):
$ sshd -T | jc --sshd-conf
or
$ jc sshd -T
or
$ cat sshd_conf | jc --sshd-conf
Usage (module):
import jc
result = jc.parse('sshd_conf', sshd_conf_output)
Schema:
{
"acceptenv": [
string
],
"addressfamily": string,
"allowagentforwarding": string,
"allowstreamlocalforwarding": string,
"allowtcpforwarding": string,
"authenticationmethods": string,
"authorizedkeyscommand": string,
"authorizedkeyscommanduser": string,
"authorizedkeysfile": [
string
],
"authorizedprincipalscommand": string,
"authorizedprincipalscommanduser": string,
"authorizedprincipalsfile": string,
"banner": string,
"casignaturealgorithms": [
string
],
"chrootdirectory": string,
"ciphers": [
string
],
"ciphers_strategy": string,
"clientalivecountmax": integer,
"clientaliveinterval": integer,
"compression": string,
"disableforwarding": string,
"exposeauthinfo": string,
"fingerprinthash": string,
"forcecommand": string,
"gatewayports": string,
"gssapiauthentication": string,
"gssapicleanupcredentials": string,
"gssapikexalgorithms": [
string
],
"gssapikeyexchange": string,
"gssapistorecredentialsonrekey": string,
"gssapistrictacceptorcheck": string,
"hostbasedacceptedalgorithms": [
string
],
"hostbasedauthentication": string,
"hostbasedusesnamefrompacketonly": string,
"hostkeyagent": string,
"hostkeyalgorithms": [
string
],
"hostkey": [
string
],
"ignorerhosts": string,
"ignoreuserknownhosts": string,
"include": [
string
],
"ipqos": [
string
],
"kbdinteractiveauthentication": string,
"kerberosauthentication": string,
"kerberosorlocalpasswd": string,
"kerberosticketcleanup": sttring,
"kexalgorithms": [
string
],
"listenaddress": [
string
],
"logingracetime": integer,
"loglevel": string,
"macs": [
string
],
"macs_strategy": string,
"maxauthtries": integer,
"maxsessions": integer,
"maxstartups": integer,
"maxstartups_rate": integer,
"maxstartups_full": integer,
"modulifile": string,
"passwordauthentication": string,
"permitemptypasswords": string,
"permitlisten": [
string
],
"permitopen": [
string
],
"permitrootlogin": string,
"permittty": string,
"permittunnel": string,
"permituserenvironment": string,
"permituserrc": string,
"persourcemaxstartups": string,
"persourcenetblocksize": string,
"pidfile": string,
"port": [
integer
],
"printlastlog": string,
"printmotd": string,
"pubkeyacceptedalgorithms": [
string
],
"pubkeyauthentication": string,
"pubkeyauthoptions": string,
"rekeylimit": integer,
"rekeylimit_time": integer,
"revokedkeys": string,
"securitykeyprovider": string,
"streamlocalbindmask": string,
"streamlocalbindunlink": string,
"strictmodes": string,
"subsystem": string,
"subsystem_command": string
"syslogfacility": string,
"tcpkeepalive": string,
"trustedusercakeys": string,
"usedns": string,
"usepam": string,
"versionaddendum": string,
"x11displayoffset": integer,
"x11forwarding": string,
"x11uselocalhost": string,
"xauthlocation": string
}
Examples:
$ sshd -T | jc --sshd-conf -p
{
"acceptenv": [
"LANG",
"LC_*"
],
"addressfamily": "any",
"allowagentforwarding": "yes",
"allowstreamlocalforwarding": "yes",
"allowtcpforwarding": "yes",
"authenticationmethods": "any",
"authorizedkeyscommand": "none",
"authorizedkeyscommanduser": "none",
"authorizedkeysfile": [
".ssh/authorized_keys",
".ssh/authorized_keys2"
],
"authorizedprincipalscommand": "none",
"authorizedprincipalscommanduser": "none",
"authorizedprincipalsfile": "none",
"banner": "none",
"casignaturealgorithms": [
"ssh-ed25519",
"ecdsa-sha2-nistp256",
"ecdsa-sha2-nistp384",
"ecdsa-sha2-nistp521",
"sk-ssh-ed25519@openssh.com",
"sk-ecdsa-sha2-nistp256@openssh.com",
"rsa-sha2-512",
"rsa-sha2-256"
],
"chrootdirectory": "none",
"ciphers": [
"chacha20-poly1305@openssh.com",
"aes128-ctr",
"aes192-ctr",
"aes256-ctr",
"aes128-gcm@openssh.com",
"aes256-gcm@openssh.com"
],
"ciphers_strategy": "+",
"clientalivecountmax": 3,
"clientaliveinterval": 0,
"compression": "yes",
"disableforwarding": "no",
"exposeauthinfo": "no",
"fingerprinthash": "SHA256",
"forcecommand": "none",
"gatewayports": "no",
"gssapiauthentication": "no",
"gssapicleanupcredentials": "yes",
"gssapikexalgorithms": [
"gss-group14-sha256-",
"gss-group16-sha512-",
"gss-nistp256-sha256-",
"gss-curve25519-sha256-",
"gss-group14-sha1-",
"gss-gex-sha1-"
],
"gssapikeyexchange": "no",
"gssapistorecredentialsonrekey": "no",
"gssapistrictacceptorcheck": "yes",
"hostbasedacceptedalgorithms": [
"ssh-ed25519-cert-v01@openssh.com",
"ecdsa-sha2-nistp256-cert-v01@openssh.com",
"ecdsa-sha2-nistp384-cert-v01@openssh.com",
"ecdsa-sha2-nistp521-cert-v01@openssh.com",
"sk-ssh-ed25519-cert-v01@openssh.com",
"sk-ecdsa-sha2-nistp256-cert-v01@openssh.com",
"rsa-sha2-512-cert-v01@openssh.com",
"rsa-sha2-256-cert-v01@openssh.com",
"ssh-ed25519",
"ecdsa-sha2-nistp256",
"ecdsa-sha2-nistp384",
"ecdsa-sha2-nistp521",
"sk-ssh-ed25519@openssh.com",
"sk-ecdsa-sha2-nistp256@openssh.com",
"rsa-sha2-512",
"rsa-sha2-256"
],
"hostbasedauthentication": "no",
"hostbasedusesnamefrompacketonly": "no",
"hostkeyagent": "none",
"hostkeyalgorithms": [
"ssh-ed25519-cert-v01@openssh.com",
"ecdsa-sha2-nistp256-cert-v01@openssh.com",
"ecdsa-sha2-nistp384-cert-v01@openssh.com",
"ecdsa-sha2-nistp521-cert-v01@openssh.com",
"sk-ssh-ed25519-cert-v01@openssh.com",
"sk-ecdsa-sha2-nistp256-cert-v01@openssh.com",
"rsa-sha2-512-cert-v01@openssh.com",
"rsa-sha2-256-cert-v01@openssh.com",
"ssh-ed25519",
"ecdsa-sha2-nistp256",
"ecdsa-sha2-nistp384",
"ecdsa-sha2-nistp521",
"sk-ssh-ed25519@openssh.com",
"sk-ecdsa-sha2-nistp256@openssh.com",
"rsa-sha2-512",
"rsa-sha2-256"
],
"hostkey": [
"/etc/ssh/ssh_host_ecdsa_key",
"/etc/ssh/ssh_host_ed25519_key",
"/etc/ssh/ssh_host_rsa_key"
],
"ignorerhosts": "yes",
"ignoreuserknownhosts": "no",
"ipqos": [
"lowdelay",
"throughput"
],
"kbdinteractiveauthentication": "no",
"kerberosauthentication": "no",
"kerberosorlocalpasswd": "yes",
"kerberosticketcleanup": "yes",
"kexalgorithms": [
"sntrup761x25519-sha512@openssh.com",
"curve25519-sha256",
"curve25519-sha256@libssh.org",
"ecdh-sha2-nistp256",
"ecdh-sha2-nistp384",
"ecdh-sha2-nistp521",
"diffie-hellman-group-exchange-sha256",
"diffie-hellman-group16-sha512",
"diffie-hellman-group18-sha512",
"diffie-hellman-group14-sha256"
],
"listenaddress": [
"0.0.0.0:22",
"[::]:22"
],
"logingracetime": 120,
"loglevel": "INFO",
"macs": [
"umac-64-etm@openssh.com",
"umac-128-etm@openssh.com",
"hmac-sha2-256-etm@openssh.com",
"hmac-sha2-512-etm@openssh.com",
"hmac-sha1-etm@openssh.com",
"umac-64@openssh.com",
"umac-128@openssh.com",
"hmac-sha2-256",
"hmac-sha2-512",
"hmac-sha1"
],
"macs_strategy": "^",
"maxauthtries": 6,
"maxsessions": 10,
"maxstartups": 10,
"modulifile": "/etc/ssh/moduli",
"passwordauthentication": "yes",
"permitemptypasswords": "no",
"permitlisten": [
"any"
],
"permitopen": [
"any"
],
"permitrootlogin": "without-password",
"permittty": "yes",
"permittunnel": "no",
"permituserenvironment": "no",
"permituserrc": "yes",
"persourcemaxstartups": "none",
"persourcenetblocksize": "32:128",
"pidfile": "/run/sshd.pid",
"port": [
22
],
"printlastlog": "yes",
"printmotd": "no",
"pubkeyacceptedalgorithms": [
"ssh-ed25519-cert-v01@openssh.com",
"ecdsa-sha2-nistp256-cert-v01@openssh.com",
"ecdsa-sha2-nistp384-cert-v01@openssh.com",
"ecdsa-sha2-nistp521-cert-v01@openssh.com",
"sk-ssh-ed25519-cert-v01@openssh.com",
"sk-ecdsa-sha2-nistp256-cert-v01@openssh.com",
"rsa-sha2-512-cert-v01@openssh.com",
"rsa-sha2-256-cert-v01@openssh.com",
"ssh-ed25519",
"ecdsa-sha2-nistp256",
"ecdsa-sha2-nistp384",
"ecdsa-sha2-nistp521",
"sk-ssh-ed25519@openssh.com",
"sk-ecdsa-sha2-nistp256@openssh.com",
"rsa-sha2-512",
"rsa-sha2-256"
],
"pubkeyauthentication": "yes",
"pubkeyauthoptions": "none",
"rekeylimit": 0,
"revokedkeys": "none",
"securitykeyprovider": "internal",
"streamlocalbindmask": "0177",
"streamlocalbindunlink": "no",
"strictmodes": "yes",
"subsystem": "sftp",
"syslogfacility": "AUTH",
"tcpkeepalive": "yes",
"trustedusercakeys": "none",
"usedns": "no",
"usepam": "yes",
"versionaddendum": "none",
"x11displayoffset": 10,
"x11forwarding": "yes",
"x11uselocalhost": "yes",
"xauthlocation": "/usr/bin/xauth",
"maxstartups_rate": 30,
"maxstartups_full": 100,
"rekeylimit_time": 0,
"subsystem_command": "/usr/lib/openssh/sftp-server"
}
$ sshd -T | jc --sshd-conf -p -r
{
"acceptenv": [
"LANG",
"LC_*"
],
"addressfamily": "any",
"allowagentforwarding": "yes",
"allowstreamlocalforwarding": "yes",
"allowtcpforwarding": "yes",
"authenticationmethods": "any",
"authorizedkeyscommand": "none",
"authorizedkeyscommanduser": "none",
"authorizedkeysfile": ".ssh/authorized_keys .ssh/authorized_keys2",
"authorizedprincipalscommand": "none",
"authorizedprincipalscommanduser": "none",
"authorizedprincipalsfile": "none",
"banner": "none",
"casignaturealgorithms": "ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-s...",
"chrootdirectory": "none",
"ciphers": "chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,...",
"ciphers_strategy": "+",
"clientalivecountmax": "3",
"clientaliveinterval": "0",
"compression": "yes",
"disableforwarding": "no",
"exposeauthinfo": "no",
"fingerprinthash": "SHA256",
"forcecommand": "none",
"gatewayports": "no",
"gssapiauthentication": "no",
"gssapicleanupcredentials": "yes",
"gssapikexalgorithms": "gss-group14-sha256-,gss-group16-sha512-,...",
"gssapikeyexchange": "no",
"gssapistorecredentialsonrekey": "no",
"gssapistrictacceptorcheck": "yes",
"hostbasedacceptedalgorithms": "ssh-ed25519-cert-v01@openssh.co...",
"hostbasedauthentication": "no",
"hostbasedusesnamefrompacketonly": "no",
"hostkeyagent": "none",
"hostkeyalgorithms": "ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2...",
"hostkey": [
"/etc/ssh/ssh_host_ecdsa_key",
"/etc/ssh/ssh_host_ed25519_key",
"/etc/ssh/ssh_host_rsa_key"
],
"ignorerhosts": "yes",
"ignoreuserknownhosts": "no",
"ipqos": "lowdelay throughput",
"kbdinteractiveauthentication": "no",
"kerberosauthentication": "no",
"kerberosorlocalpasswd": "yes",
"kerberosticketcleanup": "yes",
"kexalgorithms": "sntrup761x25519-sha512@openssh.com,curve25519...",
"listenaddress": [
"0.0.0.0:22",
"[::]:22"
],
"logingracetime": "120",
"loglevel": "INFO",
"macs": "umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac...",
"macs_strategy": "^",
"maxauthtries": "6",
"maxsessions": "10",
"maxstartups": "10:30:100",
"modulifile": "/etc/ssh/moduli",
"passwordauthentication": "yes",
"permitemptypasswords": "no",
"permitlisten": "any",
"permitopen": "any",
"permitrootlogin": "without-password",
"permittty": "yes",
"permittunnel": "no",
"permituserenvironment": "no",
"permituserrc": "yes",
"persourcemaxstartups": "none",
"persourcenetblocksize": "32:128",
"pidfile": "/run/sshd.pid",
"port": [
"22"
],
"printlastlog": "yes",
"printmotd": "no",
"pubkeyacceptedalgorithms": "ssh-ed25519-cert-v01@openssh.com,...",
"pubkeyauthentication": "yes",
"pubkeyauthoptions": "none",
"rekeylimit": "0 0",
"revokedkeys": "none",
"securitykeyprovider": "internal",
"streamlocalbindmask": "0177",
"streamlocalbindunlink": "no",
"strictmodes": "yes",
"subsystem": "sftp /usr/lib/openssh/sftp-server",
"syslogfacility": "AUTH",
"tcpkeepalive": "yes",
"trustedusercakeys": "none",
"usedns": "no",
"usepam": "yes",
"versionaddendum": "none",
"x11displayoffset": "10",
"x11forwarding": "yes",
"x11uselocalhost": "yes",
"xauthlocation": "/usr/bin/xauth"
}
"""
from typing import Set, List, Dict
from jc.jc_types import JSONDictType
import jc.utils
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.0'
description = 'sshd config file and `sshd -T` command parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
compatible = ['linux', 'darwin', 'freebsd']
magic_commands = ['sshd -T']
__version__ = info.version
def _process(proc_data: JSONDictType) -> JSONDictType:
"""
Final processing to conform to the schema.
Parameters:
proc_data: (Dictionary) raw structured data to process
Returns:
Dictionary. Structured to conform to the schema.
"""
split_fields_space: Set[str] = {
'authorizedkeysfile', 'include', 'ipqos', 'permitlisten', 'permitopen'
}
split_fields_comma: Set[str] = {
'casignaturealgorithms', 'ciphers', 'gssapikexalgorithms', 'hostbasedacceptedalgorithms',
'hostbasedacceptedkeytypes', 'hostkeyalgorithms', 'kexalgorithms', 'macs',
'pubkeyacceptedalgorithms', 'pubkeyacceptedkeytypes'
}
int_list: Set[str] = {'clientalivecountmax', 'clientaliveinterval', 'logingracetime',
'maxauthtries', 'maxsessions', 'maxstartups', 'maxstartups_rate', 'maxstartups_full',
'rekeylimit', 'rekeylimit_time', 'x11displayoffset', 'x11maxdisplays'
}
dict_copy = proc_data.copy()
for key, val in dict_copy.items():
# this is a list value
if key == 'acceptenv':
new_list: List[str] = []
for item in val:
new_list.extend(item.split())
proc_data[key] = new_list
continue
# this is a list value
if key == 'include':
new_list = []
for item in val:
new_list.extend(item.split())
proc_data[key] = new_list
continue
if key == 'maxstartups':
maxstart_split = val.split(':', maxsplit=2)
proc_data[key] = maxstart_split[0]
if len(maxstart_split) > 1:
proc_data[key + '_rate'] = maxstart_split[1]
if len(maxstart_split) > 2:
proc_data[key + '_full'] = maxstart_split[2]
continue
if key == 'port':
port_list: List[int] = []
for item in val:
port_list.append(int(item))
proc_data[key] = port_list
continue
if key == 'rekeylimit':
rekey_split = val.split(maxsplit=1)
proc_data[key] = rekey_split[0]
if len(rekey_split) > 1:
proc_data[key + '_time'] = rekey_split[1]
continue
if key == 'subsystem':
sub_split = val.split(maxsplit=1)
proc_data[key] = sub_split[0]
if len(sub_split) > 1:
proc_data[key + '_command'] = sub_split[1]
continue
if key in split_fields_space:
proc_data[key] = val.split()
continue
if key in split_fields_comma:
proc_data[key] = val.split(',')
continue
for key, val in proc_data.items():
if key in int_list:
proc_data[key] = jc.utils.convert_to_int(val)
return proc_data
def parse(
data: str,
raw: bool = False,
quiet: bool = False
) -> JSONDictType:
"""
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
Returns:
Dictionary. Raw or processed structured data.
"""
jc.utils.compatibility(__name__, info.compatible, quiet)
jc.utils.input_type_check(data)
raw_output: Dict = {}
multi_fields: Set[str] = {'acceptenv', 'hostkey', 'include', 'listenaddress', 'port'}
modified_fields: Set[str] = {
'casignaturealgorithms', 'ciphers', 'hostbasedacceptedalgorithms',
'kexalgorithms', 'macs', 'pubkeyacceptedalgorithms'
}
modifiers: Set[str] = {'+', '-', '^'}
match_block_found = False
if jc.utils.has_data(data):
for line in filter(None, data.splitlines()):
# support configuration file by skipping commented lines
if line.strip().startswith('#'):
continue
# support configuration file by ignoring all lines between
# Match xxx and Match any
if line.strip().startswith('Match all'):
match_block_found = False
continue
if line.strip().startswith('Match'):
match_block_found = True
continue
if match_block_found:
continue
key, val = line.split(maxsplit=1)
# support configuration file by converting to lower case
key = key.lower()
if key in multi_fields:
if key not in raw_output:
raw_output[key] = []
raw_output[key].append(val)
continue
if key in modified_fields and val[0] in modifiers:
raw_output[key] = val[1:]
raw_output[key + '_strategy'] = val[0]
continue
raw_output[key] = val
continue
return raw_output if raw else _process(raw_output)

View File

@@ -77,12 +77,14 @@ import jc.utils
from jc.streaming import (
add_jc_meta, streaming_input_type_check, streaming_line_input_type_check, raise_or_yield
)
from typing import Dict, Iterable
from jc.jc_types import JSONDictType, StreamingOutputType
from jc.exceptions import ParseError
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.1'
version = '1.3'
description = '`stat` command streaming parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
@@ -93,7 +95,7 @@ class info():
__version__ = info.version
def _process(proc_data):
def _process(proc_data: JSONDictType) -> JSONDictType:
"""
Final processing to conform to the schema.
@@ -105,10 +107,10 @@ def _process(proc_data):
Dictionary. Structured data to conform to the schema.
"""
int_list = {'size', 'blocks', 'io_blocks', 'inode', 'links', 'uid', 'gid',
int_list: set[str] = {'size', 'blocks', 'io_blocks', 'inode', 'links', 'uid', 'gid',
'unix_device', 'rdev', 'block_size'}
null_list = {'access_time', 'modify_time', 'change_time', 'birth_time'}
null_list: set[str] = {'access_time', 'modify_time', 'change_time', 'birth_time'}
for key in proc_data.copy():
if key in int_list:
@@ -118,15 +120,23 @@ def _process(proc_data):
if key in null_list:
if proc_data[key] == '-':
proc_data[key] = None
ts = jc.utils.timestamp(proc_data[key], format_hint=(7100, 7200))
proc_data[key + '_epoch'] = ts.naive
proc_data[key + '_epoch_utc'] = ts.utc
ts_string = proc_data[key]
if isinstance(ts_string, str) or ts_string is None:
ts = jc.utils.timestamp(ts_string, format_hint=(7100, 7200))
proc_data[key + '_epoch'] = ts.naive
proc_data[key + '_epoch_utc'] = ts.utc
return proc_data
@add_jc_meta
def parse(data, raw=False, quiet=False, ignore_exceptions=False):
def parse(
data: Iterable[str],
raw: bool = False,
quiet: bool = False,
ignore_exceptions: bool = False
) -> StreamingOutputType:
"""
Main text parsing generator function. Returns an iterable object.
@@ -146,7 +156,7 @@ def parse(data, raw=False, quiet=False, ignore_exceptions=False):
jc.utils.compatibility(__name__, info.compatible, quiet)
streaming_input_type_check(data)
output_line = {}
output_line: Dict = {}
os_type = ''
for line in data:
@@ -155,7 +165,7 @@ def parse(data, raw=False, quiet=False, ignore_exceptions=False):
line = line.rstrip()
# ignore blank lines
if line == '':
if not line.strip():
continue
# linux output

View File

@@ -48,7 +48,7 @@ Blank values converted to `null`/`None`.
]
[0] naive timestamp if "timestamp" field is parsable, else null
[1] timezone aware timestamp availabe for UTC, else null
[1] timezone aware timestamp available for UTC, else null
[2] this field exists if the syslog line is not parsable. The value
is the original syslog line.

View File

@@ -59,7 +59,7 @@ Blank values converted to `null`/`None`.
}
[0] naive timestamp if "timestamp" field is parsable, else null
[1] timezone aware timestamp availabe for UTC, else null
[1] timezone aware timestamp available for UTC, else null
[2] this field exists if the syslog line is not parsable. The value
is the original syslog line.

203
jc/parsers/udevadm.py Normal file
View File

@@ -0,0 +1,203 @@
"""jc - JSON Convert `udevadm info` command output parser
Usage (cli):
$ udevadm info --query=all /dev/sda | jc --udevadm
or
$ jc udevadm info --query=all /dev/sda
Usage (module):
import jc
result = jc.parse('udevadm', udevadm_command_output)
Schema:
{
"P": string,
"N": string,
"L": integer,
"S": [
string
],
"E": {
"<key>": string
}
}
Examples:
$ udevadm info --query=all /dev/sda | jc --udevadm -p
{
"P": "/devices/pci0000:00/0000:00:10.0/host32/target32:0:0/32:0:0:0/block/sda",
"N": "sda",
"L": 0,
"S": [
"disk/by-path/pci-0000:00:10.0-scsi-0:0:0:0"
],
"E": {
"DEVPATH": "/devices/pci0000:00/0000:00:10.0/host32/target32:0:0/32:0:0:0/block/sda",
"DEVNAME": "/dev/sda",
"DEVTYPE": "disk",
"MAJOR": "8",
"MINOR": "0",
"SUBSYSTEM": "block",
"USEC_INITIALIZED": "6100111",
"SCSI_TPGS": "0",
"SCSI_TYPE": "disk",
"SCSI_VENDOR": "VMware,",
"SCSI_VENDOR_ENC": "VMware,\\x20",
"SCSI_MODEL": "VMware_Virtual_S",
"SCSI_MODEL_ENC": "VMware\\x20Virtual\\x20S",
"SCSI_REVISION": "1.0",
"ID_SCSI": "1",
"ID_VENDOR": "VMware_",
"ID_VENDOR_ENC": "VMware\\x2c\\x20",
"ID_MODEL": "VMware_Virtual_S",
"ID_MODEL_ENC": "VMware\\x20Virtual\\x20S",
"ID_REVISION": "1.0",
"ID_TYPE": "disk",
"MPATH_SBIN_PATH": "/sbin",
"ID_BUS": "scsi",
"ID_PATH": "pci-0000:00:10.0-scsi-0:0:0:0",
"ID_PATH_TAG": "pci-0000_00_10_0-scsi-0_0_0_0",
"ID_PART_TABLE_UUID": "a5bd0c01-4210-46f2-b558-5c11c209a8f7",
"ID_PART_TABLE_TYPE": "gpt",
"DEVLINKS": "/dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:0:0",
"TAGS": ":systemd:"
}
}
$ udevadm info --query=all /dev/sda | jc --udevadm -p -r
{
"P": "/devices/pci0000:00/0000:00:10.0/host32/target32:0:0/32:0:0:0/block/sda",
"N": "sda",
"L": "0",
"S": [
"disk/by-path/pci-0000:00:10.0-scsi-0:0:0:0"
],
"E": {
"DEVPATH": "/devices/pci0000:00/0000:00:10.0/host32/target32:0:0/32:0:0:0/block/sda",
"DEVNAME": "/dev/sda",
"DEVTYPE": "disk",
"MAJOR": "8",
"MINOR": "0",
"SUBSYSTEM": "block",
"USEC_INITIALIZED": "6100111",
"SCSI_TPGS": "0",
"SCSI_TYPE": "disk",
"SCSI_VENDOR": "VMware,",
"SCSI_VENDOR_ENC": "VMware,\\x20",
"SCSI_MODEL": "VMware_Virtual_S",
"SCSI_MODEL_ENC": "VMware\\x20Virtual\\x20S",
"SCSI_REVISION": "1.0",
"ID_SCSI": "1",
"ID_VENDOR": "VMware_",
"ID_VENDOR_ENC": "VMware\\x2c\\x20",
"ID_MODEL": "VMware_Virtual_S",
"ID_MODEL_ENC": "VMware\\x20Virtual\\x20S",
"ID_REVISION": "1.0",
"ID_TYPE": "disk",
"MPATH_SBIN_PATH": "/sbin",
"ID_BUS": "scsi",
"ID_PATH": "pci-0000:00:10.0-scsi-0:0:0:0",
"ID_PATH_TAG": "pci-0000_00_10_0-scsi-0_0_0_0",
"ID_PART_TABLE_UUID": "a5bd0c01-4210-46f2-b558-5c11c209a8f7",
"ID_PART_TABLE_TYPE": "gpt",
"DEVLINKS": "/dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:0:0",
"TAGS": ":systemd:"
}
}
"""
from typing import List, Dict
from jc.jc_types import JSONDictType
import jc.utils
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.0'
description = '`udevadm info` command parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
compatible = ['linux']
magic_commands = ['udevadm info']
__version__ = info.version
def _process(proc_data: JSONDictType) -> JSONDictType:
"""
Final processing to conform to the schema.
Parameters:
proc_data: (List of Dictionaries) raw structured data to process
Returns:
List of Dictionaries. Structured to conform to the schema.
"""
if 'L' in proc_data:
proc_data['L'] = int(proc_data['L'])
return proc_data
def parse(
data: str,
raw: bool = False,
quiet: bool = False
) -> JSONDictType:
"""
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
Returns:
Dictionary. Raw or processed structured data.
"""
jc.utils.compatibility(__name__, info.compatible, quiet)
jc.utils.input_type_check(data)
raw_output: Dict = {}
s_list: List = []
e_list: List = []
if jc.utils.has_data(data):
for line in filter(None, data.splitlines()):
prefix, value = line.split(maxsplit=1)
if prefix == 'P:':
raw_output['P'] = value
continue
if prefix == 'S:':
s_list.append(value)
continue
if prefix == 'E:':
e_list.append(value)
continue
raw_output[prefix[:-1]] = value
if s_list:
raw_output['S'] = s_list
if e_list:
raw_output['E'] = {}
for item in e_list:
k, v = item.split('=')
raw_output['E'][k] = v
return raw_output if raw else _process(raw_output)

View File

@@ -100,7 +100,7 @@ from jc.exceptions import ParseError
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.1'
version = '1.2'
description = '`vmstat` command streaming parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
@@ -177,7 +177,7 @@ def parse(data, raw=False, quiet=False, ignore_exceptions=False):
output_line = {}
# skip blank lines
if line.strip() == '':
if not line.strip():
continue
# detect output type

View File

@@ -441,7 +441,7 @@ def _i2b(integer: int) -> bytes:
def _b2a(byte_string: bytes) -> str:
"""Convert a byte string to a colon-delimited hex ascii string"""
# need try/except since seperator was only introduced in python 3.8.
# need try/except since separator was only introduced in python 3.8.
# provides compatibility for python 3.6 and 3.7.
try:
return binascii.hexlify(byte_string, ':').decode('utf-8')

View File

@@ -1,5 +1,10 @@
"""jc - JSON Convert `XML` file parser
This parser adds a `@` prefix to attributes by default. This can be changed
to a `_` prefix by using the `-r` (cli) or `raw=True` (module) option.
Text values for nodes will have the key-name of `#text`.
Usage (cli):
$ cat foo.xml | jc --xml
@@ -68,10 +73,15 @@ Examples:
import jc.utils
from jc.exceptions import LibraryNotInstalled
try:
import xmltodict
except Exception:
raise LibraryNotInstalled('The xmltodict library is not installed.')
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.6'
version = '1.7'
description = 'XML file parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
@@ -82,7 +92,7 @@ class info():
__version__ = info.version
def _process(proc_data):
def _process(proc_data, has_data=False):
"""
Final processing to conform to the schema.
@@ -94,9 +104,13 @@ def _process(proc_data):
Dictionary representing an XML document.
"""
raw_output = []
# No further processing
return proc_data
if has_data:
# standard output with @ prefix for attributes
raw_output = xmltodict.parse(proc_data, dict_constructor=dict)
return raw_output
def parse(data, raw=False, quiet=False):
@@ -113,22 +127,20 @@ def parse(data, raw=False, quiet=False):
Dictionary. Raw or processed structured data.
"""
# check if xml library is installed and fail gracefully if it is not
try:
import xmltodict
except Exception:
raise LibraryNotInstalled('The xmltodict library is not installed.')
jc.utils.compatibility(__name__, info.compatible, quiet)
jc.utils.input_type_check(data)
raw_output = []
has_data = False
if jc.utils.has_data(data):
raw_output = xmltodict.parse(data)
has_data = True
if raw:
if has_data:
# modified output with _ prefix for attributes
raw_output = xmltodict.parse(data, dict_constructor=dict, attr_prefix='_')
return raw_output
else:
return _process(raw_output)
return _process(data, has_data)

View File

@@ -219,9 +219,9 @@ except ImportError:
_screen_pattern = (
r"Screen (?P<screen_number>\d+): "
+ "minimum (?P<minimum_width>\d+) x (?P<minimum_height>\d+), "
+ "current (?P<current_width>\d+) x (?P<current_height>\d+), "
+ "maximum (?P<maximum_width>\d+) x (?P<maximum_height>\d+)"
+ r"minimum (?P<minimum_width>\d+) x (?P<minimum_height>\d+), "
+ r"current (?P<current_width>\d+) x (?P<current_height>\d+), "
+ r"maximum (?P<maximum_width>\d+) x (?P<maximum_height>\d+)"
)
@@ -250,13 +250,13 @@ def _parse_screen(next_lines: List[str]) -> Optional[Screen]:
# regex101 demo link
_device_pattern = (
r"(?P<device_name>.+) "
+ "(?P<is_connected>(connected|disconnected)) ?"
+ "(?P<is_primary> primary)? ?"
+ "((?P<resolution_width>\d+)x(?P<resolution_height>\d+)"
+ "\+(?P<offset_width>\d+)\+(?P<offset_height>\d+))? "
+ "(?P<rotation>(inverted|left|right))? ?"
+ "\(normal left inverted right x axis y axis\)"
+ "( ((?P<dimension_width>\d+)mm x (?P<dimension_height>\d+)mm)?)?"
+ r"(?P<is_connected>(connected|disconnected)) ?"
+ r"(?P<is_primary> primary)? ?"
+ r"((?P<resolution_width>\d+)x(?P<resolution_height>\d+)"
+ r"\+(?P<offset_width>\d+)\+(?P<offset_height>\d+))? "
+ r"(?P<rotation>(inverted|left|right))? ?"
+ r"\(normal left inverted right x axis y axis\)"
+ r"( ((?P<dimension_width>\d+)mm x (?P<dimension_height>\d+)mm)?)?"
)

0
jc/py.typed Normal file
View File

View File

@@ -1,10 +1,14 @@
"""jc - JSON Convert streaming utils"""
from functools import wraps
from typing import Dict, Iterable
from typing import Dict, Tuple, Union, Iterable, Callable, TypeVar, cast, Any
from .jc_types import JSONDictType
def streaming_input_type_check(data: Iterable) -> None:
F = TypeVar('F', bound=Callable[..., Any])
def streaming_input_type_check(data: Iterable[Union[str, bytes]]) -> None:
"""
Ensure input data is an iterable, but not a string or bytes. Raises
`TypeError` if not.
@@ -19,7 +23,7 @@ def streaming_line_input_type_check(line: str) -> None:
raise TypeError("Input line must be a 'str' object.")
def stream_success(output_line: Dict, ignore_exceptions: bool) -> Dict:
def stream_success(output_line: JSONDictType, ignore_exceptions: bool) -> JSONDictType:
"""Add `_jc_meta` object to output line if `ignore_exceptions=True`"""
if ignore_exceptions:
output_line.update({'_jc_meta': {'success': True}})
@@ -27,7 +31,7 @@ def stream_success(output_line: Dict, ignore_exceptions: bool) -> Dict:
return output_line
def stream_error(e: BaseException, line: str) -> Dict:
def stream_error(e: BaseException, line: str) -> JSONDictType:
"""
Return an error `_jc_meta` field.
"""
@@ -41,7 +45,7 @@ def stream_error(e: BaseException, line: str) -> Dict:
}
def add_jc_meta(func):
def add_jc_meta(func: F) -> F:
"""
Decorator for streaming parsers to add stream_success and stream_error
objects. This simplifies the yield lines in the streaming parsers.
@@ -96,14 +100,14 @@ def add_jc_meta(func):
line = value[1]
yield stream_error(exception_obj, line)
return wrapper
return cast(F, wrapper)
def raise_or_yield(
ignore_exceptions: bool,
e: BaseException,
line: str
) -> tuple:
) -> Tuple[BaseException, str]:
"""
Return the exception object and line string if ignore_exceptions is
True. Otherwise, re-raise the exception from the exception object with

View File

@@ -1,3 +1,5 @@
# type: ignore
"""More comprehensive traceback formatting for Python scripts.
To enable this module, do:
import tracebackplus; tracebackplus.enable()
@@ -69,7 +71,6 @@ products or services of Licensee, or any third party.
agrees to be bound by the terms and conditions of this License
Agreement.
'''
import inspect
import keyword
import linecache

View File

@@ -6,7 +6,8 @@ import shutil
from datetime import datetime, timezone
from textwrap import TextWrapper
from functools import lru_cache
from typing import List, Iterable, Union, Optional
from typing import Any, List, Dict, Iterable, Union, Optional, TextIO
from .jc_types import TimeStampFormatType
def _asciify(string: str) -> str:
@@ -21,7 +22,13 @@ def _asciify(string: str) -> str:
return string
def _safe_print(string: str, sep=' ', end='\n', file=sys.stdout, flush=False) -> None:
def _safe_print(
string: str,
sep: str = ' ',
end: str = '\n',
file: TextIO = sys.stdout,
flush: bool = False
) -> None:
"""Output for both UTF-8 and ASCII encoding systems"""
try:
print(string, sep=sep, end=end, file=file, flush=flush)
@@ -106,7 +113,7 @@ def error_message(message_lines: List[str]) -> None:
_safe_print(message, file=sys.stderr)
def is_compatible(compatible: List) -> bool:
def is_compatible(compatible: List[str]) -> bool:
"""
Returns True if the parser is compatible with the running OS platform.
"""
@@ -120,7 +127,7 @@ def is_compatible(compatible: List) -> bool:
return platform_found
def compatibility(mod_name: str, compatible: List, quiet: bool = False) -> None:
def compatibility(mod_name: str, compatible: List[str], quiet: bool = False) -> None:
"""
Checks for the parser's compatibility with the running OS platform and
prints a warning message to `STDERR` if not compatible and
@@ -134,7 +141,7 @@ def compatibility(mod_name: str, compatible: List, quiet: bool = False) -> None:
the parser. compatible options:
linux, darwin, cygwin, win32, aix, freebsd
quiet: (bool) supress compatibility message if True
quiet: (bool) suppress compatibility message if True
Returns:
@@ -172,7 +179,7 @@ def has_data(data: Union[str, bytes]) -> bool:
return bool(data)
def convert_to_int(value: Union[str, float]) -> Optional[int]:
def convert_to_int(value: object) -> Optional[int]:
"""
Converts string and float input to int. Strips all non-numeric
characters from strings.
@@ -202,7 +209,7 @@ def convert_to_int(value: Union[str, float]) -> Optional[int]:
return None
def convert_to_float(value: Union[str, int]) -> Optional[float]:
def convert_to_float(value: object) -> Optional[float]:
"""
Converts string and int input to float. Strips all non-numeric
characters from strings.
@@ -228,7 +235,7 @@ def convert_to_float(value: Union[str, int]) -> Optional[float]:
return None
def convert_to_bool(value: Union[str, int, float]) -> bool:
def convert_to_bool(value: object) -> bool:
"""
Converts string, integer, or float input to boolean by checking
for 'truthy' values.
@@ -266,16 +273,18 @@ def convert_to_bool(value: Union[str, int, float]) -> bool:
return False
def input_type_check(data: str) -> None:
def input_type_check(data: object) -> None:
"""Ensure input data is a string. Raises `TypeError` if not."""
if not isinstance(data, str):
raise TypeError("Input data must be a 'str' object.")
class timestamp:
__slots__ = ('string', 'format', 'naive', 'utc', 'iso')
def __init__(self,
datetime_string: str,
format_hint: Optional[Iterable] = None
datetime_string: Optional[str],
format_hint: Optional[Iterable[int]] = None
) -> None:
"""
Input a datetime text string of several formats and convert to a
@@ -305,6 +314,9 @@ class timestamp:
utc (int | None): aware timestamp only if UTC timezone
detected in datetime string. None if conversion fails.
iso (str | None): ISO string - timezone information is output
only if UTC timezone is detected in the datetime string.
"""
self.string = datetime_string
@@ -317,13 +329,17 @@ class timestamp:
self.format = dt['format']
self.naive = dt['timestamp_naive']
self.utc = dt['timestamp_utc']
self.iso = dt['iso']
def __repr__(self):
return f'timestamp(string={self.string!r}, format={self.format}, naive={self.naive}, utc={self.utc})'
def __repr__(self) -> str:
return f'timestamp(string={self.string!r}, format={self.format}, naive={self.naive}, utc={self.utc}, iso={self.iso!r})'
@staticmethod
@lru_cache(maxsize=512)
def _parse_dt(dt_string, format_hint=None):
@lru_cache(maxsize=2048)
def _parse_dt(
dt_string: Optional[str],
format_hint: Optional[Iterable[int]] = None
) -> Dict[str, Any]:
"""
Input a datetime text string of several formats and convert to
a naive or timezone-aware epoch timestamp in UTC.
@@ -354,6 +370,9 @@ class timestamp:
# aware timestamp only if UTC timezone detected.
# None if conversion fails.
"timestamp_utc": int
# ISO string. None if conversion fails.
"iso": str
}
The `format` integer denotes which date_time format
@@ -368,9 +387,12 @@ class timestamp:
timezone is not found in the date-time string), then this
field will be None.
The `iso` string will only have timezone information if the
UTC timezone is detected in `dt_string`.
If the conversion completely fails, all fields will be None.
"""
formats = (
formats: tuple[TimeStampFormatType, ...] = (
{'id': 1000, 'format': '%a %b %d %H:%M:%S %Y', 'locale': None}, # manual C locale format conversion: Tue Mar 23 16:12:11 2021 or Tue Mar 23 16:12:11 IST 2021
{'id': 1100, 'format': '%a %b %d %H:%M:%S %Y %z', 'locale': None}, # git date output: Thu Mar 5 09:17:40 2020 -0800
{'id': 1300, 'format': '%Y-%m-%dT%H:%M:%S.%f%Z', 'locale': None}, # ISO Format with UTC (found in syslog 5424): 2003-10-11T22:14:15.003Z
@@ -384,6 +406,9 @@ class timestamp:
{'id': 1700, 'format': '%m/%d/%Y, %I:%M:%S %p', 'locale': None}, # Windows english format wint non-UTC tz (found in systeminfo cli output): 3/22/2021, 1:15:51 PM (UTC-0600)
{'id': 1705, 'format': '%m/%d/%Y, %I:%M:%S %p %Z', 'locale': None}, # Windows english format with UTC tz (found in systeminfo cli output): 3/22/2021, 1:15:51 PM (UTC)
{'id': 1710, 'format': '%m/%d/%Y, %I:%M:%S %p UTC%z', 'locale': None}, # Windows english format with UTC tz (found in systeminfo cli output): 3/22/2021, 1:15:51 PM (UTC+0000)
{'id': 1750, 'format': '%Y/%m/%d-%H:%M:%S.%f', 'locale': None}, # Google Big Table format with no timezone: 1970/01/01-01:00:00.000000
{'id': 1755, 'format': '%Y/%m/%d-%H:%M:%S.%f%z', 'locale': None}, # Google Big Table format with timezone: 1970/01/01-01:00:00.000000+00:00
{'id': 1800, 'format': '%d/%b/%Y:%H:%M:%S %z', 'locale': None}, # Common Log Format: 10/Oct/2000:13:55:36 -0700
{'id': 2000, 'format': '%a %d %b %Y %I:%M:%S %p %Z', 'locale': None}, # en_US.UTF-8 local format (found in upower cli output): Tue 23 Mar 2021 04:12:11 PM UTC
{'id': 3000, 'format': '%a %d %b %Y %I:%M:%S %p', 'locale': None}, # en_US.UTF-8 local format with non-UTC tz (found in upower cli output): Tue 23 Mar 2021 04:12:11 PM IST
{'id': 4000, 'format': '%A %d %B %Y %I:%M:%S %p %Z', 'locale': None}, # European-style local format (found in upower cli output): Tuesday 01 October 2019 12:50:41 PM UTC
@@ -405,7 +430,7 @@ class timestamp:
# from https://www.timeanddate.com/time/zones/
# only removed UTC timezone and added known non-UTC offsets
tz_abbr = {
tz_abbr: set[str] = {
'A', 'ACDT', 'ACST', 'ACT', 'ACWST', 'ADT', 'AEDT', 'AEST', 'AET', 'AFT', 'AKDT',
'AKST', 'ALMT', 'AMST', 'AMT', 'ANAST', 'ANAT', 'AQTT', 'ART', 'AST', 'AT', 'AWDT',
'AWST', 'AZOST', 'AZOT', 'AZST', 'AZT', 'AoE', 'B', 'BNT', 'BOT', 'BRST', 'BRT', 'BST',
@@ -433,7 +458,7 @@ class timestamp:
'UTC+1345', 'UTC+1400'
}
offset_suffixes = (
offset_suffixes: tuple[str, ...] = (
'-12:00', '-11:00', '-10:00', '-09:30', '-09:00',
'-08:00', '-07:00', '-06:00', '-05:00', '-04:00', '-03:00', '-02:30',
'-02:00', '-01:00', '+01:00', '+02:00', '+03:00', '+04:00', '+04:30',
@@ -442,19 +467,20 @@ class timestamp:
'+13:45', '+14:00'
)
data = dt_string or ''
normalized_datetime = ''
utc_tz = False
dt = None
dt_utc = None
timestamp_naive = None
timestamp_utc = None
timestamp_obj = {
data: str = dt_string or ''
normalized_datetime: str = ''
utc_tz: bool = False
dt: Optional[datetime] = None
dt_utc: Optional[datetime] = None
timestamp_naive: Optional[int] = None
timestamp_utc: Optional[int] = None
iso_string: Optional[str] = None
timestamp_obj: Dict[str, Any] = {
'format': None,
'timestamp_naive': None,
'timestamp_utc': None
'timestamp_utc': None,
'iso': None
}
utc_tz = False
# convert format_hint to a tuple so it is hashable (for lru_cache)
if not format_hint:
@@ -473,12 +499,15 @@ class timestamp:
if 'UTC+' in data or 'UTC-' in data:
utc_tz = bool('UTC+0000' in data or 'UTC-0000' in data)
elif '+0000' in data or '-0000' in data:
elif '+0000' in data \
or '-0000' in data \
or '+00:00' in data \
or '-00:00' in data:
utc_tz = True
# normalize the timezone by taking out any timezone reference, except UTC
cleandata = data.replace('(', '').replace(')', '')
normalized_datetime_list = []
normalized_datetime_list: List[str] = []
for term in cleandata.split():
if term not in tz_abbr:
normalized_datetime_list.append(term)
@@ -496,7 +525,7 @@ class timestamp:
normalized_datetime = p.sub(r'\g<1> ', normalized_datetime)
# try format hints first, then fall back to brute-force method
hint_obj_list = []
hint_obj_list: List[TimeStampFormatType] = []
for fmt_id in format_hint:
for fmt in formats:
if fmt_id == fmt['id']:
@@ -509,8 +538,9 @@ class timestamp:
try:
locale.setlocale(locale.LC_TIME, fmt['locale'])
dt = datetime.strptime(normalized_datetime, fmt['format'])
timestamp_naive = int(dt.replace(tzinfo=None).timestamp())
timestamp_obj['format'] = fmt['id']
timestamp_naive = int(dt.replace(tzinfo=None).timestamp())
iso_string = dt.replace(tzinfo=None).isoformat()
locale.setlocale(locale.LC_TIME, None)
break
except Exception:
@@ -520,9 +550,11 @@ class timestamp:
if dt and utc_tz:
dt_utc = dt.replace(tzinfo=timezone.utc)
timestamp_utc = int(dt_utc.timestamp())
iso_string = dt_utc.isoformat()
if timestamp_naive:
timestamp_obj['timestamp_naive'] = timestamp_naive
timestamp_obj['timestamp_utc'] = timestamp_utc
timestamp_obj['iso'] = iso_string
return timestamp_obj

View File

@@ -1,4 +1,4 @@
.TH jc 1 2022-09-26 1.22.0 "JSON Convert"
.TH jc 1 2022-12-16 1.22.3 "JSON Convert"
.SH NAME
\fBjc\fP \- JSON Convert JSONifies the output of many CLI tools, file-types, and strings
.SH SYNOPSIS
@@ -65,6 +65,11 @@ multi-line ASCII and Unicode table parser
\fB--blkid\fP
`blkid` command parser
.TP
.B
\fB--cbt\fP
`cbt` (Google Bigtable) command parser
.TP
.B
\fB--cef\fP
@@ -85,6 +90,16 @@ CEF string streaming parser
\fB--cksum\fP
`cksum` and `sum` command parser
.TP
.B
\fB--clf\fP
Common and Combined Log Format file parser
.TP
.B
\fB--clf-s\fP
Common and Combined Log Format file streaming parser
.TP
.B
\fB--crontab\fP
@@ -110,6 +125,11 @@ CSV file streaming parser
\fB--date\fP
`date` command parser
.TP
.B
\fB--datetime-iso\fP
ISO 8601 Datetime string parser
.TP
.B
\fB--df\fP
@@ -155,6 +175,11 @@ Email Address string parser
\fB--file\fP
`file` command parser
.TP
.B
\fB--findmnt\fP
`findmnt` command parser
.TP
.B
\fB--finger\fP
@@ -180,6 +205,11 @@ Email Address string parser
\fB--git-log-s\fP
`git log` command streaming parser
.TP
.B
\fB--git-ls-remote\fP
`git ls-remote` command parser
.TP
.B
\fB--gpg\fP
@@ -258,7 +288,7 @@ IPv4 and IPv6 Address string parser
.TP
.B
\fB--iso-datetime\fP
ISO 8601 Datetime string parser
Deprecated - please use datetime-iso
.TP
.B
@@ -315,6 +345,11 @@ Key/Value file parser
\fB--lsof\fP
`lsof` command parser
.TP
.B
\fB--lspci\fP
`lspci -mmv` command parser
.TP
.B
\fB--lsusb\fP
@@ -360,11 +395,31 @@ M3U and M3U8 file parser
\fB--ntpq\fP
`ntpq -p` command parser
.TP
.B
\fB--openvpn\fP
openvpn-status.log file parser
.TP
.B
\fB--os-prober\fP
`os-prober` command parser
.TP
.B
\fB--passwd\fP
`/etc/passwd` file parser
.TP
.B
\fB--pci-ids\fP
`pci.ids` file parser
.TP
.B
\fB--pgpass\fP
PostgreSQL password file parser
.TP
.B
\fB--pidstat\fP
@@ -680,6 +735,11 @@ PLIST file parser
\fB--rsync-s\fP
`rsync` command streaming parser
.TP
.B
\fB--semver\fP
Semantic Version string parser
.TP
.B
\fB--sfdisk\fP
@@ -695,6 +755,11 @@ PLIST file parser
\fB--ss\fP
`ss` command parser
.TP
.B
\fB--sshd-conf\fP
sshd config file and `sshd -T` command parser
.TP
.B
\fB--stat\fP
@@ -790,6 +855,11 @@ Unix Epoch Timestamp string parser
\fB--traceroute\fP
`traceroute` and `traceroute6` command parser
.TP
.B
\fB--udevadm\fP
`udevadm info` command parser
.TP
.B
\fB--ufw\fP
@@ -1027,7 +1097,7 @@ JC_COLORS=default,default,default,default
You can set the \fBNO_COLOR\fP environment variable to any value to disable color output in \fBjc\fP. Note that using the \fB-C\fP option to force color output will override both the \fBNO_COLOR\fP environment variable and the \fB-m\fP option.
.SH STREAMING PARSERS
Most parsers load all of the data from \fBSTDIN\fP, parse it, then output the entire JSON document serially. There are some streaming parsers (e.g. \fBls-s\fP, \fBping-s\fP, etc.) that immediately start processing and outputing the data line-by-line as JSON Lines (aka NDJSON) while it is being received from \fBSTDIN\fP. This can significantly reduce the amount of memory required to parse large amounts of command output (e.g. \fBls -lR /\fP) and can sometimes process the data more quickly. Streaming parsers have slightly different behavior than standard parsers as outlined below.
Most parsers load all of the data from \fBSTDIN\fP, parse it, then output the entire JSON document serially. There are some streaming parsers (e.g. \fBls-s\fP, \fBping-s\fP, etc.) that immediately start processing and outputting the data line-by-line as JSON Lines (aka NDJSON) while it is being received from \fBSTDIN\fP. This can significantly reduce the amount of memory required to parse large amounts of command output (e.g. \fBls -lR /\fP) and can sometimes process the data more quickly. Streaming parsers have slightly different behavior than standard parsers as outlined below.
.RS
Note: Streaming parsers cannot be used with the "magic" syntax

Some files were not shown because too many files have changed in this diff Show More