1
0
mirror of https://github.com/kellyjonbrazil/jc.git synced 2026-04-03 17:44:07 +02:00

Compare commits

...

190 Commits

Author SHA1 Message Date
Kelly Brazil
9b33a50bb0 Merge pull request #260 from kellyjonbrazil/dev
Dev v1.20.2
2022-07-05 19:39:40 -07:00
Kelly Brazil
b0e9d9fa9d fix encodings for windows tests 2022-07-05 19:23:22 -07:00
Kelly Brazil
991b612b62 add colon seperators to hex strings in python 3.6 and 3.7 2022-07-05 19:19:06 -07:00
Kelly Brazil
855c9363a5 add hexlify compatibility for python 3.6 and 3.7 2022-07-05 19:12:44 -07:00
Kelly Brazil
625544f53e add x509-cert tests 2022-07-05 18:59:55 -07:00
Kelly Brazil
f179da8cd1 sort set to list conversion 2022-07-05 18:59:47 -07:00
Kelly Brazil
eeee776555 add shell completion build scripts 2022-07-05 16:41:23 -07:00
Kelly Brazil
d0bde14a2a version bump 2022-07-05 16:15:35 -07:00
Kelly Brazil
69b0e407ad move pygments colors to cli_data.py 2022-07-05 16:15:26 -07:00
Kelly Brazil
ad2b9ab76c doc update 2022-07-05 15:07:10 -07:00
Kelly Brazil
32483de4f2 doc update 2022-07-05 14:59:21 -07:00
Kelly Brazil
ec713cfa1b doc update 2022-07-05 13:33:57 -07:00
Kelly Brazil
eac8c1c1a0 only attempt to decode certificate type 2022-07-05 12:55:38 -07:00
Kelly Brazil
4e15ab7924 fix _fix_objects: catch all sets and OrderedDicts 2022-07-05 11:33:56 -07:00
Kelly Brazil
5882755e7c convert larg integers to ascii-hex. simplify _fix_objects. add iso date strings. allow multiple certs in a PEM bundle 2022-07-05 11:29:24 -07:00
Kelly Brazil
13388df603 initial working x509 cert parser 2022-07-01 15:48:29 -07:00
Kelly Brazil
430385fa63 doc update 2022-06-30 09:25:01 -07:00
Kelly Brazil
7bfebdcfeb remove unnecessary list initialization 2022-06-30 09:24:05 -07:00
Kelly Brazil
e472c3b140 add gpg tests 2022-06-29 18:56:15 -07:00
Kelly Brazil
43ac3d951c doc update 2022-06-29 18:36:06 -07:00
Kelly Brazil
207fa232c3 add examples 2022-06-29 18:34:20 -07:00
Kelly Brazil
667dc1976a add support for special field types 2022-06-29 18:26:38 -07:00
Kelly Brazil
7437750de5 initial working gpg parser 2022-06-29 11:01:25 -07:00
Kelly Brazil
aab47c9e80 Merge pull request #258 from kellyjonbrazil/master
sync to dev
2022-06-28 16:07:05 -07:00
Kelly Brazil
f1ed803525 formatting 2022-06-16 18:03:51 -07:00
Kelly Brazil
093c1703d7 formatting 2022-06-16 18:00:40 -07:00
Kelly Brazil
1f2e16aeea formatting 2022-06-16 17:54:40 -07:00
Kelly Brazil
6f89745a58 formatting 2022-06-16 17:51:59 -07:00
Kelly Brazil
9b710b0b6f formatting 2022-06-16 17:51:20 -07:00
Kelly Brazil
992120b861 spelling 2022-06-16 17:48:07 -07:00
Kelly Brazil
229bef5f82 update anchor link 2022-06-16 07:52:58 -07:00
Kelly Brazil
c8a4890fc7 fix anchor link 2022-06-16 07:50:37 -07:00
Kelly Brazil
10a9848ac8 Merge branch 'master' of https://github.com/kellyjonbrazil/jc 2022-06-16 07:48:46 -07:00
Kelly Brazil
95189e37ba add streaming parsers link 2022-06-16 07:48:44 -07:00
Kelly Brazil
fe1f09f08d use github markdown renderer 2022-06-15 20:15:24 -07:00
Kelly Brazil
6adfa15742 formatting 2022-06-15 20:10:55 -07:00
Kelly Brazil
edef264f17 formatting 2022-06-15 20:04:39 -07:00
Kelly Brazil
9911b3fc9d formatting 2022-06-15 16:11:22 -07:00
Kelly Brazil
1675aa7a59 add shell completion wiki link 2022-06-15 16:08:31 -07:00
Kelly Brazil
7fd67fda13 Merge pull request #256 from kellyjonbrazil/dev
v1.20.1
2022-06-15 22:16:07 +00:00
Kelly Brazil
7583f315ce add long options tests 2022-06-15 13:21:37 -07:00
Kelly Brazil
7fd70d2088 doc update 2022-06-15 13:10:12 -07:00
Kelly Brazil
337ee73844 doc update 2022-06-15 12:51:42 -07:00
Kelly Brazil
43702a260b strip single quotes just like double quotes 2022-06-15 12:50:58 -07:00
Kelly Brazil
40208a9f76 doc update 2022-06-15 11:25:21 -07:00
Kelly Brazil
b958358389 update _snake_case comment 2022-06-15 11:15:26 -07:00
Kelly Brazil
c23aacedad doc update 2022-06-15 11:12:49 -07:00
Kelly Brazil
4f148469d7 preserve keyname case with -r 2022-06-15 11:12:43 -07:00
Kelly Brazil
247c43278c add postconf tests 2022-06-15 09:22:09 -07:00
Kelly Brazil
45f45e0511 add examples 2022-06-15 09:18:11 -07:00
Kelly Brazil
c0c469ae9b add comments 2022-06-14 17:03:54 -07:00
Kelly Brazil
4b86fd8d8a manipulate words and CURRENT to fix magic completions in Zsh 2022-06-14 16:51:45 -07:00
Kelly Brazil
8fba47f449 doc update 2022-06-14 12:42:30 -07:00
Kelly Brazil
f9ae964280 reuse arrays in zsh completions 2022-06-14 08:49:26 -07:00
Kelly Brazil
ca95615d7f partially working _normal completion for zsh 2022-06-14 07:42:55 -07:00
Kelly Brazil
747b255e34 zsh working same as bash except for magic command completions 2022-06-13 21:09:37 -07:00
Kelly Brazil
c0da9ebd6c only check previous words in bash completion 2022-06-10 17:25:19 -07:00
Kelly Brazil
5c40b38a05 fix help bash completions so they don't repeat parsers 2022-06-10 12:02:04 -07:00
Kelly Brazil
7bc03dcf06 add local vars to bash completion 2022-06-08 08:51:23 -07:00
Kelly Brazil
f62e6168fd fix help completions for bash 2022-06-08 08:50:20 -07:00
Kelly Brazil
39c1470ea6 add special options to bash completion 2022-06-07 11:57:48 -07:00
Kelly Brazil
e48b99f1c1 fix bash completion behavior 2022-06-07 11:02:05 -07:00
Kelly Brazil
71ae545907 better working bash completion 2022-06-06 16:24:52 -07:00
Kelly Brazil
b9a5eda187 formatting 2022-06-06 10:52:40 -07:00
Kelly Brazil
231a2039c2 update options 2022-06-06 10:49:17 -07:00
Kelly Brazil
a415bc23fa man page update 2022-06-06 10:32:18 -07:00
Kelly Brazil
79add35fc1 split long and short options 2022-06-05 18:09:44 -07:00
Kelly Brazil
901763fc39 fix hyphens 2022-06-05 18:06:38 -07:00
Kelly Brazil
1034cb1ea2 Merge branch 'dev' of https://github.com/kellyjonbrazil/jc into dev 2022-06-05 18:04:42 -07:00
Kelly Brazil
020093fd67 revert non-breaking hyphens 2022-06-05 18:04:35 -07:00
Kelly Brazil
f17b3fbd32 add non-breaking hyphens 2022-06-05 17:54:34 -07:00
Kelly Brazil
15b58e3a6b add long-options 2022-06-05 17:52:14 -07:00
Kelly Brazil
da6c98826b doc update 2022-06-05 17:40:05 -07:00
Kelly Brazil
26415e2978 formatting 2022-06-05 17:17:13 -07:00
Kelly Brazil
d849bd3b66 add bash and zsh completions to cli 2022-06-05 16:23:55 -07:00
Kelly Brazil
5996192455 remove indent at end of help text 2022-06-05 12:16:30 -07:00
Kelly Brazil
a38b6b5522 add long options 2022-06-05 12:12:58 -07:00
Kelly Brazil
1d6bc40bff fix no_wake_up_before_first_use to allow None 2022-06-03 10:35:49 -07:00
Kelly Brazil
437fa62cb1 update schema 2022-06-03 10:29:55 -07:00
Kelly Brazil
6cde26d9ed add process logic 2022-06-02 14:53:31 -07:00
Kelly Brazil
f6dd5a68cd doc update 2022-06-02 09:28:00 -07:00
Kelly Brazil
b1507dc576 initial postconf parser 2022-06-02 09:14:02 -07:00
Kelly Brazil
a5d5b1554f change variable name from list to my_list 2022-06-02 08:34:51 -07:00
Kelly Brazil
ec8efebc94 move _get_item outside of parse function 2022-06-02 08:31:28 -07:00
Kelly Brazil
f86cbf5527 Merge pull request #250 from pbrezina/id
id: parse output without name
2022-06-02 15:27:07 +00:00
Pavel Březina
57142d899c id: parse output without name
If a group does not exist only gid is present in the command output.
2022-06-02 12:51:11 +02:00
Kelly Brazil
504a04279e version bump 2022-06-01 12:13:50 -07:00
Kelly Brazil
3b4ef4a814 formatting 2022-06-01 12:13:41 -07:00
Kelly Brazil
a7e3d2fc86 Merge pull request #251 from kellyjonbrazil/master
sync to dev
2022-06-01 17:35:52 +00:00
Kelly Brazil
776ee66bae Merge pull request #249 from kellyjonbrazil/dev
Dev v1.20.0
2022-05-31 16:23:43 +00:00
Kelly Brazil
9bf2cd0691 doc update 2022-05-31 09:16:54 -07:00
Kelly Brazil
fec74cf305 doc formatting 2022-05-28 13:13:07 -07:00
Kelly Brazil
d3d7fbca61 doc formatting 2022-05-28 12:41:36 -07:00
Kelly Brazil
550862a415 doc update 2022-05-28 12:18:02 -07:00
Kelly Brazil
1763b530da add tests for new fields 2022-05-27 17:40:54 -07:00
Kelly Brazil
8307150cae add integer fields 2022-05-27 17:26:48 -07:00
Kelly Brazil
d2223c45d1 add new fields to schema 2022-05-27 17:20:54 -07:00
Kelly Brazil
472ee5e295 add new fields. allow for new unknown fields with warning 2022-05-27 17:07:39 -07:00
Kelly Brazil
1d0aebd836 add new fields 2022-05-27 16:54:49 -07:00
Kelly Brazil
8240626043 add idle status for newer top version support 2022-05-27 16:27:51 -07:00
Kelly Brazil
2cd14235c8 remove unnecessary pass line 2022-05-27 14:29:48 -07:00
Kelly Brazil
f774513554 add top-s tests 2022-05-27 14:29:33 -07:00
Kelly Brazil
388da9f003 remove try/except at end 2022-05-27 14:11:27 -07:00
Kelly Brazil
fa18243491 add check for top data 2022-05-27 13:48:13 -07:00
Kelly Brazil
5df1c1702b add top-s parser 2022-05-27 13:34:07 -07:00
Kelly Brazil
b6557802f4 change windows command prompt in docs 2022-05-27 08:54:37 -07:00
Kelly Brazil
15e3a511b6 doc update 2022-05-26 16:04:57 -07:00
Kelly Brazil
08fbde0e8f fix stdin stder quoting 2022-05-26 16:03:30 -07:00
Kelly Brazil
28ebb4e8dd fix notes 2022-05-26 15:54:39 -07:00
Kelly Brazil
04fda57cbe fix note 2022-05-26 15:11:58 -07:00
Kelly Brazil
b24495136c fix blockquotes 2022-05-26 15:07:54 -07:00
Kelly Brazil
7d3fa55571 doc update 2022-05-26 11:52:29 -07:00
Kelly Brazil
a76b5db8db doc update 2022-05-26 11:37:16 -07:00
Kelly Brazil
667dd01ac7 fix process table detection 2022-05-26 09:29:59 -07:00
Kelly Brazil
559bee962a add top tests 2022-05-26 09:19:47 -07:00
Kelly Brazil
9f977d06e0 fix asciitable parser for cases where centered rows cause misaligned fields 2022-05-26 08:57:35 -07:00
Kelly Brazil
f67a916940 add new parser counts to jc -a 2022-05-25 14:30:00 -07:00
Kelly Brazil
8a1308948f fix typo 2022-05-25 12:21:44 -07:00
Kelly Brazil
4e130d11a3 fix schema 2022-05-25 12:18:51 -07:00
Kelly Brazil
51aa5b268f fix for mem info in different units 2022-05-25 10:46:58 -07:00
Kelly Brazil
6869133f53 working top parser and doc update 2022-05-25 10:31:41 -07:00
Kelly Brazil
ad3d88e47f doc update 2022-05-25 08:14:14 -07:00
Kelly Brazil
6d29f8ba74 quiet option for warning messages 2022-05-25 07:37:21 -07:00
Kelly Brazil
c7cb89e91d more processing 2022-05-24 17:22:23 -07:00
Kelly Brazil
27f5118abd add top parser 2022-05-24 15:47:51 -07:00
Kelly Brazil
ac5a6e516d doc update 2022-05-23 15:12:14 -07:00
Kelly Brazil
5d5bfbc1d1 doc update 2022-05-23 13:58:24 -07:00
Kelly Brazil
9f4ba80000 refactor - create is_compatible function 2022-05-23 13:52:08 -07:00
Kelly Brazil
c21c334b73 version bump 2022-05-23 13:43:57 -07:00
Kelly Brazil
14093d9d43 Merge pull request #247 from kellyjonbrazil/yaml-dev
Yaml dev
2022-05-23 20:36:44 +00:00
Kelly Brazil
176ca2f75d docstring update 2022-05-23 13:16:57 -07:00
Kelly Brazil
cfb71c7dad add test for strings ending with colon 2022-05-22 09:13:32 -07:00
Kelly Brazil
722eab83b1 add Other Shell section 2022-05-21 07:00:43 -07:00
Kelly Brazil
d30edb2dae add yaml_out tests 2022-05-20 18:15:56 -07:00
Kelly Brazil
cba461ce8a remove JSONDecodeError detection 2022-05-20 17:48:52 -07:00
Kelly Brazil
f876505e25 add colors and ensure ascii on error 2022-05-20 17:22:56 -07:00
Kelly Brazil
5b9fcd5852 initial working yaml out 2022-05-20 14:51:23 -07:00
Kelly Brazil
8a122cd9e1 fix wiki language 2022-05-20 07:58:19 -07:00
Kelly Brazil
557e68225c add filtering capabilities for other shells 2022-05-19 11:16:25 -07:00
Kelly Brazil
dc0947b87e add other shell info. clean up parser table 2022-05-19 10:52:18 -07:00
Kelly Brazil
af2c06cd28 doc update 2022-05-15 22:28:12 -07:00
Kelly Brazil
67a4c6f797 add initial \n to first line of multiline fields 2022-05-15 22:23:00 -07:00
Kelly Brazil
0d1c857410 fix to keep first line of multiline field 2022-05-15 22:19:01 -07:00
Kelly Brazil
c7684dc94d allow multiline fields in pip-show 2022-05-15 22:06:43 -07:00
Kelly Brazil
a247572f64 changelog update 2022-05-15 20:49:41 -07:00
Kelly Brazil
306111303a Merge pull request #244 from kellyjonbrazil/master
Merge pull request #242 from kellyjonbrazil/dev sync to dev
2022-05-16 03:46:59 +00:00
Kelly Brazil
c12b48537a Merge pull request #242 from kellyjonbrazil/dev
Dev v1.19.0
2022-05-13 20:31:06 +00:00
Kelly Brazil
0bf69713ab Merge branch 'master' into dev 2022-05-13 20:26:12 +00:00
Kelly Brazil
c05f1475ed doc update 2022-05-13 13:23:14 -07:00
Kelly Brazil
416c262f02 remove top parsers 2022-05-13 13:17:16 -07:00
Kelly Brazil
40a5976f11 doc update 2022-05-13 12:45:27 -07:00
Kelly Brazil
dc582fbec8 fix for git-log coner cases 2022-05-13 12:44:01 -07:00
Kelly Brazil
7047f0a449 df fix for trailining newline char 2022-05-12 16:06:44 -07:00
Kelly Brazil
d7a884a567 doc update 2022-05-12 11:28:20 -07:00
Kelly Brazil
28de57792c formatting 2022-05-12 11:16:44 -07:00
Kelly Brazil
a230cdbd21 fix chage docs 2022-05-12 10:47:14 -07:00
Kelly Brazil
251a261679 add chage example 2022-05-11 18:56:13 -07:00
Kelly Brazil
3baef254fd maxsplit line 2022-05-11 18:47:56 -07:00
Kelly Brazil
9e69e928b2 add chage tests 2022-05-11 16:18:39 -07:00
Kelly Brazil
6eb595d2ca add chage parser 2022-05-11 16:12:29 -07:00
Kelly Brazil
acf07a5144 allow install on python 3.6 systems even though it is not supported 2022-05-11 08:00:05 -07:00
Kelly Brazil
7f53c58057 add ignore exception tests 2022-05-10 16:23:12 -07:00
Kelly Brazil
a069dc4855 add top -b parser 2022-05-10 16:08:56 -07:00
Kelly Brazil
febf544202 add git-log-s tests 2022-05-10 16:08:45 -07:00
Kelly Brazil
f2dd7b8815 doc update 2022-05-09 16:16:29 -07:00
Kelly Brazil
d758e37fe3 doc update 2022-05-09 16:10:41 -07:00
Kelly Brazil
9bc02a5623 doc update 2022-05-09 16:09:40 -07:00
Kelly Brazil
eb0ec265d0 Merge pull request #240 from fiskhest/master
Update README.md
2022-05-06 14:30:02 +00:00
fiskhest
1f236dc02a Update README.md 2022-05-06 16:20:33 +02:00
Kelly Brazil
86c279cbb2 add sep char test 2022-05-02 11:15:41 -07:00
Kelly Brazil
4a7f1bed3a doc update 2022-05-02 09:16:12 -07:00
Kelly Brazil
a730ae18c8 changelog update 2022-05-01 11:07:27 -07:00
Kelly Brazil
930bf439c0 check for separator characters in table cells. If detected, print a warning. 2022-05-01 11:04:35 -07:00
Kelly Brazil
f5f3133b87 add note about table column separators not being allowed in cell data 2022-05-01 10:27:34 -07:00
Kelly Brazil
cbca96de84 changelog update 2022-05-01 09:56:23 -07:00
Kelly Brazil
4d8ae3f124 doc update 2022-05-01 09:55:08 -07:00
Kelly Brazil
653127431d update plugin language 2022-04-29 06:50:37 -07:00
Kelly Brazil
22acef3765 generator to iterable object 2022-04-29 06:47:10 -07:00
Kelly Brazil
0d5bf11f0d change iterator to iterable 2022-04-29 06:42:11 -07:00
Kelly Brazil
a824ccaff3 add git-log-s parser 2022-04-28 16:23:56 -07:00
Kelly Brazil
97ac965ba5 version bump 2022-04-28 16:23:14 -07:00
Kelly Brazil
3e7f284df5 simplify streaming docs 2022-04-28 13:38:24 -07:00
Kelly Brazil
5c749fe26f simplify docs 2022-04-28 13:30:11 -07:00
Kelly Brazil
aa48b46f48 try page character 2022-04-28 11:09:19 -07:00
Kelly Brazil
6eaa4ae176 try arrow link 2022-04-28 11:05:43 -07:00
Kelly Brazil
60c330b342 move gentoo up 2022-04-28 10:59:12 -07:00
Kelly Brazil
fdeb994121 try using a table for parser list 2022-04-28 10:49:35 -07:00
Kelly Brazil
65ed92fe7b move safe_print_json. change argument from string to list_or_dict 2022-04-28 09:28:56 -07:00
Kelly Brazil
9c8730786b Merge pull request #238 from kellyjonbrazil/master
sync to dev
2022-04-28 16:21:22 +00:00
Kelly Brazil
7608823ea7 add git-log 2022-04-27 11:31:35 -07:00
212 changed files with 25375 additions and 889 deletions

View File

@@ -1,31 +1,63 @@
jc changelog
20220705 v1.20.2
- Add `gpg --with-colons` parser tested on linux
- Add DER and PEM encoded X.509 Certificate parser
- Add Bash and Zsh completion scripts to DEB and RPM packages
20220615 v1.20.1
- Add `postconf -M` parser tested on linux
- Update `asciitable` and `asciitable-m` parsers to preserve case in key
names when using the `-r` or `raw=True` options.
- Add long options (e.g. `--help`, `--about`, `--pretty`, etc.)
- Add shell completions for Bash and Zsh
- Fix `id` parser for cases where the user or group name is not present
20220531 v1.20.0
- Add YAML output option with `-y`
- Add `top -b` standard and streaming parsers tested on linux
- Add `plugin_parser_count`, `standard_parser_count`, and `streaming_parser_count`
keys to `jc -a` output
- Add `is_compatible` function to the `utils` module
- Fix `pip-show` parser for packages with a multi-line license field
- Fix ASCII Table parser for cases where centered headers cause mis-aligned fields
20220513 v1.19.0
- Add `chage --list` command parser tested on linux
- Add `git log` command streaming parser
- Fix `git log` standard parser for corner-cases where hash values are in messages
- Fix `df` command parser for rare instances when a newline is found at the end
- Allow jc to pip install on unsupported python version 3.6
- Fix `asciitable-m` parser to skip some rows that contain column separator
characters in cell data. A warning message will be printed to STDERR
unless `-q` or `quiet=True` is used.
20220427 v1.18.8
- Fix update-alternatives --query parser for cases where `slaves` are not present
- Fix `update-alternatives --query` parser for cases where `slaves` are not present
- Fix UnicodeEncodeError on some systems where LANG=C is set and unicode
characters are in the output
- Update history parser: do not drop non-ASCII characters if the system
- Update `history` parser: do not drop non-ASCII characters if the system
is configured for UTF-8 encoding
- Enhance "magic syntax" to always use UTF-8 encoding
20220425 v1.18.7
- Add git log command parser
- Add update-alternatives --query parser
- Add update-alternatives --get-selections parser
- Add `git log` command parser
- Add `update-alternatives --query` parser
- Add `update-alternatives --get-selections` parser
- Fix key/value and ini parsers to allow duplicate keys
- Fix yaml file parser for files including timestamp objects
- Update xrandr parser: add a 'rotation' field
- Update `xrandr` parser: add a 'rotation' field
- Fix failing tests by moving template files
- Add python interpreter version and path to -v and -a output
20220325 v1.18.6
- Add pidstat command parser tested on linux
- Add pidstat command streaming parser tested on linux
- Add mpstat command parser tested on linux
- Add mpstat command streaming parser tested on linux
- Add `pidstat` command parser tested on linux
- Add `pidstat` command streaming parser tested on linux
- Add `mpstat` command parser tested on linux
- Add `mpstat` command streaming parser tested on linux
- Add single-line ASCII and Unicode table parser
- Add multi-line ASCII and Unicode table parser
- Add documentation option to parser_info() and all_parser_info()
- Add documentation option to `parser_info()` and `all_parser_info()`
20220305 v1.18.5
- Fix date parser to ensure AM/PM period string is always uppercase

View File

@@ -265,6 +265,21 @@ blkid -o udev -ip /dev/sda2 | jc --blkid -p # or: jc -p blkid -o udev
}
]
```
### chage --list
```bash
chage --list joeuser | jc --chage -p # or: jc -p chage --list joeuser
```
```json
{
"password_last_changed": "never",
"password_expires": "never",
"password_inactive": "never",
"account_expires": "never",
"min_days_between_password_change": 0,
"max_days_between_password_change": 99999,
"warning_days_before_password_expires": 7
}
```
### cksum
```bash
cksum * | jc --cksum -p # or: jc -p cksum *
@@ -1059,6 +1074,84 @@ cat /etc/fstab | jc --fstab -p
}
]
```
### git log
```bash
git log --stat | jc --git-log -p or: jc -p git log --stat
```
```json
[
{
"commit": "728d882ed007b3c8b785018874a0eb06e1143b66",
"author": "Kelly Brazil",
"author_email": "kellyjonbrazil@gmail.com",
"date": "Wed Apr 20 09:50:19 2022 -0400",
"stats": {
"files_changed": 2,
"insertions": 90,
"deletions": 12,
"files": [
"docs/parsers/git_log.md",
"jc/parsers/git_log.py"
]
},
"message": "add timestamp docs and examples",
"epoch": 1650462619,
"epoch_utc": null
},
{
"commit": "b53e42aca623181aa9bc72194e6eeef1e9a3a237",
"author": "Kelly Brazil",
"author_email": "kellyjonbrazil@gmail.com",
"date": "Wed Apr 20 09:44:42 2022 -0400",
"stats": {
"files_changed": 5,
"insertions": 29,
"deletions": 6,
"files": [
"docs/parsers/git_log.md",
"docs/utils.md",
"jc/parsers/git_log.py",
"jc/utils.py",
"man/jc.1"
]
},
"message": "add calculated timestamp",
"epoch": 1650462282,
"epoch_utc": null
}
]
```
### gpg --with-colons
```bash
gpg --with-colons --show-keys file.gpg | jc --gpg -p # or jc -p gpg --with-colons --show-keys file.gpg
```
```json
[
{
"type": "pub",
"validity": "f",
"key_length": "1024",
"pub_key_alg": "17",
"key_id": "6C7EE1B8621CC013",
"creation_date": "899817715",
"expiration_date": "1055898235",
"certsn_uidhash_trustinfo": null,
"owner_trust": "m",
"user_id": null,
"signature_class": null,
"key_capabilities": "scESC",
"cert_fingerprint_other": null,
"flag": null,
"token_sn": null,
"hash_alg": null,
"curve_name": null,
"compliance_flags": null,
"last_update_date": null,
"origin": null,
"comment": null
}
]
```
### /etc/group file
```bash
cat /etc/group | jc --group -p
@@ -2666,6 +2759,36 @@ pip show wrapt wheel | jc --pip-show -p # or: jc -p pip show wrapt whe
}
]
```
### postconf -M
```bash
postconf -M | jc --postconf -p # or jc -p postconf -M
```
```json
[
{
"service_name": "smtp",
"service_type": "inet",
"private": false,
"unprivileged": null,
"chroot": true,
"wake_up_time": null,
"process_limit": null,
"command": "smtpd",
"no_wake_up_before_first_use": null
},
{
"service_name": "pickup",
"service_type": "unix",
"private": false,
"unprivileged": null,
"chroot": true,
"wake_up_time": 60,
"process_limit": 1,
"command": "pickup",
"no_wake_up_before_first_use": false
}
]
```
### ps
```bash
ps -ef | jc --ps -p # or: jc -p ps -ef
@@ -3403,6 +3526,105 @@ timedatectl | jc --timedatectl -p # or: jc -p timedatectl
"epoch_utc": 1583888001
}
```
### tob -b
```bash
top -b -n 1 | jc --top -p # or jc -p tob -b -n 1
```
```json
[
{
"time": "11:20:43",
"uptime": 118,
"users": 2,
"load_1m": 0.0,
"load_5m": 0.01,
"load_15m": 0.05,
"tasks_total": 108,
"tasks_running": 2,
"tasks_sleeping": 106,
"tasks_stopped": 0,
"tasks_zombie": 0,
"cpu_user": 5.6,
"cpu_sys": 11.1,
"cpu_nice": 0.0,
"cpu_idle": 83.3,
"cpu_wait": 0.0,
"cpu_hardware": 0.0,
"cpu_software": 0.0,
"cpu_steal": 0.0,
"mem_total": 3.7,
"mem_free": 3.3,
"mem_used": 0.2,
"mem_buff_cache": 0.2,
"swap_total": 2.0,
"swap_free": 2.0,
"swap_used": 0.0,
"mem_available": 3.3,
"processes": [
{
"pid": 2225,
"user": "kbrazil",
"priority": 20,
"nice": 0,
"virtual_mem": 158.1,
"resident_mem": 2.2,
"shared_mem": 1.6,
"status": "running",
"percent_cpu": 12.5,
"percent_mem": 0.1,
"time_hundredths": "0:00.02",
"command": "top",
"parent_pid": 1884,
"uid": 1000,
"real_uid": 1000,
"real_user": "kbrazil",
"saved_uid": 1000,
"saved_user": "kbrazil",
"gid": 1000,
"group": "kbrazil",
"pgrp": 2225,
"tty": "pts/0",
"tty_process_gid": 2225,
"session_id": 1884,
"thread_count": 1,
"last_used_processor": 0,
"time": "0:00",
"swap": 0.0,
"code": 0.1,
"data": 1.0,
"major_page_fault_count": 0,
"minor_page_fault_count": 736,
"dirty_pages_count": 0,
"sleeping_in_function": null,
"flags": "..4.2...",
"cgroups": "1:name=systemd:/user.slice/user-1000.+",
"supplementary_gids": [
10,
1000
],
"supplementary_groups": [
"wheel",
"kbrazil"
],
"thread_gid": 2225,
"environment_variables": [
"XDG_SESSION_ID=2",
"HOSTNAME=localhost"
],
"major_page_fault_count_delta": 0,
"minor_page_fault_count_delta": 4,
"used": 2.2,
"ipc_namespace_inode": 4026531839,
"mount_namespace_inode": 4026531840,
"net_namespace_inode": 4026531956,
"pid_namespace_inode": 4026531836,
"user_namespace_inode": 4026531837,
"nts_namespace_inode": 4026531838
}
]
}
]
```
### tracepath
```bash
tracepath6 3ffe:2400:0:109::2 | jc --tracepath -p
@@ -3900,6 +4122,85 @@ who -a | jc --who -p # or: jc -p who -a
}
]
```
### X.509 PEM and DER certificate files
```bash
cat entrust.pem | jc --x509-cert -p
```
```json
[
{
"tbs_certificate": {
"version": "v3",
"serial_number": "a6:8b:79:29:00:00:00:00:50:d0:91:f9",
"signature": {
"algorithm": "sha384_ecdsa",
"parameters": null
},
"issuer": {
"country_name": "US",
"organization_name": "Entrust, Inc.",
"organizational_unit_name": [
"See www.entrust.net/legal-terms",
"(c) 2012 Entrust, Inc. - for authorized use only"
],
"common_name": "Entrust Root Certification Authority - EC1"
},
"validity": {
"not_before": 1355844336,
"not_after": 2144764536,
"not_before_iso": "2012-12-18T15:25:36+00:00",
"not_after_iso": "2037-12-18T15:55:36+00:00"
},
"subject": {
"country_name": "US",
"organization_name": "Entrust, Inc.",
"organizational_unit_name": [
"See www.entrust.net/legal-terms",
"(c) 2012 Entrust, Inc. - for authorized use only"
],
"common_name": "Entrust Root Certification Authority - EC1"
},
"subject_public_key_info": {
"algorithm": {
"algorithm": "ec",
"parameters": "secp384r1"
},
"public_key": "04:84:13:c9:d0:ba:6d:41:7b:e2:6c:d0:eb:55:5f:66:02:1a:24:f4:5b:89:69:47:e3:b8:c2:7d:f1:f2:02:c5:9f:a0:f6:5b:d5:8b:06:19:86:4f:53:10:6d:07:24:27:a1:a0:f8:d5:47:19:61:4c:7d:ca:93:27:ea:74:0c:ef:6f:96:09:fe:63:ec:70:5d:36:ad:67:77:ae:c9:9d:7c:55:44:3a:a2:63:51:1f:f5:e3:62:d4:a9:47:07:3e:cc:20"
},
"issuer_unique_id": null,
"subject_unique_id": null,
"extensions": [
{
"extn_id": "key_usage",
"critical": true,
"extn_value": [
"key_cert_sign",
"crl_sign"
]
},
{
"extn_id": "basic_constraints",
"critical": true,
"extn_value": {
"ca": true,
"path_len_constraint": null
}
},
{
"extn_id": "key_identifier",
"critical": false,
"extn_value": "b7:63:e7:1a:dd:8d:e9:08:a6:55:83:a4:e0:6a:50:41:65:11:42:49"
}
]
},
"signature_algorithm": {
"algorithm": "sha384_ecdsa",
"parameters": null
},
"signature_value": "30:64:02:30:61:79:d8:e5:42:47:df:1c:ae:53:99:17:b6:6f:1c:7d:e1:bf:11:94:d1:03:88:75:e4:8d:89:a4:8a:77:46:de:6d:61:ef:02:f5:fb:b5:df:cc:fe:4e:ff:fe:a9:e6:a7:02:30:5b:99:d7:85:37:06:b5:7b:08:fd:eb:27:8b:4a:94:f9:e1:fa:a7:8e:26:08:e8:7c:92:68:6d:73:d8:6f:26:ac:21:02:b8:99:b7:26:41:5b:25:60:ae:d0:48:1a:ee:06"
}
]
```
### XML files
```bash
cat cd_catalog.xml

290
README.md
View File

@@ -44,8 +44,10 @@ $ jc dig example.com | jq -r '.[].answer[].data'
93.184.216.34
```
The `jc` parsers can also be used as python modules. In this case the output
will be a python dictionary, or list of dictionaries, instead of JSON:
`jc` can also be used as a python library. In this case the output will be
a python dictionary, a list of dictionaries, or even a
[lazy iterable of dictionaries](#using-streaming-parsers-as-python-modules)
instead of JSON:
```python
>>> import subprocess
>>> import jc
@@ -111,13 +113,13 @@ pip3 install jc
| Debian/Ubuntu linux | `apt-get install jc` |
| Fedora linux | `dnf install jc` |
| openSUSE linux | `zypper install jc` |
| Archlinux User Repositories (AUR) | `paru -S jc` or `aura -A jc` or `yay -S jc` |
| Archlinux Community Repository | `paru -S jc` or `aura -S jc` or `yay -S jc` |
| NixOS linux | `nix-env -iA nixpkgs.jc` or `nix-env -iA nixos.jc` |
| Guix System linux | `guix install jc` |
| Gentoo Linux | `emerge dev-python/jc` |
| macOS | `brew install jc` |
| FreeBSD | `portsnap fetch update && cd /usr/ports/textproc/py-jc && make install clean` |
| Ansible filter plugin | `ansible-galaxy collection install community.general` |
| Gentoo Linux | `emerge dev-python/jc` |
> For more OS Packages, see https://repology.org/project/jc/versions.
@@ -144,121 +146,131 @@ option.
### Parsers
- `--acpi` enables the `acpi` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/acpi))
- `--airport` enables the `airport -I` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/airport))
- `--airport-s` enables the `airport -s` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/airport_s))
- `--arp` enables the `arp` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/arp))
- `--asciitable` enables the ASCII and Unicode table parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/asciitable))
- `--asciitable-m` enables the multi-line ASCII and Unicode table parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/asciitable_m))
- `--blkid` enables the `blkid` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/blkid))
- `--cksum` enables the `cksum` and `sum` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/cksum))
- `--crontab` enables the `crontab` command and file parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/crontab))
- `--crontab-u` enables the `crontab` file parser with user support ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/crontab_u))
- `--csv` enables the CSV file parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/csv))
- `--csv-s` enables the CSV file streaming parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/csv_s))
- `--date` enables the `date` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/date))
- `--df` enables the `df` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/df))
- `--dig` enables the `dig` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/dig))
- `--dir` enables the `dir` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/dir))
- `--dmidecode` enables the `dmidecode` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/dmidecode))
- `--dpkg-l` enables the `dpkg -l` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/dpkg_l))
- `--du` enables the `du` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/du))
- `--env` enables the `env` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/env))
- `--file` enables the `file` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/file))
- `--finger` enables the `finger` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/finger))
- `--free` enables the `free` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/free))
- `--fstab` enables the `/etc/fstab` file parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/fstab))
- `--git-log` enables the `git log` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/git_log))
- `--group` enables the `/etc/group` file parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/group))
- `--gshadow` enables the `/etc/gshadow` file parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/gshadow))
- `--hash` enables the `hash` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/hash))
- `--hashsum` enables the hashsum command parser (`md5sum`, `shasum`, etc.) ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/hashsum))
- `--hciconfig` enables the `hciconfig` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/hciconfig))
- `--history` enables the `history` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/history))
- `--hosts` enables the `/etc/hosts` file parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/hosts))
- `--id` enables the `id` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/id))
- `--ifconfig` enables the `ifconfig` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ifconfig))
- `--ini` enables the INI file parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ini))
- `--iostat` enables the `iostat` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/iostat))
- `--iostat-s` enables the `iostat` command streaming parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/iostat_s))
- `--iptables` enables the `iptables` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/iptables))
- `--iw-scan` enables the `iw dev [device] scan` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/iw_scan))
- `--jar-manifest` enables the MANIFEST.MF file parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/jar_manifest))
- `--jobs` enables the `jobs` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/jobs))
- `--kv` enables the Key/Value file parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/kv))
- `--last` enables the `last` and `lastb` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/last))
- `--ls` enables the `ls` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ls))
- `--ls-s` enables the `ls` command streaming parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ls_s))
- `--lsblk` enables the `lsblk` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/lsblk))
- `--lsmod` enables the `lsmod` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/lsmod))
- `--lsof` enables the `lsof` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/lsof))
- `--lsusb` enables the `lsusb` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/lsusb))
- `--mount` enables the `mount` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/mount))
- `--mpstat` enables the `mpstat` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/mpstat))
- `--mpstat-s` enables the `mpstat` command streaming parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/mpstat_s))
- `--netstat` enables the `netstat` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/netstat))
- `--nmcli` enables the `nmcli` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/nmcli))
- `--ntpq` enables the `ntpq -p` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ntpq))
- `--passwd` enables the `/etc/passwd` file parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/passwd))
- `--pidstat` enables the `pidstat` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/pidstat))
- `--pidstat-s` enables the `pidstat` command streaming parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/pidstat_s))
- `--ping` enables the `ping` and `ping6` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ping))
- `--ping-s` enables the `ping` and `ping6` command streaming parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ping_s))
- `--pip-list` enables the `pip list` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/pip_list))
- `--pip-show` enables the `pip show` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/pip_show))
- `--ps` enables the `ps` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ps))
- `--route` enables the `route` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/route))
- `--rpm-qi` enables the `rpm -qi` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/rpm_qi))
- `--rsync` enables the `rsync` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/rsync))
- `--rsync-s` enables the `rsync` command streaming parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/rsync_s))
- `--sfdisk` enables the `sfdisk` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/sfdisk))
- `--shadow` enables the `/etc/shadow` file parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/shadow))
- `--ss` enables the `ss` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ss))
- `--stat` enables the `stat` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/stat))
- `--stat-s` enables the `stat` command streaming parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/stat_s))
- `--sysctl` enables the `sysctl` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/sysctl))
- `--systemctl` enables the `systemctl` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/systemctl))
- `--systemctl-lj` enables the `systemctl list-jobs` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/systemctl_lj))
- `--systemctl-ls` enables the `systemctl list-sockets` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/systemctl_ls))
- `--systemctl-luf` enables the `systemctl list-unit-files` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/systemctl_luf))
- `--systeminfo` enables the `systeminfo` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/systeminfo))
- `--time` enables the `/usr/bin/time` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/time))
- `--timedatectl` enables the `timedatectl status` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/timedatectl))
- `--tracepath` enables the `tracepath` and `tracepath6` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/tracepath))
- `--traceroute` enables the `traceroute` and `traceroute6` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/traceroute))
- `--ufw` enables the `ufw status` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ufw))
- `--ufw-appinfo` enables the `ufw app info [application]` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ufw_appinfo))
- `--uname` enables the `uname -a` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/uname))
- `--update-alt-gs` enables the `update-alternatives --get-selections` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/update_alt_gs))
- `--update-alt-q` enables the `update-alternatives --query` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/update_alt_q))
- `--upower` enables the `upower` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/upower))
- `--uptime` enables the `uptime` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/uptime))
- `--vmstat` enables the `vmstat` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/vmstat))
- `--vmstat-s` enables the `vmstat` command streaming parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/vmstat_s))
- `--w` enables the `w` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/w))
- `--wc` enables the `wc` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/wc))
- `--who` enables the `who` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/who))
- `--xml` enables the XML file parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/xml))
- `--xrandr` enables the `xrandr` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/xrandr))
- `--yaml` enables the YAML file parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/yaml))
- `--zipinfo` enables the `zipinfo` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/zipinfo))
| Argument | Command or Filetype | Documentation |
|-------------------|---------------------------------------------------------|----------------------------------------------------------------------------|
| ` --acpi` | `acpi` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/acpi) |
| ` --airport` | `airport -I` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/airport) |
| ` --airport-s` | `airport -s` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/airport_s) |
| ` --arp` | `arp` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/arp) |
| ` --asciitable` | ASCII and Unicode table parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/asciitable) |
| ` --asciitable-m` | multi-line ASCII and Unicode table parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/asciitable_m) |
| ` --blkid` | `blkid` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/blkid) |
| ` --chage` | `chage --list` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/chage) |
| ` --cksum` | `cksum` and `sum` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/cksum) |
| ` --crontab` | `crontab` command and file parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/crontab) |
| ` --crontab-u` | `crontab` file parser with user support | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/crontab_u) |
| ` --csv` | CSV file parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/csv) |
| ` --csv-s` | CSV file streaming parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/csv_s) |
| ` --date` | `date` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/date) |
| ` --df` | `df` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/df) |
| ` --dig` | `dig` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/dig) |
| ` --dir` | `dir` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/dir) |
| ` --dmidecode` | `dmidecode` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/dmidecode) |
| ` --dpkg-l` | `dpkg -l` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/dpkg_l) |
| ` --du` | `du` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/du) |
| ` --env` | `env` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/env) |
| ` --file` | `file` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/file) |
| ` --finger` | `finger` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/finger) |
| ` --free` | `free` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/free) |
| ` --fstab` | `/etc/fstab` file parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/fstab) |
| ` --git-log` | `git log` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/git_log) |
| ` --git-log-s` | `git log` command streaming parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/git_log_s) |
| ` --gpg` | `gpg --with-colons` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/gpg) |
| ` --group` | `/etc/group` file parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/group) |
| ` --gshadow` | `/etc/gshadow` file parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/gshadow) |
| ` --hash` | `hash` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/hash) |
| ` --hashsum` | hashsum command parser (`md5sum`, `shasum`, etc.) | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/hashsum) |
| ` --hciconfig` | `hciconfig` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/hciconfig) |
| ` --history` | `history` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/history) |
| ` --hosts` | `/etc/hosts` file parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/hosts) |
| ` --id` | `id` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/id) |
| ` --ifconfig` | `ifconfig` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/ifconfig) |
| ` --ini` | INI file parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/ini) |
| ` --iostat` | `iostat` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/iostat) |
| ` --iostat-s` | `iostat` command streaming parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/iostat_s) |
| ` --iptables` | `iptables` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/iptables) |
| ` --iw-scan` | `iw dev [device] scan` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/iw_scan) |
| ` --jar-manifest` | MANIFEST.MF file parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/jar_manifest) |
| ` --jobs` | `jobs` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/jobs) |
| ` --kv` | Key/Value file parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/kv) |
| ` --last` | `last` and `lastb` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/last) |
| ` --ls` | `ls` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/ls) |
| ` --ls-s` | `ls` command streaming parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/ls_s) |
| ` --lsblk` | `lsblk` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/lsblk) |
| ` --lsmod` | `lsmod` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/lsmod) |
| ` --lsof` | `lsof` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/lsof) |
| ` --lsusb` | `lsusb` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/lsusb) |
| ` --mount` | `mount` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/mount) |
| ` --mpstat` | `mpstat` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/mpstat) |
| ` --mpstat-s` | `mpstat` command streaming parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/mpstat_s) |
| ` --netstat` | `netstat` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/netstat) |
| ` --nmcli` | `nmcli` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/nmcli) |
| ` --ntpq` | `ntpq -p` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/ntpq) |
| ` --passwd` | `/etc/passwd` file parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/passwd) |
| ` --pidstat` | `pidstat -h` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/pidstat) |
| ` --pidstat-s` | `pidstat -h` command streaming parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/pidstat_s) |
| ` --ping` | `ping` and `ping6` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/ping) |
| ` --ping-s` | `ping` and `ping6` command streaming parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/ping_s) |
| ` --pip-list` | `pip list` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/pip_list) |
| ` --pip-show` | `pip show` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/pip_show) |
| ` --postconf` | `postconf -M` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/postconf) |
| ` --ps` | `ps` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/ps) |
| ` --route` | `route` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/route) |
| ` --rpm-qi` | `rpm -qi` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/rpm_qi) |
| ` --rsync` | `rsync` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/rsync) |
| ` --rsync-s` | `rsync` command streaming parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/rsync_s) |
| ` --sfdisk` | `sfdisk` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/sfdisk) |
| ` --shadow` | `/etc/shadow` file parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/shadow) |
| ` --ss` | `ss` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/ss) |
| ` --stat` | `stat` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/stat) |
| ` --stat-s` | `stat` command streaming parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/stat_s) |
| ` --sysctl` | `sysctl` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/sysctl) |
| ` --systemctl` | `systemctl` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/systemctl) |
| ` --systemctl-lj` | `systemctl list-jobs` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/systemctl_lj) |
| ` --systemctl-ls` | `systemctl list-sockets` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/systemctl_ls) |
| `--systemctl-luf` | `systemctl list-unit-files` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/systemctl_luf) |
| ` --systeminfo` | `systeminfo` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/systeminfo) |
| ` --time` | `/usr/bin/time` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/time) |
| ` --timedatectl` | `timedatectl status` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/timedatectl) |
| ` --top` | `top -b` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/top) |
| ` --top-s` | `top -b` command streaming parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/top_s) |
| ` --tracepath` | `tracepath` and `tracepath6` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/tracepath) |
| ` --traceroute` | `traceroute` and `traceroute6` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/traceroute) |
| ` --ufw` | `ufw status` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/ufw) |
| ` --ufw-appinfo` | `ufw app info [application]` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/ufw_appinfo) |
| ` --uname` | `uname -a` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/uname) |
| `--update-alt-gs` | `update-alternatives --get-selections` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/update_alt_gs) |
| ` --update-alt-q` | `update-alternatives --query` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/update_alt_q) |
| ` --upower` | `upower` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/upower) |
| ` --uptime` | `uptime` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/uptime) |
| ` --vmstat` | `vmstat` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/vmstat) |
| ` --vmstat-s` | `vmstat` command streaming parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/vmstat_s) |
| ` --w` | `w` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/w) |
| ` --wc` | `wc` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/wc) |
| ` --who` | `who` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/who) |
| ` --x509-cert` | X.509 PEM and DER certificate file parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/x509_cert) |
| ` --xml` | XML file parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/xml) |
| ` --xrandr` | `xrandr` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/xrandr) |
| ` --yaml` | YAML file parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/yaml) |
| ` --zipinfo` | `zipinfo` command parser | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/zipinfo) |
### Options
- `-a` about `jc`. Prints information about `jc` and the parsers (in JSON, of
course!)
- `-C` force color output even when using pipes (overrides `-m` and the
`NO_COLOR` env variable)
- `-d` debug mode. Prints trace messages if parsing issues are encountered (use
`-dd` for verbose debugging)
- `-h` help. Use `jc -h --parser_name` for parser documentation
- `-m` monochrome JSON output
- `-p` pretty format the JSON output
- `-q` quiet mode. Suppresses parser warning messages (use `-qq` to ignore
streaming parser errors)
- `-r` raw output. Provides a more literal JSON output, typically with string
values and no additional semantic processing
- `-u` unbuffer output
- `-v` version information
| Short | Long | Description |
|-------|-----------------|---------------------------------------------------------------------------------------------------------------------|
| `-a` | `--about` | About `jc`. Prints information about `jc` and the parsers (in JSON or YAML, of course!) |
| `-C` | `--force-color` | Force color output even when using pipes (overrides `-m` and the `NO_COLOR` env variable) |
| `-d` | `--debug` | Debug mode. Prints trace messages if parsing issues are encountered (use`-dd` for verbose debugging) |
| `-h` | `--help` | Help. Use `jc -h --parser_name` for parser documentation |
| `-m` | `--monochrome` | Monochrome output |
| `-p` | `--pretty` | Pretty format the JSON output |
| `-q` | `--quiet` | Quiet mode. Suppresses parser warning messages (use `-qq` to ignore streaming parser errors) |
| `-r` | `--raw` | Raw output. Provides more literal output, typically with string values and no additional semantic processing |
| `-u` | `--unbuffer` | Unbuffer output |
| `-v` | `--version` | Version information |
| `-y` | `--yaml-out` | YAML output |
| `-B` | `--bash-comp` | Generate Bash shell completion script ([more info](https://github.com/kellyjonbrazil/jc/wiki/Shell-Completions)) |
| `-Z` | `--zsh-comp` | Generate Zsh shell completion script ([more info](https://github.com/kellyjonbrazil/jc/wiki/Shell-Completions)) |
### Exit Codes
Any fatal errors within `jc` will generate an exit code of `100`, otherwise the
@@ -305,11 +317,11 @@ color output will override both the `NO_COLOR` environment variable and the `-m`
option.
### Streaming Parsers
Most parsers load all of the data from STDIN, parse it, then output the entire
Most parsers load all of the data from `STDIN`, parse it, then output the entire
JSON document serially. There are some streaming parsers (e.g. `ls-s` and
`ping-s`) that immediately start processing and outputing the data line-by-line
as [JSON Lines](https://jsonlines.org/) (aka [NDJSON](http://ndjson.org/)) while
it is being received from STDIN. This can significantly reduce the amount of
it is being received from `STDIN`. This can significantly reduce the amount of
memory required to parse large amounts of command output (e.g. `ls -lR /`) and
can sometimes process the data more quickly. Streaming parsers have slightly
different behavior than standard parsers as outlined below.
@@ -374,14 +386,14 @@ $ ping 1.1.1.1 | jc --ping-s -u | jq
#### Using Streaming Parsers as Python Modules
Streaming parsers accept any iterable object and return an iterator object
(generator) allowing lazy processing of the data. The input data should
iterate on lines of string data. Examples of good input data are `sys.stdin` or
Streaming parsers accept any iterable object and return an iterable object
allowing lazy processing of the data. The input data should iterate on lines
of string data. Examples of good input data are `sys.stdin` or
`str.splitlines()`.
To use the generator object in your code, simply loop through it or use the
[next()](https://docs.python.org/3/library/functions.html#next) builtin
function:
To use the returned iterable object in your code, simply loop through it or
use the [next()](https://docs.python.org/3/library/functions.html#next)
builtin function:
```python
import jc
@@ -404,8 +416,8 @@ or [`jc/parsers/foo_s.py (streaming)`](https://github.com/kellyjonbrazil/jc/blob
parser as a template and simply place a `.py` file in the `jcparsers` subfolder.
Local plugin filenames must be valid python module names and therefore must
start with a letter and consist entirely of alphanumerics. Local plugins
may override default parsers.
start with a letter and consist entirely of alphanumerics and underscores.
Local plugins may override default parsers.
> Note: The application data directory follows the
[XDG Base Directory Specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html)
@@ -438,15 +450,30 @@ If a UTC timezone can be detected in the text of the command output, the
timestamp will be timezone aware and have a `_utc` suffix on the key name.
(e.g. `epoch_utc`) No other timezones are supported for aware timestamps.
## Use In Other Shells
`jc` can be used in most any shell. Some modern shells have JSON deserialization
and filtering capabilities built-in which makes using `jc` even more convenient.
For example, the following is possible in [NGS](https://ngs-lang.org/)
(Next Generation Shell):
```bash
myvar = ``jc dig www.google.com``[0].answer[0].data
```
This runs `jc`, parses the output JSON, and assigs the resulting data structure
to a variable in a single line of code.
For more examples of how to use `jc` in other shells, see this
[wiki page](https://github.com/kellyjonbrazil/jc/wiki/Using-jc-With-Different-Shells).
## Compatibility
Some parsers like `dig`, `xml`, `csv`, etc. will work on any platform. Other
parsers that convert platform-specific output will generate a warning message if
they are run on an unsupported platform. To see all parser information,
including compatibility, run `jc -ap`.
You may still use a parser on an unsupported platform - for example, you may want
to parse a file with linux `lsof` output on an macOS or Windows laptop. In that
case you can suppress the warning message with the `-q` cli option or the
You may still use a parser on an unsupported platform - for example, you may
want to parse a file with linux `lsof` output on an macOS or Windows laptop. In
that case you can suppress the warning message with the `-q` cli option or the
`quiet=True` function parameter in `parse()`:
macOS:
@@ -475,7 +502,8 @@ Tested on:
## Contributions
Feel free to add/improve code or parsers! You can use the
[`jc/parsers/foo.py`](https://github.com/kellyjonbrazil/jc/blob/master/jc/parsers/foo.py)
or [`jc/parsers/foo_s.py (streaming)`](https://github.com/kellyjonbrazil/jc/blob/master/jc/parsers/foo_s.py) parsers as a template and submit your parser with a pull request.
or [`jc/parsers/foo_s.py (streaming)`](https://github.com/kellyjonbrazil/jc/blob/master/jc/parsers/foo_s.py) parsers as a template and submit your parser with a pull
request.
Please see the [Contributing Guidelines](https://github.com/kellyjonbrazil/jc/blob/master/CONTRIBUTING.md) for more information.

View File

@@ -1 +1,2 @@
theme: jekyll-theme-cayman
theme: jekyll-theme-cayman
markdown: GFM

9
build-completions.py Executable file
View File

@@ -0,0 +1,9 @@
#!/usr/bin/env python3
# build Bash and Zsh completion scripts and add to the completions folder
from jc.shell_completions import bash_completion, zsh_completion
with open('completions/jc_bash_completion.sh', 'w') as f:
print(bash_completion(), file=f)
with open('completions/jc_zsh_completion.sh', 'w') as f:
print(zsh_completion(), file=f)

View File

@@ -0,0 +1,84 @@
_jc()
{
local cur prev words cword jc_commands jc_parsers jc_options \
jc_about_options jc_about_mod_options jc_help_options jc_special_options
jc_commands=(acpi airport arp blkid chage cksum crontab date df dig dmidecode dpkg du env file finger free git gpg hciconfig id ifconfig iostat iptables iw jobs last lastb ls lsblk lsmod lsof lsusb md5 md5sum mount mpstat netstat nmcli ntpq pidstat ping ping6 pip pip3 postconf printenv ps route rpm rsync sfdisk sha1sum sha224sum sha256sum sha384sum sha512sum shasum ss stat sum sysctl systemctl systeminfo timedatectl top tracepath tracepath6 traceroute traceroute6 ufw uname update-alternatives upower uptime vdir vmstat w wc who xrandr zipinfo)
jc_parsers=(--acpi --airport --airport-s --arp --asciitable --asciitable-m --blkid --chage --cksum --crontab --crontab-u --csv --csv-s --date --df --dig --dir --dmidecode --dpkg-l --du --env --file --finger --free --fstab --git-log --git-log-s --gpg --group --gshadow --hash --hashsum --hciconfig --history --hosts --id --ifconfig --ini --iostat --iostat-s --iptables --iw-scan --jar-manifest --jobs --kv --last --ls --ls-s --lsblk --lsmod --lsof --lsusb --mount --mpstat --mpstat-s --netstat --nmcli --ntpq --passwd --pidstat --pidstat-s --ping --ping-s --pip-list --pip-show --postconf --ps --route --rpm-qi --rsync --rsync-s --sfdisk --shadow --ss --stat --stat-s --sysctl --systemctl --systemctl-lj --systemctl-ls --systemctl-luf --systeminfo --time --timedatectl --top --top-s --tracepath --traceroute --ufw --ufw-appinfo --uname --update-alt-gs --update-alt-q --upower --uptime --vmstat --vmstat-s --w --wc --who --x509-cert --xml --xrandr --yaml --zipinfo)
jc_options=(--force-color -C --debug -d --monochrome -m --pretty -p --quiet -q --raw -r --unbuffer -u --yaml-out -y)
jc_about_options=(--about -a)
jc_about_mod_options=(--pretty -p --yaml-out -y --monochrome -m --force-color -C)
jc_help_options=(--help -h)
jc_special_options=(--version -v --bash-comp -B --zsh-comp -Z)
COMPREPLY=()
_get_comp_words_by_ref cur prev words cword
# if jc_about_options are found anywhere in the line, then only complete from jc_about_mod_options
for i in "${words[@]::${#words[@]}-1}"; do
if [[ " ${jc_about_options[*]} " =~ " ${i} " ]]; then
COMPREPLY=( $( compgen -W "${jc_about_mod_options[*]}" \
-- "${cur}" ) )
return 0
fi
done
# if jc_help_options and a parser are found anywhere in the line, then no more completions
if
(
for i in "${words[@]::${#words[@]}-1}"; do
if [[ " ${jc_help_options[*]} " =~ " ${i} " ]]; then
return 0
fi
done
return 1
) && (
for i in "${words[@]::${#words[@]}-1}"; do
if [[ " ${jc_parsers[*]} " =~ " ${i} " ]]; then
return 0
fi
done
return 1
); then
return 0
fi
# if jc_help_options are found anywhere in the line, then only complete with parsers
for i in "${words[@]::${#words[@]}-1}"; do
if [[ " ${jc_help_options[*]} " =~ " ${i} " ]]; then
COMPREPLY=( $( compgen -W "${jc_parsers[*]}" \
-- "${cur}" ) )
return 0
fi
done
# if special options are found anywhere in the line, then no more completions
for i in "${words[@]::${#words[@]}-1}"; do
if [[ " ${jc_special_options[*]} " =~ " ${i} " ]]; then
return 0
fi
done
# if magic command is found anywhere in the line, use called command's autocompletion
for i in "${words[@]::${#words[@]}-1}"; do
if [[ " ${jc_commands[*]} " =~ " ${i} " ]]; then
_command
return 0
fi
done
# if a parser arg is found anywhere in the line, only show options and help options
for i in "${words[@]::${#words[@]}-1}"; do
if [[ " ${jc_parsers[*]} " =~ " ${i} " ]]; then
COMPREPLY=( $( compgen -W "${jc_options[*]} ${jc_help_options[*]}" \
-- "${cur}" ) )
return 0
fi
done
# default completion
COMPREPLY=( $( compgen -W "${jc_options[*]} ${jc_about_options[*]} ${jc_help_options[*]} ${jc_special_options[*]} ${jc_parsers[*]} ${jc_commands[*]}" \
-- "${cur}" ) )
} &&
complete -F _jc jc

View File

@@ -0,0 +1,325 @@
#compdef jc
_jc() {
local -a jc_commands jc_commands_describe \
jc_parsers jc_parsers_describe \
jc_options jc_options_describe \
jc_about_options jc_about_options_describe \
jc_about_mod_options jc_about_mod_options_describe \
jc_help_options jc_help_options_describe \
jc_special_options jc_special_options_describe
jc_commands=(acpi airport arp blkid chage cksum crontab date df dig dmidecode dpkg du env file finger free git gpg hciconfig id ifconfig iostat iptables iw jobs last lastb ls lsblk lsmod lsof lsusb md5 md5sum mount mpstat netstat nmcli ntpq pidstat ping ping6 pip pip3 postconf printenv ps route rpm rsync sfdisk sha1sum sha224sum sha256sum sha384sum sha512sum shasum ss stat sum sysctl systemctl systeminfo timedatectl top tracepath tracepath6 traceroute traceroute6 ufw uname update-alternatives upower uptime vdir vmstat w wc who xrandr zipinfo)
jc_commands_describe=(
'acpi:run "acpi" command with magic syntax.'
'airport:run "airport" command with magic syntax.'
'arp:run "arp" command with magic syntax.'
'blkid:run "blkid" command with magic syntax.'
'chage:run "chage" command with magic syntax.'
'cksum:run "cksum" command with magic syntax.'
'crontab:run "crontab" command with magic syntax.'
'date:run "date" command with magic syntax.'
'df:run "df" command with magic syntax.'
'dig:run "dig" command with magic syntax.'
'dmidecode:run "dmidecode" command with magic syntax.'
'dpkg:run "dpkg" command with magic syntax.'
'du:run "du" command with magic syntax.'
'env:run "env" command with magic syntax.'
'file:run "file" command with magic syntax.'
'finger:run "finger" command with magic syntax.'
'free:run "free" command with magic syntax.'
'git:run "git" command with magic syntax.'
'gpg:run "gpg" command with magic syntax.'
'hciconfig:run "hciconfig" command with magic syntax.'
'id:run "id" command with magic syntax.'
'ifconfig:run "ifconfig" command with magic syntax.'
'iostat:run "iostat" command with magic syntax.'
'iptables:run "iptables" command with magic syntax.'
'iw:run "iw" command with magic syntax.'
'jobs:run "jobs" command with magic syntax.'
'last:run "last" command with magic syntax.'
'lastb:run "lastb" command with magic syntax.'
'ls:run "ls" command with magic syntax.'
'lsblk:run "lsblk" command with magic syntax.'
'lsmod:run "lsmod" command with magic syntax.'
'lsof:run "lsof" command with magic syntax.'
'lsusb:run "lsusb" command with magic syntax.'
'md5:run "md5" command with magic syntax.'
'md5sum:run "md5sum" command with magic syntax.'
'mount:run "mount" command with magic syntax.'
'mpstat:run "mpstat" command with magic syntax.'
'netstat:run "netstat" command with magic syntax.'
'nmcli:run "nmcli" command with magic syntax.'
'ntpq:run "ntpq" command with magic syntax.'
'pidstat:run "pidstat" command with magic syntax.'
'ping:run "ping" command with magic syntax.'
'ping6:run "ping6" command with magic syntax.'
'pip:run "pip" command with magic syntax.'
'pip3:run "pip3" command with magic syntax.'
'postconf:run "postconf" command with magic syntax.'
'printenv:run "printenv" command with magic syntax.'
'ps:run "ps" command with magic syntax.'
'route:run "route" command with magic syntax.'
'rpm:run "rpm" command with magic syntax.'
'rsync:run "rsync" command with magic syntax.'
'sfdisk:run "sfdisk" command with magic syntax.'
'sha1sum:run "sha1sum" command with magic syntax.'
'sha224sum:run "sha224sum" command with magic syntax.'
'sha256sum:run "sha256sum" command with magic syntax.'
'sha384sum:run "sha384sum" command with magic syntax.'
'sha512sum:run "sha512sum" command with magic syntax.'
'shasum:run "shasum" command with magic syntax.'
'ss:run "ss" command with magic syntax.'
'stat:run "stat" command with magic syntax.'
'sum:run "sum" command with magic syntax.'
'sysctl:run "sysctl" command with magic syntax.'
'systemctl:run "systemctl" command with magic syntax.'
'systeminfo:run "systeminfo" command with magic syntax.'
'timedatectl:run "timedatectl" command with magic syntax.'
'top:run "top" command with magic syntax.'
'tracepath:run "tracepath" command with magic syntax.'
'tracepath6:run "tracepath6" command with magic syntax.'
'traceroute:run "traceroute" command with magic syntax.'
'traceroute6:run "traceroute6" command with magic syntax.'
'ufw:run "ufw" command with magic syntax.'
'uname:run "uname" command with magic syntax.'
'update-alternatives:run "update-alternatives" command with magic syntax.'
'upower:run "upower" command with magic syntax.'
'uptime:run "uptime" command with magic syntax.'
'vdir:run "vdir" command with magic syntax.'
'vmstat:run "vmstat" command with magic syntax.'
'w:run "w" command with magic syntax.'
'wc:run "wc" command with magic syntax.'
'who:run "who" command with magic syntax.'
'xrandr:run "xrandr" command with magic syntax.'
'zipinfo:run "zipinfo" command with magic syntax.'
)
jc_parsers=(--acpi --airport --airport-s --arp --asciitable --asciitable-m --blkid --chage --cksum --crontab --crontab-u --csv --csv-s --date --df --dig --dir --dmidecode --dpkg-l --du --env --file --finger --free --fstab --git-log --git-log-s --gpg --group --gshadow --hash --hashsum --hciconfig --history --hosts --id --ifconfig --ini --iostat --iostat-s --iptables --iw-scan --jar-manifest --jobs --kv --last --ls --ls-s --lsblk --lsmod --lsof --lsusb --mount --mpstat --mpstat-s --netstat --nmcli --ntpq --passwd --pidstat --pidstat-s --ping --ping-s --pip-list --pip-show --postconf --ps --route --rpm-qi --rsync --rsync-s --sfdisk --shadow --ss --stat --stat-s --sysctl --systemctl --systemctl-lj --systemctl-ls --systemctl-luf --systeminfo --time --timedatectl --top --top-s --tracepath --traceroute --ufw --ufw-appinfo --uname --update-alt-gs --update-alt-q --upower --uptime --vmstat --vmstat-s --w --wc --who --x509-cert --xml --xrandr --yaml --zipinfo)
jc_parsers_describe=(
'--acpi:`acpi` command parser'
'--airport:`airport -I` command parser'
'--airport-s:`airport -s` command parser'
'--arp:`arp` command parser'
'--asciitable:ASCII and Unicode table parser'
'--asciitable-m:multi-line ASCII and Unicode table parser'
'--blkid:`blkid` command parser'
'--chage:`chage --list` command parser'
'--cksum:`cksum` and `sum` command parser'
'--crontab:`crontab` command and file parser'
'--crontab-u:`crontab` file parser with user support'
'--csv:CSV file parser'
'--csv-s:CSV file streaming parser'
'--date:`date` command parser'
'--df:`df` command parser'
'--dig:`dig` command parser'
'--dir:`dir` command parser'
'--dmidecode:`dmidecode` command parser'
'--dpkg-l:`dpkg -l` command parser'
'--du:`du` command parser'
'--env:`env` command parser'
'--file:`file` command parser'
'--finger:`finger` command parser'
'--free:`free` command parser'
'--fstab:`/etc/fstab` file parser'
'--git-log:`git log` command parser'
'--git-log-s:`git log` command streaming parser'
'--gpg:`gpg --with-colons` command parser'
'--group:`/etc/group` file parser'
'--gshadow:`/etc/gshadow` file parser'
'--hash:`hash` command parser'
'--hashsum:hashsum command parser (`md5sum`, `shasum`, etc.)'
'--hciconfig:`hciconfig` command parser'
'--history:`history` command parser'
'--hosts:`/etc/hosts` file parser'
'--id:`id` command parser'
'--ifconfig:`ifconfig` command parser'
'--ini:INI file parser'
'--iostat:`iostat` command parser'
'--iostat-s:`iostat` command streaming parser'
'--iptables:`iptables` command parser'
'--iw-scan:`iw dev [device] scan` command parser'
'--jar-manifest:MANIFEST.MF file parser'
'--jobs:`jobs` command parser'
'--kv:Key/Value file parser'
'--last:`last` and `lastb` command parser'
'--ls:`ls` command parser'
'--ls-s:`ls` command streaming parser'
'--lsblk:`lsblk` command parser'
'--lsmod:`lsmod` command parser'
'--lsof:`lsof` command parser'
'--lsusb:`lsusb` command parser'
'--mount:`mount` command parser'
'--mpstat:`mpstat` command parser'
'--mpstat-s:`mpstat` command streaming parser'
'--netstat:`netstat` command parser'
'--nmcli:`nmcli` command parser'
'--ntpq:`ntpq -p` command parser'
'--passwd:`/etc/passwd` file parser'
'--pidstat:`pidstat -h` command parser'
'--pidstat-s:`pidstat -h` command streaming parser'
'--ping:`ping` and `ping6` command parser'
'--ping-s:`ping` and `ping6` command streaming parser'
'--pip-list:`pip list` command parser'
'--pip-show:`pip show` command parser'
'--postconf:`postconf -M` command parser'
'--ps:`ps` command parser'
'--route:`route` command parser'
'--rpm-qi:`rpm -qi` command parser'
'--rsync:`rsync` command parser'
'--rsync-s:`rsync` command streaming parser'
'--sfdisk:`sfdisk` command parser'
'--shadow:`/etc/shadow` file parser'
'--ss:`ss` command parser'
'--stat:`stat` command parser'
'--stat-s:`stat` command streaming parser'
'--sysctl:`sysctl` command parser'
'--systemctl:`systemctl` command parser'
'--systemctl-lj:`systemctl list-jobs` command parser'
'--systemctl-ls:`systemctl list-sockets` command parser'
'--systemctl-luf:`systemctl list-unit-files` command parser'
'--systeminfo:`systeminfo` command parser'
'--time:`/usr/bin/time` command parser'
'--timedatectl:`timedatectl status` command parser'
'--top:`top -b` command parser'
'--top-s:`top -b` command streaming parser'
'--tracepath:`tracepath` and `tracepath6` command parser'
'--traceroute:`traceroute` and `traceroute6` command parser'
'--ufw:`ufw status` command parser'
'--ufw-appinfo:`ufw app info [application]` command parser'
'--uname:`uname -a` command parser'
'--update-alt-gs:`update-alternatives --get-selections` command parser'
'--update-alt-q:`update-alternatives --query` command parser'
'--upower:`upower` command parser'
'--uptime:`uptime` command parser'
'--vmstat:`vmstat` command parser'
'--vmstat-s:`vmstat` command streaming parser'
'--w:`w` command parser'
'--wc:`wc` command parser'
'--who:`who` command parser'
'--x509-cert:X.509 PEM and DER certificate file parser'
'--xml:XML file parser'
'--xrandr:`xrandr` command parser'
'--yaml:YAML file parser'
'--zipinfo:`zipinfo` command parser'
)
jc_options=(--force-color -C --debug -d --monochrome -m --pretty -p --quiet -q --raw -r --unbuffer -u --yaml-out -y)
jc_options_describe=(
'--force-color:force color output even when using pipes (overrides -m)'
'-C:force color output even when using pipes (overrides -m)'
'--debug:debug (double for verbose debug)'
'-d:debug (double for verbose debug)'
'--monochrome:monochrome output'
'-m:monochrome output'
'--pretty:pretty print output'
'-p:pretty print output'
'--quiet:suppress warnings (double to ignore streaming errors)'
'-q:suppress warnings (double to ignore streaming errors)'
'--raw:raw output'
'-r:raw output'
'--unbuffer:unbuffer output'
'-u:unbuffer output'
'--yaml-out:YAML output'
'-y:YAML output'
)
jc_about_options=(--about -a)
jc_about_options_describe=(
'--about:about jc'
'-a:about jc'
)
jc_about_mod_options=(--pretty -p --yaml-out -y --monochrome -m --force-color -C)
jc_about_mod_options_describe=(
'--pretty:pretty print output'
'-p:pretty print output'
'--yaml-out:YAML output'
'-y:YAML output'
'--monochrome:monochrome output'
'-m:monochrome output'
'--force-color:force color output even when using pipes (overrides -m)'
'-C:force color output even when using pipes (overrides -m)'
)
jc_help_options=(--help -h)
jc_help_options_describe=(
'--help:help (--help --parser_name for parser documentation)'
'-h:help (--help --parser_name for parser documentation)'
)
jc_special_options=(--version -v --bash-comp -B --zsh-comp -Z)
jc_special_options_describe=(
'--version:version info'
'-v:version info'
'--bash-comp:gen Bash completion: jc -B > /etc/bash_completion.d/jc'
'-B:gen Bash completion: jc -B > /etc/bash_completion.d/jc'
'--zsh-comp:gen Zsh completion: jc -Z > "${fpath[1]}/_jc"'
'-Z:gen Zsh completion: jc -Z > "${fpath[1]}/_jc"'
)
# if jc_about_options are found anywhere in the line, then only complete from jc_about_mod_options
for i in ${words:0:-1}; do
if (( $jc_about_options[(Ie)${i}] )); then
_describe 'commands' jc_about_mod_options_describe
return 0
fi
done
# if jc_help_options and a parser are found anywhere in the line, then no more completions
if
(
for i in ${words:0:-1}; do
if (( $jc_help_options[(Ie)${i}] )); then
return 0
fi
done
return 1
) && (
for i in ${words:0:-1}; do
if (( $jc_parsers[(Ie)${i}] )); then
return 0
fi
done
return 1
); then
return 0
fi
# if jc_help_options are found anywhere in the line, then only complete with parsers
for i in ${words:0:-1}; do
if (( $jc_help_options[(Ie)${i}] )); then
_describe 'commands' jc_parsers_describe
return 0
fi
done
# if special options are found anywhere in the line, then no more completions
for i in ${words:0:-1}; do
if (( $jc_special_options[(Ie)${i}] )); then
return 0
fi
done
# if magic command is found anywhere in the line, use called command's autocompletion
for i in ${words:0:-1}; do
if (( $jc_commands[(Ie)${i}] )); then
# hack to remove options between jc and the magic command
shift $(( ${#words} - 2 )) words
words[1,0]=(jc)
CURRENT=${#words}
# run the magic command's completions
_arguments '*::arguments:_normal'
return 0
fi
done
# if a parser arg is found anywhere in the line, only show options and help options
for i in ${words:0:-1}; do
if (( $jc_parsers[(Ie)${i}] )); then
_describe 'commands' jc_options_describe -- jc_help_options_describe
return 0
fi
done
# default completion
_describe 'commands' jc_options_describe -- jc_about_options_describe -- jc_help_options_describe -- jc_special_options_describe -- jc_parsers_describe -- jc_commands_describe
}
_jc

View File

@@ -8,7 +8,12 @@ jc - JSON Convert `asciitable` parser
This parser converts ASCII and Unicode text tables with single-line rows.
Column headers must be at least two spaces apart from each other and must
be unique.
be unique. For best results, column headers should be left-justified. If
column separators are present, then non-left-justified headers will be fixed
automatically.
Row separators are optional and are ignored. Each non-row-separator line is
considered a separate row in the table.
For example:
@@ -54,6 +59,9 @@ etc...
Headers (keys) are converted to snake-case. All values are returned as
strings, except empty strings, which are converted to None/null.
> Note: To preserve the case of the keys use the `-r` cli option or
> `raw=True` argument in `parse()`.
Usage (cli):
$ cat table.txt | jc --asciitable
@@ -136,4 +144,4 @@ Returns:
### Parser Information
Compatibility: linux, darwin, cygwin, win32, aix, freebsd
Version 1.0 by Kelly Brazil (kellyjonbrazil@gmail.com)
Version 1.2 by Kelly Brazil (kellyjonbrazil@gmail.com)

View File

@@ -29,6 +29,15 @@ Headers (keys) are converted to snake-case and newlines between multi-line
headers are joined with an underscore. All values are returned as strings,
except empty strings, which are converted to None/null.
> Note: To preserve the case of the keys use the `-r` cli option or
> `raw=True` argument in `parse()`.
> Note: table column separator characters (e.g. `|`) cannot be present
> inside the cell data. If detected, a warning message will be printed to
> `STDERR` and the line will be skipped. The warning message can be
> suppressed by using the `-q` command option or by setting `quiet=True` in
> `parse()`.
Usage (cli):
$ cat table.txt | jc --asciitable-m
@@ -120,4 +129,4 @@ Returns:
### Parser Information
Compatibility: linux, darwin, cygwin, win32, aix, freebsd
Version 1.0 by Kelly Brazil (kellyjonbrazil@gmail.com)
Version 1.2 by Kelly Brazil (kellyjonbrazil@gmail.com)

82
docs/parsers/chage.md Normal file
View File

@@ -0,0 +1,82 @@
[Home](https://kellyjonbrazil.github.io/jc/)
<a id="jc.parsers.chage"></a>
# jc.parsers.chage
jc - JSON Convert `chage --list` command output parser
Supports `chage -l <username>` or `chage --list <username>`
Usage (cli):
$ chage -l johndoe | jc --chage
or
$ jc chage -l johndoe
Usage (module):
import jc
result = jc.parse('chage', chage_command_output)
Schema:
{
"password_last_changed": string,
"password_expires": string,
"password_inactive": string,
"account_expires": string,
"min_days_between_password_change": integer,
"max_days_between_password_change": integer,
"warning_days_before_password_expires": integer
}
Examples:
$ chage --list joeuser | jc --chage -p
{
"password_last_changed": "never",
"password_expires": "never",
"password_inactive": "never",
"account_expires": "never",
"min_days_between_password_change": 0,
"max_days_between_password_change": 99999,
"warning_days_before_password_expires": 7
}
$ chage --list joeuser | jc --chage -p -r
{
"password_last_changed": "never",
"password_expires": "never",
"password_inactive": "never",
"account_expires": "never",
"min_days_between_password_change": "0",
"max_days_between_password_change": "99999",
"warning_days_before_password_expires": "7"
}
<a id="jc.parsers.chage.parse"></a>
### parse
```python
def parse(data: str, raw: bool = False, quiet: bool = False) -> Dict
```
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
Returns:
Dictionary. Raw or processed structured data.
### Parser Information
Compatibility: linux
Version 1.0 by Kelly Brazil (kellyjonbrazil@gmail.com)

View File

@@ -5,15 +5,15 @@
jc - JSON Convert `csv` file streaming parser
> This streaming parser outputs JSON Lines (cli) or returns a Generator
iterator of Dictionaries (module)
> This streaming parser outputs JSON Lines (cli) or returns an Iterable of
> Dictionaries (module)
The `csv` streaming parser will attempt to automatically detect the
delimiter character. If the delimiter cannot be detected it will default
to comma. The first row of the file must be a header row.
Note: The first 100 rows are read into memory to enable delimiter detection,
then the rest of the rows are loaded lazily.
> Note: The first 100 rows are read into memory to enable delimiter
> detection, then the rest of the rows are loaded lazily.
Usage (cli):
@@ -68,7 +68,7 @@ Examples:
def parse(data, raw=False, quiet=False, ignore_exceptions=False)
```
Main text parsing generator function. Returns an iterator object.
Main text parsing generator function. Returns an iterable object.
Parameters:
@@ -79,13 +79,9 @@ Parameters:
quiet: (boolean) suppress warning messages if True
ignore_exceptions: (boolean) ignore parsing exceptions if True
Yields:
Dictionary. Raw or processed structured data.
Returns:
Iterator object (generator)
Iterable of Dictionaries
### Parser Information
Compatibility: linux, darwin, cygwin, win32, aix, freebsd

View File

@@ -120,4 +120,4 @@ Returns:
### Parser Information
Compatibility: linux, darwin, freebsd
Version 1.9 by Kelly Brazil (kellyjonbrazil@gmail.com)
Version 1.10 by Kelly Brazil (kellyjonbrazil@gmail.com)

View File

@@ -19,7 +19,7 @@ time of the system the parser is run on)
Usage (cli):
C:> dir | jc --dir
C:\> dir | jc --dir
Usage (module):
@@ -42,7 +42,7 @@ Schema:
Examples:
C:> dir | jc --dir -p
C:\> dir | jc --dir -p
[
{
"date": "03/24/2021",
@@ -83,7 +83,7 @@ Examples:
...
]
C:> dir | jc --dir -p -r
C:\> dir | jc --dir -p -r
[
{
"date": "03/24/2021",

View File

@@ -43,8 +43,8 @@ Schema:
"author": string,
"author_email": string,
"date": string,
"epoch": integer, [0]
"epoch_utc": integer, [1]
"epoch": integer, # [0]
"epoch_utc": integer, # [1]
"commit_by": string,
"commit_by_email": string,
"commit_by_date": string,
@@ -172,4 +172,4 @@ Returns:
### Parser Information
Compatibility: linux, darwin, cygwin, win32, aix, freebsd
Version 1.0 by Kelly Brazil (kellyjonbrazil@gmail.com)
Version 1.1 by Kelly Brazil (kellyjonbrazil@gmail.com)

111
docs/parsers/git_log_s.md Normal file
View File

@@ -0,0 +1,111 @@
[Home](https://kellyjonbrazil.github.io/jc/)
<a id="jc.parsers.git_log_s"></a>
# jc.parsers.git\_log\_s
jc - JSON Convert `git log` command output streaming parser
> This streaming parser outputs JSON Lines (cli) or returns an Iterable of
> Dictionaries (module)
Can be used with the following format options:
- `oneline`
- `short`
- `medium`
- `full`
- `fuller`
Additional options supported:
- `--stat`
- `--shortstat`
The `epoch` calculated timestamp field is naive. (i.e. based on the
local time of the system the parser is run on)
The `epoch_utc` calculated timestamp field is timezone-aware and is
only available if the timezone field is UTC.
Usage (cli):
$ git log | jc --git-log-s
Usage (module):
import jc
result = jc.parse('git_log_s', git_log_command_output.splitlines())
for item in result:
# do something
Schema:
{
"commit": string,
"author": string,
"author_email": string,
"date": string,
"epoch": integer, # [0]
"epoch_utc": integer, # [1]
"commit_by": string,
"commit_by_email": string,
"commit_by_date": string,
"message": string,
"stats" : {
"files_changed": integer,
"insertions": integer,
"deletions": integer,
"files": [
string
]
}
# below object only exists if using -qq or ignore_exceptions=True
"_jc_meta": {
"success": boolean, # false if error parsing
"error": string, # exists if "success" is false
"line": string # exists if "success" is false
}
}
[0] naive timestamp if "date" field is parsable, else null
[1] timezone aware timestamp availabe for UTC, else null
Examples:
$ git log | jc --git-log-s
{"commit":"a730ae18c8e81c5261db132df73cd74f272a0a26","author":"Kelly...}
{"commit":"930bf439c06c48a952baec05a9896c8d92b7693e","author":"Kelly...}
...
<a id="jc.parsers.git_log_s.parse"></a>
### parse
```python
@add_jc_meta
def parse(data: Iterable[str],
raw: bool = False,
quiet: bool = False,
ignore_exceptions: bool = False) -> Union[Iterable[Dict], tuple]
```
Main text parsing generator function. Returns an iterable object.
Parameters:
data: (iterable) line-based text data to parse
(e.g. sys.stdin or str.splitlines())
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
ignore_exceptions: (boolean) ignore parsing exceptions if True
Returns:
Iterable of Dictionaries
### Parser Information
Compatibility: linux, darwin, cygwin, win32, aix, freebsd
Version 1.1 by Kelly Brazil (kellyjonbrazil@gmail.com)

145
docs/parsers/gpg.md Normal file
View File

@@ -0,0 +1,145 @@
[Home](https://kellyjonbrazil.github.io/jc/)
<a id="jc.parsers.gpg"></a>
# jc.parsers.gpg
jc - JSON Convert `gpg --with-colons` command output parser
Usage (cli):
$ gpg --with-colons --show-keys file.gpg | jc --gpg
or
$ jc gpg --with-colons --show-keys file.gpg
Usage (module):
import jc
result = jc.parse('gpg', gpg_command_output)
Schema:
Field definitions from https://git.gnupg.org/cgi-bin/gitweb.cgi?p=gnupg.git;a=blob_plain;f=doc/DETAILS
> Note: Number values are not converted to integers because many field
> specifications are overloaded and future augmentations are implied in the
> documentation.
[
{
"type": string,
"validity": string,
"key_length": string,
"pub_key_alg": string,
"key_id": string,
"creation_date": string,
"expiration_date": string,
"certsn_uidhash_trustinfo": string,
"owner_trust": string,
"user_id": string,
"signature_class": string,
"key_capabilities": string,
"cert_fingerprint_other": string,
"flag": string,
"token_sn": string,
"hash_alg": string,
"curve_name": string,
"compliance_flags": string,
"last_update_date": string,
"origin": string,
"comment": string,
"index": string, # [0]
"bits": string, # [0]
"value": string, # [0]
"version": string, # [1], [4]
"signature_count": string, # [1]
"encryption_count": string, # [1]
"policy": string, # [1]
"signature_first_seen": string, # [1]
"signature_most_recent_seen": string, # [1]
"encryption_first_done": string, # [1]
"encryption_most_recent_done": string, # [1]
"staleness_reason": string, # [2]
"trust_model": string, # [2]
"trust_db_created": string, # [2]
"trust_db_expires": string, # [2]
"marginally_trusted_users": string, # [2]
"completely_trusted_users": string, # [2]
"cert_chain_max_depth": string, # [2]
"subpacket_number": string, # [3]
"hex_flags": string, # [3]
"subpacket_length": string, # [3]
"subpacket_data": string, # [3]
"pubkey": string, # [4]
"cipher": string, # [4]
"digest": string, # [4]
"compress": string, # [4]
"group": string, # [4]
"members": string, # [4]
"curve_names": string, # [4]
}
]
All blank values are converted to null/None.
[0] for 'pkd' type
[1] for 'tfs' type
[2] for 'tru' type
[3] for 'skp' type
[4] for 'cfg' type
Examples:
$ gpg --with-colons --show-keys file.gpg | jc --gpg -p
[
{
"type": "pub",
"validity": "f",
"key_length": "1024",
"pub_key_alg": "17",
"key_id": "6C7EE1B8621CC013",
"creation_date": "899817715",
"expiration_date": "1055898235",
"certsn_uidhash_trustinfo": null,
"owner_trust": "m",
"user_id": null,
"signature_class": null,
"key_capabilities": "scESC",
"cert_fingerprint_other": null,
"flag": null,
"token_sn": null,
"hash_alg": null,
"curve_name": null,
"compliance_flags": null,
"last_update_date": null,
"origin": null,
"comment": null
},
...
]
<a id="jc.parsers.gpg.parse"></a>
### parse
```python
def parse(data: str, raw: bool = False, quiet: bool = False) -> List[Dict]
```
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
Returns:
List of Dictionaries. Raw or processed structured data.
### Parser Information
Compatibility: linux
Version 1.0 by Kelly Brazil (kellyjonbrazil@gmail.com)

View File

@@ -128,4 +128,4 @@ Returns:
### Parser Information
Compatibility: linux, darwin, aix, freebsd
Version 1.4 by Kelly Brazil (kellyjonbrazil@gmail.com)
Version 1.5 by Kelly Brazil (kellyjonbrazil@gmail.com)

View File

@@ -5,7 +5,7 @@
jc - JSON Convert `ifconfig` command output parser
Note: No `ifconfig` options are supported.
> Note: No `ifconfig` options are supported.
Usage (cli):

View File

@@ -11,9 +11,10 @@ Parses standard `INI` files and files containing simple key/value pairs.
- Comment prefix can be `#` or `;`. Comments must be on their own line.
- If duplicate keys are found, only the last value will be used.
> Note: Values starting and ending with quotation marks will have the marks
removed. If you would like to keep the quotation marks, use the `-r`
command-line argument or the `raw=True` argument in `parse()`.
> Note: Values starting and ending with double or single quotation marks
> will have the marks removed. If you would like to keep the quotation
> marks, use the `-r` command-line argument or the `raw=True` argument in
> `parse()`.
Usage (cli):
@@ -26,8 +27,8 @@ Usage (module):
Schema:
ini or key/value document converted to a dictionary - see the
configparser standard library documentation for more details.
ini or key/value document converted to a dictionary - see the configparser
standard library documentation for more details.
{
"key1": string,
@@ -91,4 +92,4 @@ Returns:
### Parser Information
Compatibility: linux, darwin, cygwin, win32, aix, freebsd
Version 1.6 by Kelly Brazil (kellyjonbrazil@gmail.com)
Version 1.7 by Kelly Brazil (kellyjonbrazil@gmail.com)

View File

@@ -5,7 +5,7 @@
jc - JSON Convert `iostat` command output parser
Note: `iostat` version 11 and higher include a JSON output option
> Note: `iostat` version 11 and higher include a JSON output option
Usage (cli):

View File

@@ -5,21 +5,21 @@
jc - JSON Convert `iostat` command output streaming parser
> This streaming parser outputs JSON Lines (cli) or returns a Generator
iterator of Dictionaries (module)
> This streaming parser outputs JSON Lines (cli) or returns an Iterable of
> Dictionaries (module)
Note: `iostat` version 11 and higher include a JSON output option
> Note: `iostat` version 11 and higher include a JSON output option
Usage (cli):
$ iostat | jc --iostat-s
> Note: When piping `jc` converted `iostat` output to other processes it may
appear the output is hanging due to the OS pipe buffers. This is because
`iostat` output is too small to quickly fill up the buffer. Use the `-u`
option to unbuffer the `jc` output if you would like immediate output. See
the [readme](https://github.com/kellyjonbrazil/jc/tree/master#unbuffering-output)
for more information.
> appear the output is hanging due to the OS pipe buffers. This is because
> `iostat` output is too small to quickly fill up the buffer. Use the `-u`
> option to unbuffer the `jc` output if you would like immediate output. See
> the [readme](https://github.com/kellyjonbrazil/jc/tree/master#unbuffering-output)
> for more information.
Usage (module):
@@ -112,7 +112,7 @@ Examples:
def parse(data, raw=False, quiet=False, ignore_exceptions=False)
```
Main text parsing generator function. Returns an iterator object.
Main text parsing generator function. Returns an iterable object.
Parameters:
@@ -123,13 +123,9 @@ Parameters:
quiet: (boolean) suppress warning messages if True
ignore_exceptions: (boolean) ignore parsing exceptions if True
Yields:
Dictionary. Raw or processed structured data.
Returns:
Iterator object (generator)
Iterable of Dictionaries
### Parser Information
Compatibility: linux

View File

@@ -186,4 +186,4 @@ Returns:
### Parser Information
Compatibility: linux
Version 1.7 by Kelly Brazil (kellyjonbrazil@gmail.com)
Version 1.8 by Kelly Brazil (kellyjonbrazil@gmail.com)

View File

@@ -12,8 +12,8 @@ Supports files containing simple key/value pairs.
- If duplicate keys are found, only the last value will be used.
> Note: Values starting and ending with quotation marks will have the marks
removed. If you would like to keep the quotation marks, use the `-r`
command-line argument or the `raw=True` argument in `parse()`.
> removed. If you would like to keep the quotation marks, use the `-r`
> command-line argument or the `raw=True` argument in `parse()`.
Usage (cli):
@@ -26,8 +26,8 @@ Usage (module):
Schema:
key/value document converted to a dictionary - see the
configparser standard library documentation for more details.
key/value document converted to a dictionary - see the configparser standard
library documentation for more details.
{
"key1": string,

View File

@@ -9,11 +9,12 @@ Options supported:
- `lbaR1`
- `--time-style=full-iso`
Note: The `-1`, `-l`, or `-b` option of `ls` should be used to correctly
parse filenames that include newline characters. Since `ls` does not encode
newlines in filenames when outputting to a pipe it will cause `jc` to see
multiple files instead of a single file if `-1`, `-l`, or `-b` is not used.
Alternatively, `vdir` can be used, which is the same as running `ls -lb`.
> Note: The `-1`, `-l`, or `-b` option of `ls` should be used to correctly
> parse filenames that include newline characters. Since `ls` does not
> encode newlines in filenames when outputting to a pipe it will cause `jc`
> to see multiple files instead of a single file if `-1`, `-l`, or `-b` is
> not used. Alternatively, `vdir` can be used, which is the same as running
> `ls -lb`.
The `epoch` calculated timestamp field is naive. (i.e. based on the local
time of the system the parser is run on)

View File

@@ -5,8 +5,8 @@
jc - JSON Convert `ls` and `vdir` command output streaming parser
> This streaming parser outputs JSON Lines (cli) or returns a Generator
iterator of Dictionaries (module)
> This streaming parser outputs JSON Lines (cli) or returns an Iterable of
> Dictionaries (module)
Requires the `-l` option to be used on `ls`. If there are newline characters
in the filename, then make sure to use the `-b` option on `ls`.
@@ -81,7 +81,7 @@ Examples:
def parse(data, raw=False, quiet=False, ignore_exceptions=False)
```
Main text parsing generator function. Returns an iterator object.
Main text parsing generator function. Returns an iterable object.
Parameters:
@@ -92,13 +92,9 @@ Parameters:
quiet: (boolean) suppress warning messages if True
ignore_exceptions: (boolean) ignore parsing exceptions if True
Yields:
Dictionary. Raw or processed structured data.
Returns:
Iterator object (generator)
Iterable of Dictionaries
### Parser Information
Compatibility: linux, darwin, cygwin, aix, freebsd

View File

@@ -22,9 +22,9 @@ Usage (module):
Schema:
Note: <item> object keynames are assigned directly from the lsusb
output. If there are duplicate <item> names in a section, only the
last one is converted.
> Note: <item> object keynames are assigned directly from the lsusb
> output. If there are duplicate <item> names in a section, only the
> last one is converted.
[
{

View File

@@ -5,7 +5,7 @@
jc - JSON Convert `mpstat` command output parser
Note: Latest versions of `mpstat` support JSON output (v11.5.1+)
> Note: Latest versions of `mpstat` support JSON output (v11.5.1+)
Usage (cli):

View File

@@ -5,10 +5,10 @@
jc - JSON Convert `mpstat` command output streaming parser
> This streaming parser outputs JSON Lines (cli) or returns a Generator
iterator of Dictionaries (module)
> This streaming parser outputs JSON Lines (cli) or returns an Iterable of
> Dictionaries (module)
Note: Latest versions of `mpstat` support JSON output (v11.5.1+)
> Note: Latest versions of `mpstat` support JSON output (v11.5.1+)
Usage (cli):
@@ -101,15 +101,13 @@ Examples:
```python
@add_jc_meta
def parse(
data: Iterable[str],
raw: bool = False,
quiet: bool = False,
ignore_exceptions: bool = False
) -> Union[Generator[Dict, None, None], tuple]
def parse(data: Iterable[str],
raw: bool = False,
quiet: bool = False,
ignore_exceptions: bool = False) -> Union[Iterable[Dict], tuple]
```
Main text parsing generator function. Returns an iterator object.
Main text parsing generator function. Returns an iterable object.
Parameters:
@@ -120,13 +118,9 @@ Parameters:
quiet: (boolean) suppress warning messages if True
ignore_exceptions: (boolean) ignore parsing exceptions if True
Yields:
Dictionary. Raw or processed structured data.
Returns:
Iterator object (generator)
Iterable of Dictionaries
### Parser Information
Compatibility: linux

View File

@@ -29,18 +29,18 @@ Usage (module):
Schema:
Because there are so many options, the schema is not strictly defined.
Integer and Float value conversions are attempted and the original
values are kept if they fail. If you don't want automatic conversion,
then use the -r or raw=True option to disable it.
Because there are so many options, the schema is not strictly defined.
Integer and Float value conversions are attempted and the original
values are kept if they fail. If you don't want automatic conversion,
then use the `-r` or `raw=True` option to disable it.
The structure is flat, for the most part, but there are a couple of
"well-known" keys that are further parsed into objects for convenience.
These are documented below.
The structure is flat, for the most part, but there are a couple of
"well-known" keys that are further parsed into objects for convenience.
These are documented below.
[
{
"<key>": string/integer/float, [0]
"<key>": string/integer/float, # [0]
"dhcp4_option_x": {
"name": string,
"value": string/integer/float,

View File

@@ -3,7 +3,7 @@
# jc.parsers.pidstat
jc - JSON Convert `pidstat` command output parser
jc - JSON Convert `pidstat -h` command output parser
Must use the `-h` option in `pidstat`. All other `pidstat` options are
supported in combination with `-h`.

View File

@@ -3,10 +3,10 @@
# jc.parsers.pidstat\_s
jc - JSON Convert `pidstat` command output streaming parser
jc - JSON Convert `pidstat -h` command output streaming parser
> This streaming parser outputs JSON Lines (cli) or returns a Generator
iterator of Dictionaries (module)
> This streaming parser outputs JSON Lines (cli) or returns an Iterable of
> Dictionaries (module)
Must use the `-h` option in `pidstat`. All other `pidstat` options are
supported in combination with `-h`.
@@ -16,11 +16,11 @@ Usage (cli):
$ pidstat | jc --pidstat-s
> Note: When piping `jc` converted `pidstat` output to other processes it
may appear the output is hanging due to the OS pipe buffers. This is
because `pidstat` output is too small to quickly fill up the buffer. Use
the `-u` option to unbuffer the `jc` output if you would like immediate
output. See the [readme](https://github.com/kellyjonbrazil/jc/tree/master#unbuffering-output)
for more information.
> may appear the output is hanging due to the OS pipe buffers. This is
> because `pidstat` output is too small to quickly fill up the buffer. Use
> the `-u` option to unbuffer the `jc` output if you would like immediate
> output. See the [readme](https://github.com/kellyjonbrazil/jc/tree/master#unbuffering-output)
> for more information.
Usage (module):
@@ -83,15 +83,13 @@ Examples:
```python
@add_jc_meta
def parse(
data: Iterable[str],
raw: bool = False,
quiet: bool = False,
ignore_exceptions: bool = False
) -> Union[Generator[Dict, None, None], tuple]
def parse(data: Iterable[str],
raw: bool = False,
quiet: bool = False,
ignore_exceptions: bool = False) -> Union[Iterable[Dict], tuple]
```
Main text parsing generator function. Returns an iterator object.
Main text parsing generator function. Returns an iterable object.
Parameters:
@@ -102,13 +100,9 @@ Parameters:
quiet: (boolean) suppress warning messages if True
ignore_exceptions: (boolean) ignore parsing exceptions if True
Yields:
Dictionary. Raw or processed structured data.
Returns:
Iterator object (generator)
Iterable of Dictionaries
### Parser Information
Compatibility: linux

View File

@@ -9,8 +9,8 @@ Supports `ping` and `ping6` output.
Usage (cli):
Note: Use the ping `-c` (count) option, otherwise data will not be
piped to `jc`.
> Note: Use the ping `-c` (count) option, otherwise data will not be
> piped to `jc`.
$ ping -c 3 1.2.3.4 | jc --ping

View File

@@ -5,21 +5,21 @@
jc - JSON Convert `ping` command output streaming parser
> This streaming parser outputs JSON Lines (cli) or returns a Generator
iterator of Dictionaries (module)
> This streaming parser outputs JSON Lines (cli) or returns an Iterable of
> Dictionaries (module)
Supports `ping` and `ping6` output.
Usage (cli):
$ ping | jc --ping-s
$ ping 1.2.3.4 | jc --ping-s
> Note: When piping `jc` converted `ping` output to other processes it may
appear the output is hanging due to the OS pipe buffers. This is because
`ping` output is too small to quickly fill up the buffer. Use the `-u`
option to unbuffer the `jc` output if you would like immediate output.
See the [readme](https://github.com/kellyjonbrazil/jc/tree/master#unbuffering-output)
for more information.
> appear the output is hanging due to the OS pipe buffers. This is because
> `ping` output is too small to quickly fill up the buffer. Use the `-u`
> option to unbuffer the `jc` output if you would like immediate output.
> See the [readme](https://github.com/kellyjonbrazil/jc/tree/master#unbuffering-output)
> for more information.
Usage (module):
@@ -88,7 +88,7 @@ Examples:
def parse(data, raw=False, quiet=False, ignore_exceptions=False)
```
Main text parsing generator function. Returns an iterator object.
Main text parsing generator function. Returns an iterable object.
Parameters:
@@ -99,13 +99,9 @@ Parameters:
quiet: (boolean) suppress warning messages if True
ignore_exceptions: (boolean) ignore parsing exceptions if True
Yields:
Dictionary. Raw or processed structured data.
Returns:
Iterator object (generator)
Iterable of Dictionaries
### Parser Information
Compatibility: linux, darwin, freebsd

View File

@@ -70,7 +70,7 @@ Examples:
### parse
```python
def parse(data, raw=False, quiet=False)
def parse(data: str, raw: bool = False, quiet: bool = False) -> List[Dict]
```
Main text parsing function
@@ -88,4 +88,4 @@ Returns:
### Parser Information
Compatibility: linux, darwin, cygwin, win32, aix, freebsd
Version 1.3 by Kelly Brazil (kellyjonbrazil@gmail.com)
Version 1.4 by Kelly Brazil (kellyjonbrazil@gmail.com)

115
docs/parsers/postconf.md Normal file
View File

@@ -0,0 +1,115 @@
[Home](https://kellyjonbrazil.github.io/jc/)
<a id="jc.parsers.postconf"></a>
# jc.parsers.postconf
jc - JSON Convert `postconf -M` command output parser
Usage (cli):
$ postconf -M | jc --postconf
or
$ jc postconf -M
Usage (module):
import jc
result = jc.parse('postconf', postconf_command_output)
Schema:
[
{
"service_name": string,
"service_type": string,
"private": boolean/null, # [0]
"unprivileged": boolean/null, # [0]
"chroot": boolean/null, # [0]
"wake_up_time": integer/null, # [0]
"no_wake_up_before_first_use": boolean/null, # [1]
"process_limit": integer/null, # [0]
"command": string
}
]
[0] '-' converted to null/None
[1] null/None if `wake_up_time` is null/None
Examples:
$ postconf -M | jc --postconf -p
[
{
"service_name": "smtp",
"service_type": "inet",
"private": false,
"unprivileged": null,
"chroot": true,
"wake_up_time": null,
"process_limit": null,
"command": "smtpd",
"no_wake_up_before_first_use": null
},
{
"service_name": "pickup",
"service_type": "unix",
"private": false,
"unprivileged": null,
"chroot": true,
"wake_up_time": 60,
"process_limit": 1,
"command": "pickup",
"no_wake_up_before_first_use": false
}
]
$ postconf -M | jc --postconf -p -r
[
{
"service_name": "smtp",
"service_type": "inet",
"private": "n",
"unprivileged": "-",
"chroot": "y",
"wake_up_time": "-",
"process_limit": "-",
"command": "smtpd"
},
{
"service_name": "pickup",
"service_type": "unix",
"private": "n",
"unprivileged": "-",
"chroot": "y",
"wake_up_time": "60",
"process_limit": "1",
"command": "pickup"
}
]
<a id="jc.parsers.postconf.parse"></a>
### parse
```python
def parse(data: str, raw: bool = False, quiet: bool = False) -> List[Dict]
```
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
Returns:
List of Dictionaries. Raw or processed structured data.
### Parser Information
Compatibility: linux
Version 1.0 by Kelly Brazil (kellyjonbrazil@gmail.com)

View File

@@ -22,20 +22,20 @@ Schema:
[
{
"destination": string,
"gateway": string,
"genmask": string,
"flags": string,
"destination": string,
"gateway": string,
"genmask": string,
"flags": string,
"flags_pretty": [
string
string
]
"metric": integer,
"ref": integer,
"use": integer,
"mss": integer,
"window": integer,
"irtt": integer,
"iface": string
"metric": integer,
"ref": integer,
"use": integer,
"mss": integer,
"window": integer,
"irtt": integer,
"iface": string
}
]

View File

@@ -10,7 +10,7 @@ Works with `rpm -qi [package]` or `rpm -qia`.
The `..._epoch` calculated timestamp fields are naive. (i.e. based on the
local time of the system the parser is run on)
The `..._epoch_utc` calculated timestamp fields are timezone-aware and is
The `..._epoch_utc` calculated timestamp fields are timezone-aware and are
only available if the timezone field is UTC.
Usage (cli):

View File

@@ -6,7 +6,7 @@
jc - JSON Convert `rsync` command output parser
Supports the `-i` or `--itemize-changes` options with all levels of
verbosity. This parser will process the STDOUT output or a log file
verbosity. This parser will process the `STDOUT` output or a log file
generated with the `--log-file` option.
Usage (cli):
@@ -51,8 +51,8 @@ Schema:
"time": string,
"process": integer,
"metadata": string,
"update_type": string/null, [0]
"file_type": string/null, [1]
"update_type": string/null, # [0]
"file_type": string/null, # [1]
"checksum_or_value_different": bool/null,
"size_different": bool/null,
"modification_time_different": bool/null,
@@ -61,7 +61,7 @@ Schema:
"group_different": bool/null,
"acl_different": bool/null,
"extended_attribute_different": bool/null,
"epoch": integer, [2]
"epoch": integer, # [2]
}
]
}

View File

@@ -5,11 +5,11 @@
jc - JSON Convert `rsync` command output streaming parser
> This streaming parser outputs JSON Lines (cli) or returns a Generator
iterator of Dictionaries (module)
> This streaming parser outputs JSON Lines (cli) or returns an Iterable of
> Dictionaries (module)
Supports the `-i` or `--itemize-changes` options with all levels of
verbosity. This parser will process the STDOUT output or a log file
verbosity. This parser will process the `STDOUT` output or a log file
generated with the `--log-file` option.
Usage (cli):
@@ -49,8 +49,8 @@ Schema:
"time": string,
"process": integer,
"metadata": string,
"update_type": string/null, [0]
"file_type": string/null, [1]
"update_type": string/null, # [0]
"file_type": string/null, # [1]
"checksum_or_value_different": bool/null,
"size_different": bool/null,
"modification_time_different": bool/null,
@@ -59,7 +59,7 @@ Schema:
"group_different": bool/null,
"acl_different": bool/null,
"extended_attribute_different": bool/null,
"epoch": integer, [2]
"epoch": integer, # [2]
# below object only exists if using -qq or ignore_exceptions=True
"_jc_meta": {
@@ -90,15 +90,13 @@ Examples:
```python
@add_jc_meta
def parse(
data: Iterable[str],
raw: bool = False,
quiet: bool = False,
ignore_exceptions: bool = False
) -> Union[Generator[Dict, None, None], tuple]
def parse(data: Iterable[str],
raw: bool = False,
quiet: bool = False,
ignore_exceptions: bool = False) -> Union[Iterable[Dict], tuple]
```
Main text parsing generator function. Returns an iterator object.
Main text parsing generator function. Returns an iterable object.
Parameters:
@@ -109,13 +107,9 @@ Parameters:
quiet: (boolean) suppress warning messages if True
ignore_exceptions: (boolean) ignore parsing exceptions if True
Yields:
Dictionary. Raw or processed structured data.
Returns:
Iterator object (generator)
Iterable of Dictionaries
### Parser Information
Compatibility: linux, darwin, freebsd

View File

@@ -5,8 +5,8 @@
jc - JSON Convert `ss` command output parser
Extended information options like -e and -p are not supported and may cause
parsing irregularities.
Extended information options like `-e` and `-p` are not supported and may
cause parsing irregularities.
Usage (cli):
@@ -23,8 +23,8 @@ Usage (module):
Schema:
Information from https://www.cyberciti.biz/files/ss.html used to define
field names
Information from https://www.cyberciti.biz/files/ss.html used to define
field names
[
{

View File

@@ -5,8 +5,8 @@
jc - JSON Convert `stat` command output streaming parser
> This streaming parser outputs JSON Lines (cli) or returns a Generator
iterator of Dictionaries (module)
> This streaming parser outputs JSON Lines (cli) or returns an Iterable of
> Dictionaries (module)
The `xxx_epoch` calculated timestamp fields are naive. (i.e. based on the
local time of the system the parser is run on).
@@ -86,7 +86,7 @@ Examples:
def parse(data, raw=False, quiet=False, ignore_exceptions=False)
```
Main text parsing generator function. Returns an iterator object.
Main text parsing generator function. Returns an iterable object.
Parameters:
@@ -97,13 +97,9 @@ Parameters:
quiet: (boolean) suppress warning messages if True
ignore_exceptions: (boolean) ignore parsing exceptions if True
Yields:
Dictionary. Raw or processed structured data.
Returns:
Iterator object (generator)
Iterable of Dictionaries
### Parser Information
Compatibility: linux, darwin, freebsd

View File

@@ -5,10 +5,10 @@
jc - JSON Convert `sysctl -a` command output parser
Note: Since `sysctl` output is not easily parsable only a very simple
key/value object will be output. An attempt is made to convert obvious
integers and floats. If no conversion is desired, use the `-r`
command-line argument or the `raw=True` argument in `parse()`.
> Note: Since `sysctl` output is not easily parsable only a very simple
> key/value object will be output. An attempt is made to convert obvious
> integers and floats. If no conversion is desired, use the `-r`
> command-line argument or the `raw=True` argument in `parse()`.
Usage (cli):

View File

@@ -11,8 +11,8 @@ used to redirect the output to a file that can be read by `jc`.
Alternatively, the output from `/usr/bin/time` can be redirected to `STDOUT`
so `jc` can receive it.
Note: `/usr/bin/time` is similar but different from the Bash builtin
`time` command.
> Note: `/usr/bin/time` is similar but different from the Bash builtin
> `time` command.
Usage (cli):
@@ -26,8 +26,8 @@ Usage (module):
Schema:
Source: https://www.freebsd.org/cgi/man.cgi?query=getrusage
https://man7.org/linux/man-pages/man1/time.1.html
Source: https://www.freebsd.org/cgi/man.cgi?query=getrusage,
https://man7.org/linux/man-pages/man1/time.1.html
{
"real_time": float,

339
docs/parsers/top.md Normal file
View File

@@ -0,0 +1,339 @@
[Home](https://kellyjonbrazil.github.io/jc/)
<a id="jc.parsers.top"></a>
# jc.parsers.top
jc - JSON Convert `top -b` command output parser
Requires batch mode (`-b`). The `-n` option must also be used to limit
the number of times `top` is run.
Warning messages will be printed to `STDERR` if truncated fields are
detected. These warnings can be suppressed with the `-q` or `quiet=True`
option.
Usage (cli):
$ top -b -n 3 | jc --top
or
$ jc top -b -n 3
Usage (module):
import jc
result = jc.parse('top', top_command_output)
Schema:
All `-` values are converted to `null`
[
{
"time": string,
"uptime": integer,
"users": integer,
"load_1m": float,
"load_5m": float,
"load_15m": float,
"tasks_total": integer,
"tasks_running": integer,
"tasks_sleeping": integer,
"tasks_stopped": integer,
"tasks_zombie": integer,
"cpu_user": float,
"cpu_sys": float,
"cpu_nice": float,
"cpu_idle": float,
"cpu_wait": float,
"cpu_hardware": float,
"cpu_software": float,
"cpu_steal": float,
"mem_total": float, # [0]
"mem_free": float, # [0]
"mem_used": float, # [0]
"mem_buff_cache": float, # [0]
"swap_total": float, # [0]
"swap_free": float, # [0]
"swap_used": float, # [0]
"mem_available": float, # [0]
"processes": [
{
"pid": integer,
"user": string,
"priority": integer,
"nice": integer,
"virtual_mem": float, # [1]
"resident_mem": float, # [1]
"shared_mem": float, # [1]
"status": string,
"percent_cpu": float,
"percent_mem": float,
"time_hundredths": string,
"command": string,
"parent_pid": integer,
"uid": integer,
"real_uid": integer,
"real_user": string,
"saved_uid": integer,
"saved_user": string,
"gid": integer,
"group": string,
"pgrp": integer,
"tty": string,
"tty_process_gid": integer,
"session_id": integer,
"thread_count": integer,
"last_used_processor": integer,
"time": string,
"swap": float, # [1]
"code": float, # [1]
"data": float, # [1]
"major_page_fault_count": integer,
"minor_page_fault_count": integer,
"dirty_pages_count": integer,
"sleeping_in_function": string,
"flags": string,
"cgroups": string,
"supplementary_gids": [
integer
],
"supplementary_groups": [
string
],
"thread_gid": integer,
"environment_variables": [
string
]
"major_page_fault_count_delta": integer,
"minor_page_fault_count_delta": integer,
"used": float, # [1]
"ipc_namespace_inode": integer,
"mount_namespace_inode": integer,
"net_namespace_inode": integer,
"pid_namespace_inode": integer,
"user_namespace_inode": integer,
"nts_namespace_inode": integer,
"control_group_name": string,
"lxc_container_name": string,
"numa_node": integer,
"out_of_mem_adjustment": integer,
"out_of_mem_score": integer,
"resident_anon_mem": integer,
"resident_file_backed_mem": integer,
"resident_locked_mem": integer,
"resident_shared_mem": integer
}
]
}
]
[0] Values are in the units output by `top`
[1] Unit suffix stripped during float conversion
Examples:
$ top -b -n 3 | jc --top -p
[
{
"time": "11:20:43",
"uptime": 118,
"users": 2,
"load_1m": 0.0,
"load_5m": 0.01,
"load_15m": 0.05,
"tasks_total": 108,
"tasks_running": 2,
"tasks_sleeping": 106,
"tasks_stopped": 0,
"tasks_zombie": 0,
"cpu_user": 5.6,
"cpu_sys": 11.1,
"cpu_nice": 0.0,
"cpu_idle": 83.3,
"cpu_wait": 0.0,
"cpu_hardware": 0.0,
"cpu_software": 0.0,
"cpu_steal": 0.0,
"mem_total": 3.7,
"mem_free": 3.3,
"mem_used": 0.2,
"mem_buff_cache": 0.2,
"swap_total": 2.0,
"swap_free": 2.0,
"swap_used": 0.0,
"mem_available": 3.3,
"processes": [
{
"pid": 2225,
"user": "kbrazil",
"priority": 20,
"nice": 0,
"virtual_mem": 158.1,
"resident_mem": 2.2,
"shared_mem": 1.6,
"status": "running",
"percent_cpu": 12.5,
"percent_mem": 0.1,
"time_hundredths": "0:00.02",
"command": "top",
"parent_pid": 1884,
"uid": 1000,
"real_uid": 1000,
"real_user": "kbrazil",
"saved_uid": 1000,
"saved_user": "kbrazil",
"gid": 1000,
"group": "kbrazil",
"pgrp": 2225,
"tty": "pts/0",
"tty_process_gid": 2225,
"session_id": 1884,
"thread_count": 1,
"last_used_processor": 0,
"time": "0:00",
"swap": 0.0,
"code": 0.1,
"data": 1.0,
"major_page_fault_count": 0,
"minor_page_fault_count": 736,
"dirty_pages_count": 0,
"sleeping_in_function": null,
"flags": "..4.2...",
"cgroups": "1:name=systemd:/user.slice/user-1000.+",
"supplementary_gids": [
10,
1000
],
"supplementary_groups": [
"wheel",
"kbrazil"
],
"thread_gid": 2225,
"environment_variables": [
"XDG_SESSION_ID=2",
"HOSTNAME=localhost"
],
"major_page_fault_count_delta": 0,
"minor_page_fault_count_delta": 4,
"used": 2.2,
"ipc_namespace_inode": 4026531839,
"mount_namespace_inode": 4026531840,
"net_namespace_inode": 4026531956,
"pid_namespace_inode": 4026531836,
"user_namespace_inode": 4026531837,
"nts_namespace_inode": 4026531838
},
...
]
}
]
$ top -b -n 3 | jc --top -p -r
[
{
"time": "11:20:43",
"uptime": "1:18",
"users": "2",
"load_1m": "0.00",
"load_5m": "0.01",
"load_15m": "0.05",
"tasks_total": "108",
"tasks_running": "2",
"tasks_sleeping": "106",
"tasks_stopped": "0",
"tasks_zombie": "0",
"cpu_user": "5.6",
"cpu_sys": "11.1",
"cpu_nice": "0.0",
"cpu_idle": "83.3",
"cpu_wait": "0.0",
"cpu_hardware": "0.0",
"cpu_software": "0.0",
"cpu_steal": "0.0",
"swap_total": "2.0",
"swap_free": "2.0",
"swap_used": "0.0",
"mem_available": "3.3",
"processes": [
{
"PID": "2225",
"USER": "kbrazil",
"PR": "20",
"NI": "0",
"VIRT": "158.1m",
"RES": "2.2m",
"SHR": "1.6m",
"S": "R",
"%CPU": "12.5",
"%MEM": "0.1",
"TIME+": "0:00.02",
"COMMAND": "top",
"PPID": "1884",
"UID": "1000",
"RUID": "1000",
"RUSER": "kbrazil",
"SUID": "1000",
"SUSER": "kbrazil",
"GID": "1000",
"GROUP": "kbrazil",
"PGRP": "2225",
"TTY": "pts/0",
"TPGID": "2225",
"SID": "1884",
"nTH": "1",
"P": "0",
"TIME": "0:00",
"SWAP": "0.0m",
"CODE": "0.1m",
"DATA": "1.0m",
"nMaj": "0",
"nMin": "736",
"nDRT": "0",
"WCHAN": "-",
"Flags": "..4.2...",
"CGROUPS": "1:name=systemd:/user.slice/user-1000.+",
"SUPGIDS": "10,1000",
"SUPGRPS": "wheel,kbrazil",
"TGID": "2225",
"ENVIRON": "XDG_SESSION_ID=2 HOSTNAME=localhost S+",
"vMj": "0",
"vMn": "4",
"USED": "2.2m",
"nsIPC": "4026531839",
"nsMNT": "4026531840",
"nsNET": "4026531956",
"nsPID": "4026531836",
"nsUSER": "4026531837",
"nsUTS": "4026531838"
},
...
]
}
]
<a id="jc.parsers.top.parse"></a>
### parse
```python
def parse(data: str, raw: bool = False, quiet: bool = False) -> List[Dict]
```
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
Returns:
List of Dictionaries. Raw or processed structured data.
### Parser Information
Compatibility: linux
Version 1.0 by Kelly Brazil (kellyjonbrazil@gmail.com)

180
docs/parsers/top_s.md Normal file
View File

@@ -0,0 +1,180 @@
[Home](https://kellyjonbrazil.github.io/jc/)
<a id="jc.parsers.top_s"></a>
# jc.parsers.top\_s
jc - JSON Convert `top -b` command output streaming parser
> This streaming parser outputs JSON Lines (cli) or returns an Iterable of
> Dictionaries (module)
Requires batch mode (`-b`).
Warning messages will be printed to `STDERR` if truncated fields are
detected. These warnings can be suppressed with the `-q` or `quiet=True`
option.
Usage (cli):
$ top -b | jc --top-s
Usage (module):
import jc
result = jc.parse('top_s', top_command_output.splitlines())
for item in result:
# do something
Schema:
{
"time": string,
"uptime": integer,
"users": integer,
"load_1m": float,
"load_5m": float,
"load_15m": float,
"tasks_total": integer,
"tasks_running": integer,
"tasks_sleeping": integer,
"tasks_stopped": integer,
"tasks_zombie": integer,
"cpu_user": float,
"cpu_sys": float,
"cpu_nice": float,
"cpu_idle": float,
"cpu_wait": float,
"cpu_hardware": float,
"cpu_software": float,
"cpu_steal": float,
"mem_total": float, # [0]
"mem_free": float, # [0]
"mem_used": float, # [0]
"mem_buff_cache": float, # [0]
"swap_total": float, # [0]
"swap_free": float, # [0]
"swap_used": float, # [0]
"mem_available": float, # [0]
"processes": [
{
"pid": integer,
"user": string,
"priority": integer,
"nice": integer,
"virtual_mem": float, # [1]
"resident_mem": float, # [1]
"shared_mem": float, # [1]
"status": string,
"percent_cpu": float,
"percent_mem": float,
"time_hundredths": string,
"command": string,
"parent_pid": integer,
"uid": integer,
"real_uid": integer,
"real_user": string,
"saved_uid": integer,
"saved_user": string,
"gid": integer,
"group": string,
"pgrp": integer,
"tty": string,
"tty_process_gid": integer,
"session_id": integer,
"thread_count": integer,
"last_used_processor": integer,
"time": string,
"swap": float, # [1]
"code": float, # [1]
"data": float, # [1]
"major_page_fault_count": integer,
"minor_page_fault_count": integer,
"dirty_pages_count": integer,
"sleeping_in_function": string,
"flags": string,
"cgroups": string,
"supplementary_gids": [
integer
],
"supplementary_groups": [
string
],
"thread_gid": integer,
"environment_variables": [
string
]
"major_page_fault_count_delta": integer,
"minor_page_fault_count_delta": integer,
"used": float, # [1]
"ipc_namespace_inode": integer,
"mount_namespace_inode": integer,
"net_namespace_inode": integer,
"pid_namespace_inode": integer,
"user_namespace_inode": integer,
"nts_namespace_inode": integer,
"control_group_name": string,
"lxc_container_name": string,
"numa_node": integer,
"out_of_mem_adjustment": integer,
"out_of_mem_score": integer,
"resident_anon_mem": integer,
"resident_file_backed_mem": integer,
"resident_locked_mem": integer,
"resident_shared_mem": integer
}
],
# below object only exists if using -qq or ignore_exceptions=True
"_jc_meta": {
"success": boolean, # false if error parsing
"error": string, # exists if "success" is false
"line": string # exists if "success" is false
}
}
[0] Values are in the units output by `top`
[1] Unit suffix stripped during float conversion
Examples:
$ top -b | jc --top-s
{"time":"11:24:50","uptime":2,"users":2,"load_1m":0.23,"load_5m":...}
...
$ top -b | jc --top-s -r
{"time":"11:24:50","uptime":"2 min","users":"2","load_1m":"0.23","lo...}
...
<a id="jc.parsers.top_s.parse"></a>
### parse
```python
@add_jc_meta
def parse(data: Iterable[str],
raw: bool = False,
quiet: bool = False,
ignore_exceptions: bool = False) -> Union[Iterable[Dict], tuple]
```
Main text parsing generator function. Returns an iterable object.
Parameters:
data: (iterable) line-based text data to parse
(e.g. sys.stdin or str.splitlines())
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
ignore_exceptions: (boolean) ignore parsing exceptions if True
Returns:
Iterable of Dictionaries
### Parser Information
Compatibility: linux
Version 1.0 by Kelly Brazil (kellyjonbrazil@gmail.com)

View File

@@ -7,12 +7,12 @@ jc - JSON Convert `traceroute` command output parser
Supports `traceroute` and `traceroute6` output.
Note: On some operating systems you will need to redirect `STDERR` to
`STDOUT` for destination info since the header line is sent to
`STDERR`. A warning message will be printed to `STDERR` if the
header row is not found.
e.g. `$ traceroute 8.8.8.8 2>&1 | jc --traceroute`
> Note: On some operating systems you will need to redirect `STDERR` to
> `STDOUT` for destination info since the header line is sent to
> `STDERR`. A warning message will be printed to `STDERR` if the
> header row is not found.
>
> e.g. `$ traceroute 8.8.8.8 2>&1 | jc --traceroute`
Usage (cli):

View File

@@ -5,7 +5,7 @@
jc - JSON Convert `uname -a` command output parser
Note: Must use `uname -a`
> Note: Must use `uname -a`
Usage (cli):

View File

@@ -5,8 +5,8 @@
jc - JSON Convert `vmstat` command output streaming parser
> This streaming parser outputs JSON Lines (cli) or returns a Generator
iterator of Dictionaries (module)
> This streaming parser outputs JSON Lines (cli) or returns an Iterable of
> Dictionaries (module)
Options supported: `-a`, `-w`, `-d`, `-t`
@@ -21,11 +21,11 @@ Usage (cli):
$ vmstat | jc --vmstat-s
> Note: When piping `jc` converted `vmstat` output to other processes it may
appear the output is hanging due to the OS pipe buffers. This is because
`vmstat` output is too small to quickly fill up the buffer. Use the `-u`
option to unbuffer the `jc` output if you would like immediate output. See
the [readme](https://github.com/kellyjonbrazil/jc/tree/master#unbuffering-output)
for more information.
> appear the output is hanging due to the OS pipe buffers. This is because
> `vmstat` output is too small to quickly fill up the buffer. Use the `-u`
> option to unbuffer the `jc` output if you would like immediate output. See
> the [readme](https://github.com/kellyjonbrazil/jc/tree/master#unbuffering-output)
> for more information.
Usage (module):
@@ -75,9 +75,9 @@ Schema:
# below object only exists if using -qq or ignore_exceptions=True
"_jc_meta": {
"success": boolean, # [2]
"error": string, # [3]
"line": string # [3]
"success": boolean, # [2]
"error": string, # [3]
"line": string # [3]
}
}
@@ -105,7 +105,7 @@ Examples:
def parse(data, raw=False, quiet=False, ignore_exceptions=False)
```
Main text parsing generator function. Returns an iterator object.
Main text parsing generator function. Returns an iterable object.
Parameters:
@@ -116,13 +116,9 @@ Parameters:
quiet: (boolean) suppress warning messages if True
ignore_exceptions: (boolean) ignore parsing exceptions if True
Yields:
Dictionary. Raw or processed structured data.
Returns:
Iterator object (generator)
Iterable of Dictionaries
### Parser Information
Compatibility: linux

225
docs/parsers/x509_cert.md Normal file
View File

@@ -0,0 +1,225 @@
[Home](https://kellyjonbrazil.github.io/jc/)
<a id="jc.parsers.x509_cert"></a>
# jc.parsers.x509\_cert
jc - JSON Convert X.509 Certificate format file parser
This parser will convert DER and PEM encoded X.509 certificate files.
Usage (cli):
$ cat certificate.pem | jc --x509-cert
Usage (module):
import jc
result = jc.parse('x509_cert', x509_cert_file_output)
Schema:
[
{
"tbs_certificate": {
"version": string,
"serial_number": string, # [0]
"signature": {
"algorithm": string,
"parameters": string/null,
},
"issuer": {
"country_name": string,
"state_or_province_name" string,
"locality_name": string,
"organization_name": array/string,
"organizational_unit_name": array/string,
"common_name": string,
"email_address": string
},
"validity": {
"not_before": integer, # [1]
"not_after": integer, # [1]
"not_before_iso": string,
"not_after_iso": string
},
"subject": {
"country_name": string,
"state_or_province_name": string,
"locality_name": string,
"organization_name": array/string,
"organizational_unit_name": array/string,
"common_name": string,
"email_address": string
},
"subject_public_key_info": {
"algorithm": {
"algorithm": string,
"parameters": string/null,
},
"public_key": {
"modulus": string, # [0]
"public_exponent": integer
}
},
"issuer_unique_id": string/null,
"subject_unique_id": string/null,
"extensions": [
{
"extn_id": string,
"critical": boolean,
"extn_value": array/object/string/integer # [2]
}
]
},
"signature_algorithm": {
"algorithm": string,
"parameters": string/null
},
"signature_value": string # [0]
}
]
[0] in colon-delimited hex notation
[1] time-zone-aware (UTC) epoch timestamp
[2] See below for well-known Extension schemas:
Basic Constraints:
{
"extn_id": "basic_constraints",
"critical": boolean,
"extn_value": {
"ca": boolean,
"path_len_constraint": string/null
}
}
Key Usage:
{
"extn_id": "key_usage",
"critical": boolean,
"extn_value": [
string
]
}
Key Identifier:
{
"extn_id": "key_identifier",
"critical": boolean,
"extn_value": string # [0]
}
Authority Key Identifier:
{
"extn_id": "authority_key_identifier",
"critical": boolean,
"extn_value": {
"key_identifier": string, # [0]
"authority_cert_issuer": string/null,
"authority_cert_serial_number": string/null
}
}
Examples:
$ cat entrust-ec1.pem| jc --x509-cert -p
[
{
"tbs_certificate": {
"version": "v3",
"serial_number": "a6:8b:79:29:00:00:00:00:50:d0:91:f9",
"signature": {
"algorithm": "sha384_ecdsa",
"parameters": null
},
"issuer": {
"country_name": "US",
"organization_name": "Entrust, Inc.",
"organizational_unit_name": [
"See www.entrust.net/legal-terms",
"(c) 2012 Entrust, Inc. - for authorized use only"
],
"common_name": "Entrust Root Certification Authority - EC1"
},
"validity": {
"not_before": 1355844336,
"not_after": 2144764536,
"not_before_iso": "2012-12-18T15:25:36+00:00",
"not_after_iso": "2037-12-18T15:55:36+00:00"
},
"subject": {
"country_name": "US",
"organization_name": "Entrust, Inc.",
"organizational_unit_name": [
"See www.entrust.net/legal-terms",
"(c) 2012 Entrust, Inc. - for authorized use only"
],
"common_name": "Entrust Root Certification Authority - EC1"
},
"subject_public_key_info": {
"algorithm": {
"algorithm": "ec",
"parameters": "secp384r1"
},
"public_key": "04:84:13:c9:d0:ba:6d:41:7b:e2:6c:d0:eb:55:..."
},
"issuer_unique_id": null,
"subject_unique_id": null,
"extensions": [
{
"extn_id": "key_usage",
"critical": true,
"extn_value": [
"crl_sign",
"key_cert_sign"
]
},
{
"extn_id": "basic_constraints",
"critical": true,
"extn_value": {
"ca": true,
"path_len_constraint": null
}
},
{
"extn_id": "key_identifier",
"critical": false,
"extn_value": "b7:63:e7:1a:dd:8d:e9:08:a6:55:83:a4:e0:6a:..."
}
]
},
"signature_algorithm": {
"algorithm": "sha384_ecdsa",
"parameters": null
},
"signature_value": "30:64:02:30:61:79:d8:e5:42:47:df:1c:ae:53:..."
}
]
<a id="jc.parsers.x509_cert.parse"></a>
### parse
```python
def parse(data: Union[str, bytes],
raw: bool = False,
quiet: bool = False) -> List[Dict]
```
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
Returns:
List of Dictionaries. Raw or processed structured data.
### Parser Information
Compatibility: linux, darwin, cygwin, win32, aix, freebsd
Version 1.0 by Kelly Brazil (kellyjonbrazil@gmail.com)

View File

@@ -16,8 +16,8 @@ Usage (module):
Schema:
XML Document converted to a Dictionary
See https://github.com/martinblech/xmltodict for details
XML Document converted to a Dictionary. See https://github.com/martinblech/xmltodict
for details.
{
"key1": string/object,

View File

@@ -5,7 +5,7 @@
jc - JSON Convert `YAML` file parser
Note: datetime objects will be converted to strings.
> Note: `datetime` objects will be converted to strings.
Usage (cli):

View File

@@ -5,10 +5,7 @@
jc - JSON Convert `zipinfo` command output parser
Options supported:
- none
Note: The default listing format.
> Note: No `zipinfo` options are supported.
Usage (cli):

View File

@@ -26,11 +26,12 @@ https://github.com/kellyjonbrazil/jc/tree/master/docs
### Specific Version
Replace `<full_version_number>` - e.g. `1.17.7`:
`https://github.com/kellyjonbrazil/jc/tree/v<full_version_number>/docs`
Specific versions can also be selected by tag in the branch dropdown menu.
> Replace `<full_version_number>` - e.g. `1.18.0`:
Specific versions can also be selected by tag in the Github branch dropdown
menu.
## Usage Example
@@ -93,14 +94,14 @@ its module name.
### parser_mod_list
parser_mod_list() -> list
parser_mod_list() -> list[str]
Get a list of all available parser module names to be used in
`parse()`, `parser_info()`, and `get_help()`.
### plugin_parser_mod_list
plugin_parser_mod_list() -> list
plugin_parser_mod_list() -> list[str]
Get a list of plugin parser module names to be used in
`parse()`, `parser_info()`, and `get_help()`. This list is a subset of
@@ -108,7 +109,7 @@ Get a list of plugin parser module names to be used in
### standard_parser_mod_list
standard_parser_mod_list() -> list
standard_parser_mod_list() -> list[str]
Get a list of standard parser module names to be used in
`parse()`, `parser_info()`, and `get_help()`. This list is a subset of
@@ -116,7 +117,7 @@ Get a list of standard parser module names to be used in
### streaming_parser_mod_list
streaming_parser_mod_list() -> list
streaming_parser_mod_list() -> list[str]
Get a list of streaming parser module names to be used in
`parse()`, `parser_info()`, and `get_help()`. This list is a subset of

View File

@@ -3,6 +3,7 @@
* [jc.utils](#jc.utils)
* [warning\_message](#jc.utils.warning_message)
* [error\_message](#jc.utils.error_message)
* [is\_compatible](#jc.utils.is_compatible)
* [compatibility](#jc.utils.compatibility)
* [has\_data](#jc.utils.has_data)
* [convert\_to\_int](#jc.utils.convert_to_int)
@@ -26,8 +27,8 @@ jc - JSON Convert utils
def warning_message(message_lines: List[str]) -> None
```
Prints warning message for non-fatal issues. The first line is
prepended with 'jc: Warning - ' and subsequent lines are indented.
Prints warning message to `STDERR` for non-fatal issues. The first line
is prepended with 'jc: Warning - ' and subsequent lines are indented.
Wraps text as needed based on the terminal width.
Parameters:
@@ -46,7 +47,7 @@ Returns:
def error_message(message_lines: List[str]) -> None
```
Prints an error message for fatal issues. The first line is
Prints an error message to `STDERR` for fatal issues. The first line is
prepended with 'jc: Error - ' and subsequent lines are indented.
Wraps text as needed based on the terminal width.
@@ -58,6 +59,16 @@ Returns:
None - just prints output to STDERR
<a id="jc.utils.is_compatible"></a>
### is\_compatible
```python
def is_compatible(compatible: List) -> bool
```
Returns True if the parser is compatible with the running OS platform.
<a id="jc.utils.compatibility"></a>
### compatibility
@@ -68,8 +79,9 @@ def compatibility(mod_name: str,
quiet: bool = False) -> None
```
Checks for the parser's compatibility with the running OS
platform.
Checks for the parser's compatibility with the running OS platform and
prints a warning message to `STDERR` if not compatible and
`quiet=False.`
Parameters:
@@ -90,20 +102,23 @@ Returns:
### has\_data
```python
def has_data(data: str) -> bool
def has_data(data: Union[str, bytes]) -> bool
```
Checks if the input contains data. If there are any non-whitespace
characters then return True, else return False.
Checks if the string input contains data. If there are any
non-whitespace characters then return `True`, else return `False`.
For bytes, returns True if there is any data.
Parameters:
data: (string) input to check whether it contains data
data: (string, bytes) input to check whether it contains data
Returns:
Boolean True if input string (data) contains non-whitespace
characters, otherwise False
characters, otherwise False. For bytes data, returns
True if there is any data, otherwise False.
<a id="jc.utils.convert_to_int"></a>

View File

@@ -22,11 +22,12 @@ https://github.com/kellyjonbrazil/jc/tree/master/docs
### Specific Version
Replace `<full_version_number>` - e.g. `1.17.7`:
`https://github.com/kellyjonbrazil/jc/tree/v<full_version_number>/docs`
Specific versions can also be selected by tag in the branch dropdown menu.
> Replace `<full_version_number>` - e.g. `1.18.0`:
Specific versions can also be selected by tag in the Github branch dropdown
menu.
## Usage Example
@@ -89,14 +90,14 @@ its module name.
### parser_mod_list
parser_mod_list() -> list
parser_mod_list() -> list[str]
Get a list of all available parser module names to be used in
`parse()`, `parser_info()`, and `get_help()`.
### plugin_parser_mod_list
plugin_parser_mod_list() -> list
plugin_parser_mod_list() -> list[str]
Get a list of plugin parser module names to be used in
`parse()`, `parser_info()`, and `get_help()`. This list is a subset of
@@ -104,7 +105,7 @@ Get a list of plugin parser module names to be used in
### standard_parser_mod_list
standard_parser_mod_list() -> list
standard_parser_mod_list() -> list[str]
Get a list of standard parser module names to be used in
`parse()`, `parser_info()`, and `get_help()`. This list is a subset of
@@ -112,7 +113,7 @@ Get a list of standard parser module names to be used in
### streaming_parser_mod_list
streaming_parser_mod_list() -> list
streaming_parser_mod_list() -> list[str]
Get a list of streaming parser module names to be used in
`parse()`, `parser_info()`, and `get_help()`. This list is a subset of

295
jc/cli.py
View File

@@ -2,16 +2,20 @@
JC cli module
"""
import io
import sys
import os
import textwrap
import signal
import shlex
import subprocess
import json
from typing import List, Dict
from .lib import (__version__, parser_info, all_parser_info, parsers,
_get_parser, _parser_is_streaming)
_get_parser, _parser_is_streaming, standard_parser_mod_list,
plugin_parser_mod_list, streaming_parser_mod_list)
from . import utils
from .cli_data import long_options_map, new_pygments_colors, old_pygments_colors
from .shell_completions import bash_completion, zsh_completion
from . import tracebackplus
from .exceptions import LibraryNotInstalled, ParseError
@@ -21,13 +25,12 @@ try:
from pygments import highlight
from pygments.style import Style
from pygments.token import (Name, Number, String, Keyword)
from pygments.lexers import JsonLexer
from pygments.lexers.data import JsonLexer, YamlLexer
from pygments.formatters import Terminal256Formatter
PYGMENTS_INSTALLED = True
except Exception:
PYGMENTS_INSTALLED = False
JC_ERROR_EXIT = 100
@@ -45,63 +48,9 @@ class info():
# startswith is sufficient and avoids potential exceptions from split and int.
if PYGMENTS_INSTALLED:
if pygments.__version__.startswith('2.3.'):
PYGMENT_COLOR = {
'black': '#ansiblack',
'red': '#ansidarkred',
'green': '#ansidarkgreen',
'yellow': '#ansibrown',
'blue': '#ansidarkblue',
'magenta': '#ansipurple',
'cyan': '#ansiteal',
'gray': '#ansilightgray',
'brightblack': '#ansidarkgray',
'brightred': '#ansired',
'brightgreen': '#ansigreen',
'brightyellow': '#ansiyellow',
'brightblue': '#ansiblue',
'brightmagenta': '#ansifuchsia',
'brightcyan': '#ansiturquoise',
'white': '#ansiwhite',
}
PYGMENT_COLOR = old_pygments_colors
else:
PYGMENT_COLOR = {
'black': 'ansiblack',
'red': 'ansired',
'green': 'ansigreen',
'yellow': 'ansiyellow',
'blue': 'ansiblue',
'magenta': 'ansimagenta',
'cyan': 'ansicyan',
'gray': 'ansigray',
'brightblack': 'ansibrightblack',
'brightred': 'ansibrightred',
'brightgreen': 'ansibrightgreen',
'brightyellow': 'ansibrightyellow',
'brightblue': 'ansibrightblue',
'brightmagenta': 'ansibrightmagenta',
'brightcyan': 'ansibrightcyan',
'white': 'ansiwhite',
}
def safe_print_json(string, pretty=None, env_colors=None, mono=None,
piped_out=None, flush=None):
"""Safely prints JSON output in both UTF-8 and ASCII systems"""
try:
print(json_out(string,
pretty=pretty,
env_colors=env_colors,
mono=mono,
piped_out=piped_out),
flush=flush)
except UnicodeEncodeError:
print(json_out(string,
pretty=pretty,
env_colors=env_colors,
mono=mono,
piped_out=piped_out,
ascii_only=True),
flush=flush)
PYGMENT_COLOR = new_pygments_colors
def set_env_colors(env_colors=None):
@@ -154,7 +103,7 @@ def set_env_colors(env_colors=None):
def piped_output(force_color):
"""
Return False if stdout is a TTY. True if output is being piped to
Return False if `STDOUT` is a TTY. True if output is being piped to
another program and foce_color is True. This allows forcing of ANSI
color codes even when using pipes.
"""
@@ -186,6 +135,22 @@ def parsers_text(indent=0, pad=0):
return ptext
def options_text(indent=0, pad=0):
"""Return the argument and description information from each option"""
otext = ''
padding_char = ' '
for option in long_options_map:
o_short = '-' + long_options_map[option][0]
o_desc = long_options_map[option][1]
o_combined = o_short + ', ' + option
padding = pad - len(o_combined)
indent_text = padding_char * indent
padding_text = padding_char * padding
otext += indent_text + o_combined + padding_text + o_desc + '\n'
return otext
def about_jc():
"""Return jc info and the contents of each parser.info as a dictionary"""
return {
@@ -200,48 +165,44 @@ def about_jc():
'python_version': '.'.join((str(sys.version_info.major), str(sys.version_info.minor), str(sys.version_info.micro))),
'python_path': sys.executable,
'parser_count': len(all_parser_info()),
'standard_parser_count': len(standard_parser_mod_list()),
'streaming_parser_count': len(streaming_parser_mod_list()),
'plugin_parser_count': len(plugin_parser_mod_list()),
'parsers': all_parser_info()
}
def helptext():
"""Return the help text with the list of parsers"""
parsers_string = parsers_text(indent=12, pad=17)
parsers_string = parsers_text(indent=4, pad=20)
options_string = options_text(indent=4, pad=20)
helptext_string = f'''\
jc converts the output of many commands and file-types to JSON
jc converts the output of many commands and file-types to JSON or YAML
Usage: COMMAND | jc PARSER [OPTIONS]
Usage:
COMMAND | jc PARSER [OPTIONS]
or magic syntax:
or magic syntax:
jc [OPTIONS] COMMAND
jc [OPTIONS] COMMAND
Parsers:
Parsers:
{parsers_string}
Options:
-a about jc
-C force color output even when using pipes (overrides -m)
-d debug (-dd for verbose debug)
-h help (-h --parser_name for parser documentation)
-m monochrome output
-p pretty print output
-q quiet - suppress parser warnings (-qq to ignore streaming errors)
-r raw JSON output
-u unbuffer output
-v version info
Options:
{options_string}
Examples:
Standard Syntax:
$ dig www.google.com | jc --dig --pretty
Examples:
Standard Syntax:
$ dig www.google.com | jc --dig -p
Magic Syntax:
$ jc --pretty dig www.google.com
Magic Syntax:
$ jc -p dig www.google.com
Parser Documentation:
$ jc --help --dig
'''
Parser Documentation:
$ jc -h --dig
'''
return textwrap.dedent(helptext_string)
return helptext_string
def help_doc(options):
@@ -283,11 +244,52 @@ def versiontext():
return textwrap.dedent(versiontext_string)
def yaml_out(data, pretty=False, env_colors=None, mono=False, piped_out=False, ascii_only=False):
"""
Return a YAML formatted string. String may include color codes. If the
YAML library is not installed, output will fall back to JSON with a
warning message to STDERR"""
# make ruamel.yaml import optional
try:
from ruamel.yaml import YAML
YAML_INSTALLED = True
except Exception:
YAML_INSTALLED = False
if YAML_INSTALLED:
y_string_buf = io.BytesIO()
# monkey patch to disable plugins since we don't use them and in
# ruamel.yaml versions prior to 0.17.0 the use of __file__ in the
# plugin code is incompatible with the pyoxidizer packager
YAML.official_plug_ins = lambda a: []
yaml = YAML()
yaml.default_flow_style = False
yaml.explicit_start = True
yaml.allow_unicode = not ascii_only
yaml.encoding = 'utf-8'
yaml.dump(data, y_string_buf)
y_string = y_string_buf.getvalue().decode('utf-8')[:-1]
if not mono and not piped_out:
# set colors
class JcStyle(Style):
styles = set_env_colors(env_colors)
return str(highlight(y_string, YamlLexer(), Terminal256Formatter(style=JcStyle))[0:-1])
return y_string
utils.warning_message(['YAML Library not installed. Reverting to JSON output.'])
return json_out(data, pretty=pretty, env_colors=env_colors, mono=mono, piped_out=piped_out, ascii_only=ascii_only)
def json_out(data, pretty=False, env_colors=None, mono=False, piped_out=False, ascii_only=False):
"""
Return a JSON formatted string. String may include color codes or be
pretty printed.
"""
import json
separators = (',', ':')
indent = None
@@ -307,6 +309,44 @@ def json_out(data, pretty=False, env_colors=None, mono=False, piped_out=False, a
return j_string
def safe_print_out(list_or_dict, pretty=None, env_colors=None, mono=None,
piped_out=None, flush=None, yaml=None):
"""Safely prints JSON or YAML output in both UTF-8 and ASCII systems"""
if yaml:
try:
print(yaml_out(list_or_dict,
pretty=pretty,
env_colors=env_colors,
mono=mono,
piped_out=piped_out),
flush=flush)
except UnicodeEncodeError:
print(yaml_out(list_or_dict,
pretty=pretty,
env_colors=env_colors,
mono=mono,
piped_out=piped_out,
ascii_only=True),
flush=flush)
else:
try:
print(json_out(list_or_dict,
pretty=pretty,
env_colors=env_colors,
mono=mono,
piped_out=piped_out),
flush=flush)
except UnicodeEncodeError:
print(json_out(list_or_dict,
pretty=pretty,
env_colors=env_colors,
mono=mono,
piped_out=piped_out,
ascii_only=True),
flush=flush)
def magic_parser(args):
"""
Parse command arguments for magic syntax: jc -p ls -al
@@ -318,7 +358,7 @@ def magic_parser(args):
jc_options (list) list of jc options
"""
# bail immediately if there are no args or a parser is defined
if len(args) <= 1 or args[1].startswith('--'):
if len(args) <= 1 or (args[1].startswith('--') and args[1] not in long_options_map):
return False, None, None, []
args_given = args[1:]
@@ -326,6 +366,12 @@ def magic_parser(args):
# find the options
for arg in list(args_given):
# long option found - populate option list
if arg in long_options_map:
options.extend(long_options_map[arg][0])
args_given.pop(0)
continue
# parser found - use standard syntax
if arg.startswith('--'):
return False, None, None, []
@@ -420,6 +466,9 @@ def main():
# find options if magic_parser did not find a command
if not valid_command:
for opt in sys.argv:
if opt in long_options_map:
options.extend(long_options_map[opt][0])
if opt.startswith('-') and not opt.startswith('--'):
options.extend(opt[1:])
@@ -435,6 +484,9 @@ def main():
raw = 'r' in options
unbuffer = 'u' in options
version_info = 'v' in options
yaml_out = 'y' in options
bash_comp = 'B' in options
zsh_comp = 'Z' in options
if verbose_debug:
tracebackplus.enable(context=11)
@@ -443,11 +495,12 @@ def main():
mono = True
if about:
safe_print_json(about_jc(),
pretty=pretty,
env_colors=jc_colors,
mono=mono,
piped_out=piped_output(force_color))
safe_print_out(about_jc(),
pretty=pretty,
env_colors=jc_colors,
mono=mono,
piped_out=piped_output(force_color),
yaml=yaml_out)
sys.exit(0)
if help_me:
@@ -458,6 +511,14 @@ def main():
utils._safe_print(versiontext())
sys.exit(0)
if bash_comp:
utils._safe_print(bash_completion())
sys.exit(0)
if zsh_comp:
utils._safe_print(zsh_completion())
sys.exit(0)
# if magic syntax used, try to run the command and error if it's not found, etc.
magic_stdout, magic_stderr, magic_exit_code = None, None, 0
if run_command:
@@ -527,35 +588,45 @@ def main():
try:
# differentiate between regular and streaming parsers
# streaming
# streaming (only supports UTF-8 string data for now)
if _parser_is_streaming(parser):
result = parser.parse(sys.stdin,
raw=raw,
quiet=quiet,
ignore_exceptions=ignore_exceptions)
for line in result:
safe_print_json(line,
pretty=pretty,
env_colors=jc_colors,
mono=mono,
piped_out=piped_output(force_color),
flush=unbuffer)
safe_print_out(line,
pretty=pretty,
env_colors=jc_colors,
mono=mono,
piped_out=piped_output(force_color),
flush=unbuffer,
yaml=yaml_out)
sys.exit(combined_exit_code(magic_exit_code, 0))
# regular
# regular (supports binary and UTF-8 string data)
else:
data = magic_stdout or sys.stdin.read()
data = magic_stdout or sys.stdin.buffer.read()
# convert to UTF-8, if possible. Otherwise, leave as bytes
try:
if isinstance(data, bytes):
data = data.decode('utf-8')
except UnicodeDecodeError:
pass
result = parser.parse(data,
raw=raw,
quiet=quiet)
safe_print_json(result,
pretty=pretty,
env_colors=jc_colors,
mono=mono,
piped_out=piped_output(force_color),
flush=unbuffer)
safe_print_out(result,
pretty=pretty,
env_colors=jc_colors,
mono=mono,
piped_out=piped_output(force_color),
flush=unbuffer,
yaml=yaml_out)
sys.exit(combined_exit_code(magic_exit_code, 0))
@@ -570,14 +641,6 @@ def main():
])
sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
except json.JSONDecodeError:
if debug:
raise
utils.error_message(['There was an issue generating the JSON output.',
'For details use the -d or -dd option.'])
sys.exit(combined_exit_code(magic_exit_code, JC_ERROR_EXIT))
except Exception:
if debug:
raise

56
jc/cli_data.py Normal file
View File

@@ -0,0 +1,56 @@
"""jc - JSON Convert cli_data module"""
from typing import List, Dict
long_options_map: Dict[str, List[str]] = {
'--about': ['a', 'about jc'],
'--force-color': ['C', 'force color output even when using pipes (overrides -m)'],
'--debug': ['d', 'debug (double for verbose debug)'],
'--help': ['h', 'help (--help --parser_name for parser documentation)'],
'--monochrome': ['m', 'monochrome output'],
'--pretty': ['p', 'pretty print output'],
'--quiet': ['q', 'suppress warnings (double to ignore streaming errors)'],
'--raw': ['r', 'raw output'],
'--unbuffer': ['u', 'unbuffer output'],
'--version': ['v', 'version info'],
'--yaml-out': ['y', 'YAML output'],
'--bash-comp': ['B', 'gen Bash completion: jc -B > /etc/bash_completion.d/jc'],
'--zsh-comp': ['Z', 'gen Zsh completion: jc -Z > "${fpath[1]}/_jc"']
}
new_pygments_colors = {
'black': 'ansiblack',
'red': 'ansired',
'green': 'ansigreen',
'yellow': 'ansiyellow',
'blue': 'ansiblue',
'magenta': 'ansimagenta',
'cyan': 'ansicyan',
'gray': 'ansigray',
'brightblack': 'ansibrightblack',
'brightred': 'ansibrightred',
'brightgreen': 'ansibrightgreen',
'brightyellow': 'ansibrightyellow',
'brightblue': 'ansibrightblue',
'brightmagenta': 'ansibrightmagenta',
'brightcyan': 'ansibrightcyan',
'white': 'ansiwhite',
}
old_pygments_colors = {
'black': '#ansiblack',
'red': '#ansidarkred',
'green': '#ansidarkgreen',
'yellow': '#ansibrown',
'blue': '#ansidarkblue',
'magenta': '#ansipurple',
'cyan': '#ansiteal',
'gray': '#ansilightgray',
'brightblack': '#ansidarkgray',
'brightred': '#ansired',
'brightgreen': '#ansigreen',
'brightyellow': '#ansiyellow',
'brightblue': '#ansiblue',
'brightmagenta': '#ansifuchsia',
'brightcyan': '#ansiturquoise',
'white': '#ansiwhite',
}

View File

@@ -6,7 +6,7 @@ import importlib
from typing import Dict, List, Iterable, Union, Iterator
from jc import appdirs
__version__ = '1.18.8'
__version__ = '1.20.2'
parsers = [
'acpi',
@@ -16,6 +16,7 @@ parsers = [
'asciitable',
'asciitable-m',
'blkid',
'chage',
'cksum',
'crontab',
'crontab-u',
@@ -34,6 +35,8 @@ parsers = [
'free',
'fstab',
'git-log',
'git-log-s',
'gpg',
'group',
'gshadow',
'hash',
@@ -71,6 +74,7 @@ parsers = [
'ping-s',
'pip-list',
'pip-show',
'postconf',
'ps',
'route',
'rpm-qi',
@@ -89,6 +93,8 @@ parsers = [
'systeminfo',
'time',
'timedatectl',
'top',
'top-s',
'tracepath',
'traceroute',
'ufw',
@@ -103,6 +109,7 @@ parsers = [
'w',
'wc',
'who',
'x509-cert',
'xml',
'xrandr',
'yaml',

View File

@@ -3,7 +3,12 @@
This parser converts ASCII and Unicode text tables with single-line rows.
Column headers must be at least two spaces apart from each other and must
be unique.
be unique. For best results, column headers should be left-justified. If
column separators are present, then non-left-justified headers will be fixed
automatically.
Row separators are optional and are ignored. Each non-row-separator line is
considered a separate row in the table.
For example:
@@ -49,6 +54,9 @@ etc...
Headers (keys) are converted to snake-case. All values are returned as
strings, except empty strings, which are converted to None/null.
> Note: To preserve the case of the keys use the `-r` cli option or
> `raw=True` argument in `parse()`.
Usage (cli):
$ cat table.txt | jc --asciitable
@@ -117,7 +125,7 @@ from jc.parsers.universal import sparse_table_parse
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.0'
version = '1.2'
description = 'ASCII and Unicode table parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
@@ -139,6 +147,12 @@ def _process(proc_data: List[Dict]) -> List[Dict]:
List of Dictionaries. Structured to conform to the schema.
"""
# normalize keys: convert to lowercase
for item in proc_data:
for key in item.copy():
k_new = key.lower()
item[k_new] = item.pop(key)
return proc_data
@@ -222,18 +236,19 @@ def _is_separator(line: str) -> bool:
def _snake_case(line: str) -> str:
"""
replace spaces between words and special characters with an underscore
and set to lowercase
Replace spaces between words and special characters with an underscore.
Ignore the replacement char (�) used for header padding.
"""
line = re.sub(r'[^a-zA-Z0-9 ]', '_', line)
return re.sub(r'\b \b', '_', line).lower()
line = re.sub(r'[^a-zA-Z0-9 ]', '_', line) # special characters
line = re.sub(r'\b \b', '_', line) # spaces between words
return line
def _normalize_rows(table: str) -> List[str]:
"""
returns a List of row strings. Header is snake-cased
"""
result = []
result: List[str] = []
for line in table.splitlines():
# skip blank lines
if not line.strip():
@@ -243,7 +258,36 @@ def _normalize_rows(table: str) -> List[str]:
if _is_separator(line):
continue
# data row - remove column separators
# header or data row found - remove column separators
if not result: # this is the header row
# normalize the separator
line = line.replace('', '|')\
.replace('', '|')\
.replace('', '|')\
.replace('', '|')\
.replace('', '|')\
.replace('', '|')\
.replace('', '|')\
.replace('', '|')\
.replace('', '|')
# find the number of chars to pad in front of headers that are too
# far away from the separator. Replace spaces with unicode char: �
# we will remove this char from headers after sparse_table_parse
problem_header_pattern = re.compile(r'(?:\| )( +)([^|]+)')
problem_headers = problem_header_pattern.findall(line)
if problem_headers:
for p_header in problem_headers:
old_header = p_header[0] + p_header[1]
sub_chars = '' * len(p_header[0])
new_header = sub_chars + p_header[1]
line = line.replace(old_header, new_header)
line = line.replace('|', ' ')
result.append(_snake_case(line))
continue
# this is a data row
line = line.replace('|', ' ')\
.replace('', ' ')\
.replace('', ' ')\
@@ -256,7 +300,6 @@ def _normalize_rows(table: str) -> List[str]:
.replace('', ' ')
result.append(line)
result[0] = _snake_case(result[0])
return result
@@ -266,7 +309,8 @@ def _fixup_headers(table: List[Dict]) -> List[Dict]:
for row in table:
new_row = row.copy()
for k in row:
k_new = k
# remove replacement character
k_new = k.replace('', '')
# remove consecutive underscores
k_new = re.sub(r'__+', '_', k_new)
# remove trailing underscores

View File

@@ -24,6 +24,15 @@ Headers (keys) are converted to snake-case and newlines between multi-line
headers are joined with an underscore. All values are returned as strings,
except empty strings, which are converted to None/null.
> Note: To preserve the case of the keys use the `-r` cli option or
> `raw=True` argument in `parse()`.
> Note: table column separator characters (e.g. `|`) cannot be present
> inside the cell data. If detected, a warning message will be printed to
> `STDERR` and the line will be skipped. The warning message can be
> suppressed by using the `-q` command option or by setting `quiet=True` in
> `parse()`.
Usage (cli):
$ cat table.txt | jc --asciitable-m
@@ -101,7 +110,7 @@ from jc.exceptions import ParseError
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.0'
version = '1.2'
description = 'multi-line ASCII and Unicode table parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
@@ -123,6 +132,12 @@ def _process(proc_data: List[Dict]) -> List[Dict]:
List of Dictionaries. Structured to conform to the schema.
"""
# normalize keys: convert to lowercase
for item in proc_data:
for key in item.copy():
k_new = key.lower()
item[k_new] = item.pop(key)
return proc_data
@@ -227,12 +242,11 @@ def _is_separator(line: str) -> bool:
def _snake_case(line: str) -> str:
"""
replace spaces between words and special characters with an underscore
and set to lowercase
replace spaces between words and special characters with an underscore.
"""
# must include all column separator characters in regex
line = re.sub(r'[^a-zA-Z0-9 |│┃┆┇┊┋╎╏║]', '_', line)
return re.sub(r'\b \b', '_', line).lower()
return re.sub(r'\b \b', '_', line)
def _fixup_separators(line: str) -> str:
@@ -377,20 +391,27 @@ def _collapse_headers(table: List[List[str]]) -> List[str]:
return result
def _collapse_data(table: List[List[List[str]]]) -> List[List[str]]:
def _collapse_data(table: List[List[List[str]]], quiet=False) -> List[List[str]]:
"""combine data rows to return a simple list of lists"""
result: List[List[str]] = []
for row in table:
new_row: List[str] = []
for line in row:
if new_row:
for i, item in enumerate(line):
new_row[i] = (new_row[i] + '\n' + item).strip()
else:
new_row = line
for index, row in enumerate(table):
try:
new_row: List[str] = []
for line in row:
if new_row:
for i, item in enumerate(line):
new_row[i] = (new_row[i] + '\n' + item).strip()
else:
new_row = line
result.append(new_row)
result.append(new_row)
except IndexError:
if not quiet:
row_string = '\n'.join([' | '.join(l) for l in row])
jc.utils.warning_message(
[f'Possible table separator character found in row {index}: {row_string}. Skipping.']
)
return result
@@ -409,14 +430,14 @@ def _create_table_dict(header: List[str], data: List[List[str]]) -> List[Dict[st
return table_list_dict
def _parse_pretty(string: str) -> List[Dict[str, Optional[str]]]:
def _parse_pretty(string: str, quiet: bool =False) -> List[Dict[str, Optional[str]]]:
string_lines: List[str] = string.splitlines()
clean: List[Tuple[int, List[str]]] = _normalize_rows(string_lines)
raw_headers: List[List[str]] = _get_headers(clean)
raw_data: List[List[List[str]]] = _get_data(clean)
new_headers: List[str] = _collapse_headers(raw_headers)
new_data: List[List[str]] = _collapse_data(raw_data)
new_data: List[List[str]] = _collapse_data(raw_data, quiet)
final_table: List[Dict[str, Optional[str]]] = _create_table_dict(new_headers, new_data)
return final_table
@@ -452,7 +473,7 @@ def parse(
table_type = _table_sniff(data)
if table_type == 'pretty':
raw_output = _parse_pretty(data)
raw_output = _parse_pretty(data, quiet)
elif table_type == 'markdown':
raise ParseError('Only "pretty" tables supported with multiline. "markdown" table detected. Please try the "asciitable" parser.')
else:

View File

@@ -0,0 +1,47 @@
# coding: utf-8
from __future__ import unicode_literals, division, absolute_import, print_function
from .version import __version__, __version_info__
__all__ = [
'__version__',
'__version_info__',
'load_order',
]
def load_order():
"""
Returns a list of the module and sub-module names for asn1crypto in
dependency load order, for the sake of live reloading code
:return:
A list of unicode strings of module names, as they would appear in
sys.modules, ordered by which module should be reloaded first
"""
return [
'asn1crypto._errors',
'asn1crypto._int',
'asn1crypto._ordereddict',
'asn1crypto._teletex_codec',
'asn1crypto._types',
'asn1crypto._inet',
'asn1crypto._iri',
'asn1crypto.version',
'asn1crypto.pem',
'asn1crypto.util',
'asn1crypto.parser',
'asn1crypto.core',
'asn1crypto.algos',
'asn1crypto.keys',
'asn1crypto.x509',
'asn1crypto.crl',
'asn1crypto.csr',
'asn1crypto.ocsp',
'asn1crypto.cms',
'asn1crypto.pdf',
'asn1crypto.pkcs12',
'asn1crypto.tsp',
'asn1crypto',
]

View File

@@ -0,0 +1,54 @@
# coding: utf-8
"""
Exports the following items:
- unwrap()
- APIException()
"""
from __future__ import unicode_literals, division, absolute_import, print_function
import re
import textwrap
class APIException(Exception):
"""
An exception indicating an API has been removed from asn1crypto
"""
pass
def unwrap(string, *params):
"""
Takes a multi-line string and does the following:
- dedents
- converts newlines with text before and after into a single line
- strips leading and trailing whitespace
:param string:
The string to format
:param *params:
Params to interpolate into the string
:return:
The formatted string
"""
output = textwrap.dedent(string)
# Unwrap lines, taking into account bulleted lists, ordered lists and
# underlines consisting of = signs
if output.find('\n') != -1:
output = re.sub('(?<=\\S)\n(?=[^ \n\t\\d\\*\\-=])', ' ', output)
if params:
output = output % params
output = output.strip()
return output

View File

@@ -0,0 +1,170 @@
# coding: utf-8
from __future__ import unicode_literals, division, absolute_import, print_function
import socket
import struct
from ._errors import unwrap
from ._types import byte_cls, bytes_to_list, str_cls, type_name
def inet_ntop(address_family, packed_ip):
"""
Windows compatibility shim for socket.inet_ntop().
:param address_family:
socket.AF_INET for IPv4 or socket.AF_INET6 for IPv6
:param packed_ip:
A byte string of the network form of an IP address
:return:
A unicode string of the IP address
"""
if address_family not in set([socket.AF_INET, socket.AF_INET6]):
raise ValueError(unwrap(
'''
address_family must be socket.AF_INET (%s) or socket.AF_INET6 (%s),
not %s
''',
repr(socket.AF_INET),
repr(socket.AF_INET6),
repr(address_family)
))
if not isinstance(packed_ip, byte_cls):
raise TypeError(unwrap(
'''
packed_ip must be a byte string, not %s
''',
type_name(packed_ip)
))
required_len = 4 if address_family == socket.AF_INET else 16
if len(packed_ip) != required_len:
raise ValueError(unwrap(
'''
packed_ip must be %d bytes long - is %d
''',
required_len,
len(packed_ip)
))
if address_family == socket.AF_INET:
return '%d.%d.%d.%d' % tuple(bytes_to_list(packed_ip))
octets = struct.unpack(b'!HHHHHHHH', packed_ip)
runs_of_zero = {}
longest_run = 0
zero_index = None
for i, octet in enumerate(octets + (-1,)):
if octet != 0:
if zero_index is not None:
length = i - zero_index
if length not in runs_of_zero:
runs_of_zero[length] = zero_index
longest_run = max(longest_run, length)
zero_index = None
elif zero_index is None:
zero_index = i
hexed = [hex(o)[2:] for o in octets]
if longest_run < 2:
return ':'.join(hexed)
zero_start = runs_of_zero[longest_run]
zero_end = zero_start + longest_run
return ':'.join(hexed[:zero_start]) + '::' + ':'.join(hexed[zero_end:])
def inet_pton(address_family, ip_string):
"""
Windows compatibility shim for socket.inet_ntop().
:param address_family:
socket.AF_INET for IPv4 or socket.AF_INET6 for IPv6
:param ip_string:
A unicode string of an IP address
:return:
A byte string of the network form of the IP address
"""
if address_family not in set([socket.AF_INET, socket.AF_INET6]):
raise ValueError(unwrap(
'''
address_family must be socket.AF_INET (%s) or socket.AF_INET6 (%s),
not %s
''',
repr(socket.AF_INET),
repr(socket.AF_INET6),
repr(address_family)
))
if not isinstance(ip_string, str_cls):
raise TypeError(unwrap(
'''
ip_string must be a unicode string, not %s
''',
type_name(ip_string)
))
if address_family == socket.AF_INET:
octets = ip_string.split('.')
error = len(octets) != 4
if not error:
ints = []
for o in octets:
o = int(o)
if o > 255 or o < 0:
error = True
break
ints.append(o)
if error:
raise ValueError(unwrap(
'''
ip_string must be a dotted string with four integers in the
range of 0 to 255, got %s
''',
repr(ip_string)
))
return struct.pack(b'!BBBB', *ints)
error = False
omitted = ip_string.count('::')
if omitted > 1:
error = True
elif omitted == 0:
octets = ip_string.split(':')
error = len(octets) != 8
else:
begin, end = ip_string.split('::')
begin_octets = begin.split(':')
end_octets = end.split(':')
missing = 8 - len(begin_octets) - len(end_octets)
octets = begin_octets + (['0'] * missing) + end_octets
if not error:
ints = []
for o in octets:
o = int(o, 16)
if o > 65535 or o < 0:
error = True
break
ints.append(o)
return struct.pack(b'!HHHHHHHH', *ints)
raise ValueError(unwrap(
'''
ip_string must be a valid ipv6 string, got %s
''',
repr(ip_string)
))

View File

@@ -0,0 +1,22 @@
# coding: utf-8
from __future__ import unicode_literals, division, absolute_import, print_function
def fill_width(bytes_, width):
"""
Ensure a byte string representing a positive integer is a specific width
(in bytes)
:param bytes_:
The integer byte string
:param width:
The desired width as an integer
:return:
A byte string of the width specified
"""
while len(bytes_) < width:
bytes_ = b'\x00' + bytes_
return bytes_

View File

@@ -0,0 +1,291 @@
# coding: utf-8
"""
Functions to convert unicode IRIs into ASCII byte string URIs and back. Exports
the following items:
- iri_to_uri()
- uri_to_iri()
"""
from __future__ import unicode_literals, division, absolute_import, print_function
from encodings import idna # noqa
import codecs
import re
import sys
from ._errors import unwrap
from ._types import byte_cls, str_cls, type_name, bytes_to_list, int_types
if sys.version_info < (3,):
from urlparse import urlsplit, urlunsplit
from urllib import (
quote as urlquote,
unquote as unquote_to_bytes,
)
else:
from urllib.parse import (
quote as urlquote,
unquote_to_bytes,
urlsplit,
urlunsplit,
)
def iri_to_uri(value, normalize=False):
"""
Encodes a unicode IRI into an ASCII byte string URI
:param value:
A unicode string of an IRI
:param normalize:
A bool that controls URI normalization
:return:
A byte string of the ASCII-encoded URI
"""
if not isinstance(value, str_cls):
raise TypeError(unwrap(
'''
value must be a unicode string, not %s
''',
type_name(value)
))
scheme = None
# Python 2.6 doesn't split properly is the URL doesn't start with http:// or https://
if sys.version_info < (2, 7) and not value.startswith('http://') and not value.startswith('https://'):
real_prefix = None
prefix_match = re.match('^[^:]*://', value)
if prefix_match:
real_prefix = prefix_match.group(0)
value = 'http://' + value[len(real_prefix):]
parsed = urlsplit(value)
if real_prefix:
value = real_prefix + value[7:]
scheme = _urlquote(real_prefix[:-3])
else:
parsed = urlsplit(value)
if scheme is None:
scheme = _urlquote(parsed.scheme)
hostname = parsed.hostname
if hostname is not None:
hostname = hostname.encode('idna')
# RFC 3986 allows userinfo to contain sub-delims
username = _urlquote(parsed.username, safe='!$&\'()*+,;=')
password = _urlquote(parsed.password, safe='!$&\'()*+,;=')
port = parsed.port
if port is not None:
port = str_cls(port).encode('ascii')
netloc = b''
if username is not None:
netloc += username
if password:
netloc += b':' + password
netloc += b'@'
if hostname is not None:
netloc += hostname
if port is not None:
default_http = scheme == b'http' and port == b'80'
default_https = scheme == b'https' and port == b'443'
if not normalize or (not default_http and not default_https):
netloc += b':' + port
# RFC 3986 allows a path to contain sub-delims, plus "@" and ":"
path = _urlquote(parsed.path, safe='/!$&\'()*+,;=@:')
# RFC 3986 allows the query to contain sub-delims, plus "@", ":" , "/" and "?"
query = _urlquote(parsed.query, safe='/?!$&\'()*+,;=@:')
# RFC 3986 allows the fragment to contain sub-delims, plus "@", ":" , "/" and "?"
fragment = _urlquote(parsed.fragment, safe='/?!$&\'()*+,;=@:')
if normalize and query is None and fragment is None and path == b'/':
path = None
# Python 2.7 compat
if path is None:
path = ''
output = urlunsplit((scheme, netloc, path, query, fragment))
if isinstance(output, str_cls):
output = output.encode('latin1')
return output
def uri_to_iri(value):
"""
Converts an ASCII URI byte string into a unicode IRI
:param value:
An ASCII-encoded byte string of the URI
:return:
A unicode string of the IRI
"""
if not isinstance(value, byte_cls):
raise TypeError(unwrap(
'''
value must be a byte string, not %s
''',
type_name(value)
))
parsed = urlsplit(value)
scheme = parsed.scheme
if scheme is not None:
scheme = scheme.decode('ascii')
username = _urlunquote(parsed.username, remap=[':', '@'])
password = _urlunquote(parsed.password, remap=[':', '@'])
hostname = parsed.hostname
if hostname:
hostname = hostname.decode('idna')
port = parsed.port
if port and not isinstance(port, int_types):
port = port.decode('ascii')
netloc = ''
if username is not None:
netloc += username
if password:
netloc += ':' + password
netloc += '@'
if hostname is not None:
netloc += hostname
if port is not None:
netloc += ':' + str_cls(port)
path = _urlunquote(parsed.path, remap=['/'], preserve=True)
query = _urlunquote(parsed.query, remap=['&', '='], preserve=True)
fragment = _urlunquote(parsed.fragment)
return urlunsplit((scheme, netloc, path, query, fragment))
def _iri_utf8_errors_handler(exc):
"""
Error handler for decoding UTF-8 parts of a URI into an IRI. Leaves byte
sequences encoded in %XX format, but as part of a unicode string.
:param exc:
The UnicodeDecodeError exception
:return:
A 2-element tuple of (replacement unicode string, integer index to
resume at)
"""
bytes_as_ints = bytes_to_list(exc.object[exc.start:exc.end])
replacements = ['%%%02x' % num for num in bytes_as_ints]
return (''.join(replacements), exc.end)
codecs.register_error('iriutf8', _iri_utf8_errors_handler)
def _urlquote(string, safe=''):
"""
Quotes a unicode string for use in a URL
:param string:
A unicode string
:param safe:
A unicode string of character to not encode
:return:
None (if string is None) or an ASCII byte string of the quoted string
"""
if string is None or string == '':
return None
# Anything already hex quoted is pulled out of the URL and unquoted if
# possible
escapes = []
if re.search('%[0-9a-fA-F]{2}', string):
# Try to unquote any percent values, restoring them if they are not
# valid UTF-8. Also, requote any safe chars since encoded versions of
# those are functionally different than the unquoted ones.
def _try_unescape(match):
byte_string = unquote_to_bytes(match.group(0))
unicode_string = byte_string.decode('utf-8', 'iriutf8')
for safe_char in list(safe):
unicode_string = unicode_string.replace(safe_char, '%%%02x' % ord(safe_char))
return unicode_string
string = re.sub('(?:%[0-9a-fA-F]{2})+', _try_unescape, string)
# Once we have the minimal set of hex quoted values, removed them from
# the string so that they are not double quoted
def _extract_escape(match):
escapes.append(match.group(0).encode('ascii'))
return '\x00'
string = re.sub('%[0-9a-fA-F]{2}', _extract_escape, string)
output = urlquote(string.encode('utf-8'), safe=safe.encode('utf-8'))
if not isinstance(output, byte_cls):
output = output.encode('ascii')
# Restore the existing quoted values that we extracted
if len(escapes) > 0:
def _return_escape(_):
return escapes.pop(0)
output = re.sub(b'%00', _return_escape, output)
return output
def _urlunquote(byte_string, remap=None, preserve=None):
"""
Unquotes a URI portion from a byte string into unicode using UTF-8
:param byte_string:
A byte string of the data to unquote
:param remap:
A list of characters (as unicode) that should be re-mapped to a
%XX encoding. This is used when characters are not valid in part of a
URL.
:param preserve:
A bool - indicates that the chars to be remapped if they occur in
non-hex form, should be preserved. E.g. / for URL path.
:return:
A unicode string
"""
if byte_string is None:
return byte_string
if byte_string == b'':
return ''
if preserve:
replacements = ['\x1A', '\x1C', '\x1D', '\x1E', '\x1F']
preserve_unmap = {}
for char in remap:
replacement = replacements.pop(0)
preserve_unmap[replacement] = char
byte_string = byte_string.replace(char.encode('ascii'), replacement.encode('ascii'))
byte_string = unquote_to_bytes(byte_string)
if remap:
for char in remap:
byte_string = byte_string.replace(char.encode('ascii'), ('%%%02x' % ord(char)).encode('ascii'))
output = byte_string.decode('utf-8', 'iriutf8')
if preserve:
for replacement, original in preserve_unmap.items():
output = output.replace(replacement, original)
return output

View File

@@ -0,0 +1,135 @@
# Copyright (c) 2009 Raymond Hettinger
#
# Permission is hereby granted, free of charge, to any person
# obtaining a copy of this software and associated documentation files
# (the "Software"), to deal in the Software without restriction,
# including without limitation the rights to use, copy, modify, merge,
# publish, distribute, sublicense, and/or sell copies of the Software,
# and to permit persons to whom the Software is furnished to do so,
# subject to the following conditions:
#
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
# OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
# OTHER DEALINGS IN THE SOFTWARE.
import sys
if not sys.version_info < (2, 7):
from collections import OrderedDict
else:
from UserDict import DictMixin
class OrderedDict(dict, DictMixin):
def __init__(self, *args, **kwds):
if len(args) > 1:
raise TypeError('expected at most 1 arguments, got %d' % len(args))
try:
self.__end
except AttributeError:
self.clear()
self.update(*args, **kwds)
def clear(self):
self.__end = end = []
end += [None, end, end] # sentinel node for doubly linked list
self.__map = {} # key --> [key, prev, next]
dict.clear(self)
def __setitem__(self, key, value):
if key not in self:
end = self.__end
curr = end[1]
curr[2] = end[1] = self.__map[key] = [key, curr, end]
dict.__setitem__(self, key, value)
def __delitem__(self, key):
dict.__delitem__(self, key)
key, prev, next_ = self.__map.pop(key)
prev[2] = next_
next_[1] = prev
def __iter__(self):
end = self.__end
curr = end[2]
while curr is not end:
yield curr[0]
curr = curr[2]
def __reversed__(self):
end = self.__end
curr = end[1]
while curr is not end:
yield curr[0]
curr = curr[1]
def popitem(self, last=True):
if not self:
raise KeyError('dictionary is empty')
if last:
key = reversed(self).next()
else:
key = iter(self).next()
value = self.pop(key)
return key, value
def __reduce__(self):
items = [[k, self[k]] for k in self]
tmp = self.__map, self.__end
del self.__map, self.__end
inst_dict = vars(self).copy()
self.__map, self.__end = tmp
if inst_dict:
return (self.__class__, (items,), inst_dict)
return self.__class__, (items,)
def keys(self):
return list(self)
setdefault = DictMixin.setdefault
update = DictMixin.update
pop = DictMixin.pop
values = DictMixin.values
items = DictMixin.items
iterkeys = DictMixin.iterkeys
itervalues = DictMixin.itervalues
iteritems = DictMixin.iteritems
def __repr__(self):
if not self:
return '%s()' % (self.__class__.__name__,)
return '%s(%r)' % (self.__class__.__name__, self.items())
def copy(self):
return self.__class__(self)
@classmethod
def fromkeys(cls, iterable, value=None):
d = cls()
for key in iterable:
d[key] = value
return d
def __eq__(self, other):
if isinstance(other, OrderedDict):
if len(self) != len(other):
return False
for p, q in zip(self.items(), other.items()):
if p != q:
return False
return True
return dict.__eq__(self, other)
def __ne__(self, other):
return not self == other

View File

@@ -0,0 +1,331 @@
# coding: utf-8
"""
Implementation of the teletex T.61 codec. Exports the following items:
- register()
"""
from __future__ import unicode_literals, division, absolute_import, print_function
import codecs
class TeletexCodec(codecs.Codec):
def encode(self, input_, errors='strict'):
return codecs.charmap_encode(input_, errors, ENCODING_TABLE)
def decode(self, input_, errors='strict'):
return codecs.charmap_decode(input_, errors, DECODING_TABLE)
class TeletexIncrementalEncoder(codecs.IncrementalEncoder):
def encode(self, input_, final=False):
return codecs.charmap_encode(input_, self.errors, ENCODING_TABLE)[0]
class TeletexIncrementalDecoder(codecs.IncrementalDecoder):
def decode(self, input_, final=False):
return codecs.charmap_decode(input_, self.errors, DECODING_TABLE)[0]
class TeletexStreamWriter(TeletexCodec, codecs.StreamWriter):
pass
class TeletexStreamReader(TeletexCodec, codecs.StreamReader):
pass
def teletex_search_function(name):
"""
Search function for teletex codec that is passed to codecs.register()
"""
if name != 'teletex':
return None
return codecs.CodecInfo(
name='teletex',
encode=TeletexCodec().encode,
decode=TeletexCodec().decode,
incrementalencoder=TeletexIncrementalEncoder,
incrementaldecoder=TeletexIncrementalDecoder,
streamreader=TeletexStreamReader,
streamwriter=TeletexStreamWriter,
)
def register():
"""
Registers the teletex codec
"""
codecs.register(teletex_search_function)
# http://en.wikipedia.org/wiki/ITU_T.61
DECODING_TABLE = (
'\u0000'
'\u0001'
'\u0002'
'\u0003'
'\u0004'
'\u0005'
'\u0006'
'\u0007'
'\u0008'
'\u0009'
'\u000A'
'\u000B'
'\u000C'
'\u000D'
'\u000E'
'\u000F'
'\u0010'
'\u0011'
'\u0012'
'\u0013'
'\u0014'
'\u0015'
'\u0016'
'\u0017'
'\u0018'
'\u0019'
'\u001A'
'\u001B'
'\u001C'
'\u001D'
'\u001E'
'\u001F'
'\u0020'
'\u0021'
'\u0022'
'\ufffe'
'\ufffe'
'\u0025'
'\u0026'
'\u0027'
'\u0028'
'\u0029'
'\u002A'
'\u002B'
'\u002C'
'\u002D'
'\u002E'
'\u002F'
'\u0030'
'\u0031'
'\u0032'
'\u0033'
'\u0034'
'\u0035'
'\u0036'
'\u0037'
'\u0038'
'\u0039'
'\u003A'
'\u003B'
'\u003C'
'\u003D'
'\u003E'
'\u003F'
'\u0040'
'\u0041'
'\u0042'
'\u0043'
'\u0044'
'\u0045'
'\u0046'
'\u0047'
'\u0048'
'\u0049'
'\u004A'
'\u004B'
'\u004C'
'\u004D'
'\u004E'
'\u004F'
'\u0050'
'\u0051'
'\u0052'
'\u0053'
'\u0054'
'\u0055'
'\u0056'
'\u0057'
'\u0058'
'\u0059'
'\u005A'
'\u005B'
'\ufffe'
'\u005D'
'\ufffe'
'\u005F'
'\ufffe'
'\u0061'
'\u0062'
'\u0063'
'\u0064'
'\u0065'
'\u0066'
'\u0067'
'\u0068'
'\u0069'
'\u006A'
'\u006B'
'\u006C'
'\u006D'
'\u006E'
'\u006F'
'\u0070'
'\u0071'
'\u0072'
'\u0073'
'\u0074'
'\u0075'
'\u0076'
'\u0077'
'\u0078'
'\u0079'
'\u007A'
'\ufffe'
'\u007C'
'\ufffe'
'\ufffe'
'\u007F'
'\u0080'
'\u0081'
'\u0082'
'\u0083'
'\u0084'
'\u0085'
'\u0086'
'\u0087'
'\u0088'
'\u0089'
'\u008A'
'\u008B'
'\u008C'
'\u008D'
'\u008E'
'\u008F'
'\u0090'
'\u0091'
'\u0092'
'\u0093'
'\u0094'
'\u0095'
'\u0096'
'\u0097'
'\u0098'
'\u0099'
'\u009A'
'\u009B'
'\u009C'
'\u009D'
'\u009E'
'\u009F'
'\u00A0'
'\u00A1'
'\u00A2'
'\u00A3'
'\u0024'
'\u00A5'
'\u0023'
'\u00A7'
'\u00A4'
'\ufffe'
'\ufffe'
'\u00AB'
'\ufffe'
'\ufffe'
'\ufffe'
'\ufffe'
'\u00B0'
'\u00B1'
'\u00B2'
'\u00B3'
'\u00D7'
'\u00B5'
'\u00B6'
'\u00B7'
'\u00F7'
'\ufffe'
'\ufffe'
'\u00BB'
'\u00BC'
'\u00BD'
'\u00BE'
'\u00BF'
'\ufffe'
'\u0300'
'\u0301'
'\u0302'
'\u0303'
'\u0304'
'\u0306'
'\u0307'
'\u0308'
'\ufffe'
'\u030A'
'\u0327'
'\u0332'
'\u030B'
'\u0328'
'\u030C'
'\ufffe'
'\ufffe'
'\ufffe'
'\ufffe'
'\ufffe'
'\ufffe'
'\ufffe'
'\ufffe'
'\ufffe'
'\ufffe'
'\ufffe'
'\ufffe'
'\ufffe'
'\ufffe'
'\ufffe'
'\ufffe'
'\u2126'
'\u00C6'
'\u00D0'
'\u00AA'
'\u0126'
'\ufffe'
'\u0132'
'\u013F'
'\u0141'
'\u00D8'
'\u0152'
'\u00BA'
'\u00DE'
'\u0166'
'\u014A'
'\u0149'
'\u0138'
'\u00E6'
'\u0111'
'\u00F0'
'\u0127'
'\u0131'
'\u0133'
'\u0140'
'\u0142'
'\u00F8'
'\u0153'
'\u00DF'
'\u00FE'
'\u0167'
'\u014B'
'\ufffe'
)
ENCODING_TABLE = codecs.charmap_build(DECODING_TABLE)

View File

@@ -0,0 +1,46 @@
# coding: utf-8
from __future__ import unicode_literals, division, absolute_import, print_function
import inspect
import sys
if sys.version_info < (3,):
str_cls = unicode # noqa
byte_cls = str
int_types = (int, long) # noqa
def bytes_to_list(byte_string):
return [ord(b) for b in byte_string]
chr_cls = chr
else:
str_cls = str
byte_cls = bytes
int_types = int
bytes_to_list = list
def chr_cls(num):
return bytes([num])
def type_name(value):
"""
Returns a user-readable name for the type of an object
:param value:
A value to get the type name of
:return:
A unicode string of the object's type name
"""
if inspect.isclass(value):
cls = value
else:
cls = value.__class__
if cls.__module__ in set(['builtins', '__builtin__']):
return cls.__name__
return '%s.%s' % (cls.__module__, cls.__name__)

File diff suppressed because it is too large Load Diff

1003
jc/parsers/asn1crypto/cms.py Normal file

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,536 @@
# coding: utf-8
"""
ASN.1 type classes for certificate revocation lists (CRL). Exports the
following items:
- CertificateList()
Other type classes are defined that help compose the types listed above.
"""
from __future__ import unicode_literals, division, absolute_import, print_function
import hashlib
from .algos import SignedDigestAlgorithm
from .core import (
Boolean,
Enumerated,
GeneralizedTime,
Integer,
ObjectIdentifier,
OctetBitString,
ParsableOctetString,
Sequence,
SequenceOf,
)
from .x509 import (
AuthorityInfoAccessSyntax,
AuthorityKeyIdentifier,
CRLDistributionPoints,
DistributionPointName,
GeneralNames,
Name,
ReasonFlags,
Time,
)
# The structures in this file are taken from https://tools.ietf.org/html/rfc5280
class Version(Integer):
_map = {
0: 'v1',
1: 'v2',
2: 'v3',
}
class IssuingDistributionPoint(Sequence):
_fields = [
('distribution_point', DistributionPointName, {'explicit': 0, 'optional': True}),
('only_contains_user_certs', Boolean, {'implicit': 1, 'default': False}),
('only_contains_ca_certs', Boolean, {'implicit': 2, 'default': False}),
('only_some_reasons', ReasonFlags, {'implicit': 3, 'optional': True}),
('indirect_crl', Boolean, {'implicit': 4, 'default': False}),
('only_contains_attribute_certs', Boolean, {'implicit': 5, 'default': False}),
]
class TBSCertListExtensionId(ObjectIdentifier):
_map = {
'2.5.29.18': 'issuer_alt_name',
'2.5.29.20': 'crl_number',
'2.5.29.27': 'delta_crl_indicator',
'2.5.29.28': 'issuing_distribution_point',
'2.5.29.35': 'authority_key_identifier',
'2.5.29.46': 'freshest_crl',
'1.3.6.1.5.5.7.1.1': 'authority_information_access',
}
class TBSCertListExtension(Sequence):
_fields = [
('extn_id', TBSCertListExtensionId),
('critical', Boolean, {'default': False}),
('extn_value', ParsableOctetString),
]
_oid_pair = ('extn_id', 'extn_value')
_oid_specs = {
'issuer_alt_name': GeneralNames,
'crl_number': Integer,
'delta_crl_indicator': Integer,
'issuing_distribution_point': IssuingDistributionPoint,
'authority_key_identifier': AuthorityKeyIdentifier,
'freshest_crl': CRLDistributionPoints,
'authority_information_access': AuthorityInfoAccessSyntax,
}
class TBSCertListExtensions(SequenceOf):
_child_spec = TBSCertListExtension
class CRLReason(Enumerated):
_map = {
0: 'unspecified',
1: 'key_compromise',
2: 'ca_compromise',
3: 'affiliation_changed',
4: 'superseded',
5: 'cessation_of_operation',
6: 'certificate_hold',
8: 'remove_from_crl',
9: 'privilege_withdrawn',
10: 'aa_compromise',
}
@property
def human_friendly(self):
"""
:return:
A unicode string with revocation description that is suitable to
show to end-users. Starts with a lower case letter and phrased in
such a way that it makes sense after the phrase "because of" or
"due to".
"""
return {
'unspecified': 'an unspecified reason',
'key_compromise': 'a compromised key',
'ca_compromise': 'the CA being compromised',
'affiliation_changed': 'an affiliation change',
'superseded': 'certificate supersession',
'cessation_of_operation': 'a cessation of operation',
'certificate_hold': 'a certificate hold',
'remove_from_crl': 'removal from the CRL',
'privilege_withdrawn': 'privilege withdrawl',
'aa_compromise': 'the AA being compromised',
}[self.native]
class CRLEntryExtensionId(ObjectIdentifier):
_map = {
'2.5.29.21': 'crl_reason',
'2.5.29.23': 'hold_instruction_code',
'2.5.29.24': 'invalidity_date',
'2.5.29.29': 'certificate_issuer',
}
class CRLEntryExtension(Sequence):
_fields = [
('extn_id', CRLEntryExtensionId),
('critical', Boolean, {'default': False}),
('extn_value', ParsableOctetString),
]
_oid_pair = ('extn_id', 'extn_value')
_oid_specs = {
'crl_reason': CRLReason,
'hold_instruction_code': ObjectIdentifier,
'invalidity_date': GeneralizedTime,
'certificate_issuer': GeneralNames,
}
class CRLEntryExtensions(SequenceOf):
_child_spec = CRLEntryExtension
class RevokedCertificate(Sequence):
_fields = [
('user_certificate', Integer),
('revocation_date', Time),
('crl_entry_extensions', CRLEntryExtensions, {'optional': True}),
]
_processed_extensions = False
_critical_extensions = None
_crl_reason_value = None
_invalidity_date_value = None
_certificate_issuer_value = None
_issuer_name = False
def _set_extensions(self):
"""
Sets common named extensions to private attributes and creates a list
of critical extensions
"""
self._critical_extensions = set()
for extension in self['crl_entry_extensions']:
name = extension['extn_id'].native
attribute_name = '_%s_value' % name
if hasattr(self, attribute_name):
setattr(self, attribute_name, extension['extn_value'].parsed)
if extension['critical'].native:
self._critical_extensions.add(name)
self._processed_extensions = True
@property
def critical_extensions(self):
"""
Returns a set of the names (or OID if not a known extension) of the
extensions marked as critical
:return:
A set of unicode strings
"""
if not self._processed_extensions:
self._set_extensions()
return self._critical_extensions
@property
def crl_reason_value(self):
"""
This extension indicates the reason that a certificate was revoked.
:return:
None or a CRLReason object
"""
if self._processed_extensions is False:
self._set_extensions()
return self._crl_reason_value
@property
def invalidity_date_value(self):
"""
This extension indicates the suspected date/time the private key was
compromised or the certificate became invalid. This would usually be
before the revocation date, which is when the CA processed the
revocation.
:return:
None or a GeneralizedTime object
"""
if self._processed_extensions is False:
self._set_extensions()
return self._invalidity_date_value
@property
def certificate_issuer_value(self):
"""
This extension indicates the issuer of the certificate in question,
and is used in indirect CRLs. CRL entries without this extension are
for certificates issued from the last seen issuer.
:return:
None or an x509.GeneralNames object
"""
if self._processed_extensions is False:
self._set_extensions()
return self._certificate_issuer_value
@property
def issuer_name(self):
"""
:return:
None, or an asn1crypto.x509.Name object for the issuer of the cert
"""
if self._issuer_name is False:
self._issuer_name = None
if self.certificate_issuer_value:
for general_name in self.certificate_issuer_value:
if general_name.name == 'directory_name':
self._issuer_name = general_name.chosen
break
return self._issuer_name
class RevokedCertificates(SequenceOf):
_child_spec = RevokedCertificate
class TbsCertList(Sequence):
_fields = [
('version', Version, {'optional': True}),
('signature', SignedDigestAlgorithm),
('issuer', Name),
('this_update', Time),
('next_update', Time, {'optional': True}),
('revoked_certificates', RevokedCertificates, {'optional': True}),
('crl_extensions', TBSCertListExtensions, {'explicit': 0, 'optional': True}),
]
class CertificateList(Sequence):
_fields = [
('tbs_cert_list', TbsCertList),
('signature_algorithm', SignedDigestAlgorithm),
('signature', OctetBitString),
]
_processed_extensions = False
_critical_extensions = None
_issuer_alt_name_value = None
_crl_number_value = None
_delta_crl_indicator_value = None
_issuing_distribution_point_value = None
_authority_key_identifier_value = None
_freshest_crl_value = None
_authority_information_access_value = None
_issuer_cert_urls = None
_delta_crl_distribution_points = None
_sha1 = None
_sha256 = None
def _set_extensions(self):
"""
Sets common named extensions to private attributes and creates a list
of critical extensions
"""
self._critical_extensions = set()
for extension in self['tbs_cert_list']['crl_extensions']:
name = extension['extn_id'].native
attribute_name = '_%s_value' % name
if hasattr(self, attribute_name):
setattr(self, attribute_name, extension['extn_value'].parsed)
if extension['critical'].native:
self._critical_extensions.add(name)
self._processed_extensions = True
@property
def critical_extensions(self):
"""
Returns a set of the names (or OID if not a known extension) of the
extensions marked as critical
:return:
A set of unicode strings
"""
if not self._processed_extensions:
self._set_extensions()
return self._critical_extensions
@property
def issuer_alt_name_value(self):
"""
This extension allows associating one or more alternative names with
the issuer of the CRL.
:return:
None or an x509.GeneralNames object
"""
if self._processed_extensions is False:
self._set_extensions()
return self._issuer_alt_name_value
@property
def crl_number_value(self):
"""
This extension adds a monotonically increasing number to the CRL and is
used to distinguish different versions of the CRL.
:return:
None or an Integer object
"""
if self._processed_extensions is False:
self._set_extensions()
return self._crl_number_value
@property
def delta_crl_indicator_value(self):
"""
This extension indicates a CRL is a delta CRL, and contains the CRL
number of the base CRL that it is a delta from.
:return:
None or an Integer object
"""
if self._processed_extensions is False:
self._set_extensions()
return self._delta_crl_indicator_value
@property
def issuing_distribution_point_value(self):
"""
This extension includes information about what types of revocations
and certificates are part of the CRL.
:return:
None or an IssuingDistributionPoint object
"""
if self._processed_extensions is False:
self._set_extensions()
return self._issuing_distribution_point_value
@property
def authority_key_identifier_value(self):
"""
This extension helps in identifying the public key with which to
validate the authenticity of the CRL.
:return:
None or an AuthorityKeyIdentifier object
"""
if self._processed_extensions is False:
self._set_extensions()
return self._authority_key_identifier_value
@property
def freshest_crl_value(self):
"""
This extension is used in complete CRLs to indicate where a delta CRL
may be located.
:return:
None or a CRLDistributionPoints object
"""
if self._processed_extensions is False:
self._set_extensions()
return self._freshest_crl_value
@property
def authority_information_access_value(self):
"""
This extension is used to provide a URL with which to download the
certificate used to sign this CRL.
:return:
None or an AuthorityInfoAccessSyntax object
"""
if self._processed_extensions is False:
self._set_extensions()
return self._authority_information_access_value
@property
def issuer(self):
"""
:return:
An asn1crypto.x509.Name object for the issuer of the CRL
"""
return self['tbs_cert_list']['issuer']
@property
def authority_key_identifier(self):
"""
:return:
None or a byte string of the key_identifier from the authority key
identifier extension
"""
if not self.authority_key_identifier_value:
return None
return self.authority_key_identifier_value['key_identifier'].native
@property
def issuer_cert_urls(self):
"""
:return:
A list of unicode strings that are URLs that should contain either
an individual DER-encoded X.509 certificate, or a DER-encoded CMS
message containing multiple certificates
"""
if self._issuer_cert_urls is None:
self._issuer_cert_urls = []
if self.authority_information_access_value:
for entry in self.authority_information_access_value:
if entry['access_method'].native == 'ca_issuers':
location = entry['access_location']
if location.name != 'uniform_resource_identifier':
continue
url = location.native
if url.lower()[0:7] == 'http://':
self._issuer_cert_urls.append(url)
return self._issuer_cert_urls
@property
def delta_crl_distribution_points(self):
"""
Returns delta CRL URLs - only applies to complete CRLs
:return:
A list of zero or more DistributionPoint objects
"""
if self._delta_crl_distribution_points is None:
self._delta_crl_distribution_points = []
if self.freshest_crl_value is not None:
for distribution_point in self.freshest_crl_value:
distribution_point_name = distribution_point['distribution_point']
# RFC 5280 indicates conforming CA should not use the relative form
if distribution_point_name.name == 'name_relative_to_crl_issuer':
continue
# This library is currently only concerned with HTTP-based CRLs
for general_name in distribution_point_name.chosen:
if general_name.name == 'uniform_resource_identifier':
self._delta_crl_distribution_points.append(distribution_point)
return self._delta_crl_distribution_points
@property
def signature(self):
"""
:return:
A byte string of the signature
"""
return self['signature'].native
@property
def sha1(self):
"""
:return:
The SHA1 hash of the DER-encoded bytes of this certificate list
"""
if self._sha1 is None:
self._sha1 = hashlib.sha1(self.dump()).digest()
return self._sha1
@property
def sha256(self):
"""
:return:
The SHA-256 hash of the DER-encoded bytes of this certificate list
"""
if self._sha256 is None:
self._sha256 = hashlib.sha256(self.dump()).digest()
return self._sha256

View File

@@ -0,0 +1,133 @@
# coding: utf-8
"""
ASN.1 type classes for certificate signing requests (CSR). Exports the
following items:
- CertificationRequest()
Other type classes are defined that help compose the types listed above.
"""
from __future__ import unicode_literals, division, absolute_import, print_function
from .algos import SignedDigestAlgorithm
from .core import (
Any,
BitString,
BMPString,
Integer,
ObjectIdentifier,
OctetBitString,
Sequence,
SetOf,
UTF8String
)
from .keys import PublicKeyInfo
from .x509 import DirectoryString, Extensions, Name
# The structures in this file are taken from https://tools.ietf.org/html/rfc2986
# and https://tools.ietf.org/html/rfc2985
class Version(Integer):
_map = {
0: 'v1',
}
class CSRAttributeType(ObjectIdentifier):
_map = {
'1.2.840.113549.1.9.7': 'challenge_password',
'1.2.840.113549.1.9.9': 'extended_certificate_attributes',
'1.2.840.113549.1.9.14': 'extension_request',
# https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-wcce/a5eaae36-e9f3-4dc5-a687-bfa7115954f1
'1.3.6.1.4.1.311.13.2.2': 'microsoft_enrollment_csp_provider',
# https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-wcce/7c677cba-030d-48be-ba2b-01e407705f34
'1.3.6.1.4.1.311.13.2.3': 'microsoft_os_version',
# https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-wcce/64e5ff6d-c6dd-4578-92f7-b3d895f9b9c7
'1.3.6.1.4.1.311.21.20': 'microsoft_request_client_info',
}
class SetOfDirectoryString(SetOf):
_child_spec = DirectoryString
class Attribute(Sequence):
_fields = [
('type', ObjectIdentifier),
('values', SetOf, {'spec': Any}),
]
class SetOfAttributes(SetOf):
_child_spec = Attribute
class SetOfExtensions(SetOf):
_child_spec = Extensions
class MicrosoftEnrollmentCSProvider(Sequence):
_fields = [
('keyspec', Integer),
('cspname', BMPString), # cryptographic service provider name
('signature', BitString),
]
class SetOfMicrosoftEnrollmentCSProvider(SetOf):
_child_spec = MicrosoftEnrollmentCSProvider
class MicrosoftRequestClientInfo(Sequence):
_fields = [
('clientid', Integer),
('machinename', UTF8String),
('username', UTF8String),
('processname', UTF8String),
]
class SetOfMicrosoftRequestClientInfo(SetOf):
_child_spec = MicrosoftRequestClientInfo
class CRIAttribute(Sequence):
_fields = [
('type', CSRAttributeType),
('values', Any),
]
_oid_pair = ('type', 'values')
_oid_specs = {
'challenge_password': SetOfDirectoryString,
'extended_certificate_attributes': SetOfAttributes,
'extension_request': SetOfExtensions,
'microsoft_enrollment_csp_provider': SetOfMicrosoftEnrollmentCSProvider,
'microsoft_os_version': SetOfDirectoryString,
'microsoft_request_client_info': SetOfMicrosoftRequestClientInfo,
}
class CRIAttributes(SetOf):
_child_spec = CRIAttribute
class CertificationRequestInfo(Sequence):
_fields = [
('version', Version),
('subject', Name),
('subject_pk_info', PublicKeyInfo),
('attributes', CRIAttributes, {'implicit': 0, 'optional': True}),
]
class CertificationRequest(Sequence):
_fields = [
('certification_request_info', CertificationRequestInfo),
('signature_algorithm', SignedDigestAlgorithm),
('signature', OctetBitString),
]

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,703 @@
# coding: utf-8
"""
ASN.1 type classes for the online certificate status protocol (OCSP). Exports
the following items:
- OCSPRequest()
- OCSPResponse()
Other type classes are defined that help compose the types listed above.
"""
from __future__ import unicode_literals, division, absolute_import, print_function
from ._errors import unwrap
from .algos import DigestAlgorithm, SignedDigestAlgorithm
from .core import (
Boolean,
Choice,
Enumerated,
GeneralizedTime,
IA5String,
Integer,
Null,
ObjectIdentifier,
OctetBitString,
OctetString,
ParsableOctetString,
Sequence,
SequenceOf,
)
from .crl import AuthorityInfoAccessSyntax, CRLReason
from .keys import PublicKeyAlgorithm
from .x509 import Certificate, GeneralName, GeneralNames, Name
# The structures in this file are taken from https://tools.ietf.org/html/rfc6960
class Version(Integer):
_map = {
0: 'v1'
}
class CertId(Sequence):
_fields = [
('hash_algorithm', DigestAlgorithm),
('issuer_name_hash', OctetString),
('issuer_key_hash', OctetString),
('serial_number', Integer),
]
class ServiceLocator(Sequence):
_fields = [
('issuer', Name),
('locator', AuthorityInfoAccessSyntax),
]
class RequestExtensionId(ObjectIdentifier):
_map = {
'1.3.6.1.5.5.7.48.1.7': 'service_locator',
}
class RequestExtension(Sequence):
_fields = [
('extn_id', RequestExtensionId),
('critical', Boolean, {'default': False}),
('extn_value', ParsableOctetString),
]
_oid_pair = ('extn_id', 'extn_value')
_oid_specs = {
'service_locator': ServiceLocator,
}
class RequestExtensions(SequenceOf):
_child_spec = RequestExtension
class Request(Sequence):
_fields = [
('req_cert', CertId),
('single_request_extensions', RequestExtensions, {'explicit': 0, 'optional': True}),
]
_processed_extensions = False
_critical_extensions = None
_service_locator_value = None
def _set_extensions(self):
"""
Sets common named extensions to private attributes and creates a list
of critical extensions
"""
self._critical_extensions = set()
for extension in self['single_request_extensions']:
name = extension['extn_id'].native
attribute_name = '_%s_value' % name
if hasattr(self, attribute_name):
setattr(self, attribute_name, extension['extn_value'].parsed)
if extension['critical'].native:
self._critical_extensions.add(name)
self._processed_extensions = True
@property
def critical_extensions(self):
"""
Returns a set of the names (or OID if not a known extension) of the
extensions marked as critical
:return:
A set of unicode strings
"""
if not self._processed_extensions:
self._set_extensions()
return self._critical_extensions
@property
def service_locator_value(self):
"""
This extension is used when communicating with an OCSP responder that
acts as a proxy for OCSP requests
:return:
None or a ServiceLocator object
"""
if self._processed_extensions is False:
self._set_extensions()
return self._service_locator_value
class Requests(SequenceOf):
_child_spec = Request
class ResponseType(ObjectIdentifier):
_map = {
'1.3.6.1.5.5.7.48.1.1': 'basic_ocsp_response',
}
class AcceptableResponses(SequenceOf):
_child_spec = ResponseType
class PreferredSignatureAlgorithm(Sequence):
_fields = [
('sig_identifier', SignedDigestAlgorithm),
('cert_identifier', PublicKeyAlgorithm, {'optional': True}),
]
class PreferredSignatureAlgorithms(SequenceOf):
_child_spec = PreferredSignatureAlgorithm
class TBSRequestExtensionId(ObjectIdentifier):
_map = {
'1.3.6.1.5.5.7.48.1.2': 'nonce',
'1.3.6.1.5.5.7.48.1.4': 'acceptable_responses',
'1.3.6.1.5.5.7.48.1.8': 'preferred_signature_algorithms',
}
class TBSRequestExtension(Sequence):
_fields = [
('extn_id', TBSRequestExtensionId),
('critical', Boolean, {'default': False}),
('extn_value', ParsableOctetString),
]
_oid_pair = ('extn_id', 'extn_value')
_oid_specs = {
'nonce': OctetString,
'acceptable_responses': AcceptableResponses,
'preferred_signature_algorithms': PreferredSignatureAlgorithms,
}
class TBSRequestExtensions(SequenceOf):
_child_spec = TBSRequestExtension
class TBSRequest(Sequence):
_fields = [
('version', Version, {'explicit': 0, 'default': 'v1'}),
('requestor_name', GeneralName, {'explicit': 1, 'optional': True}),
('request_list', Requests),
('request_extensions', TBSRequestExtensions, {'explicit': 2, 'optional': True}),
]
class Certificates(SequenceOf):
_child_spec = Certificate
class Signature(Sequence):
_fields = [
('signature_algorithm', SignedDigestAlgorithm),
('signature', OctetBitString),
('certs', Certificates, {'explicit': 0, 'optional': True}),
]
class OCSPRequest(Sequence):
_fields = [
('tbs_request', TBSRequest),
('optional_signature', Signature, {'explicit': 0, 'optional': True}),
]
_processed_extensions = False
_critical_extensions = None
_nonce_value = None
_acceptable_responses_value = None
_preferred_signature_algorithms_value = None
def _set_extensions(self):
"""
Sets common named extensions to private attributes and creates a list
of critical extensions
"""
self._critical_extensions = set()
for extension in self['tbs_request']['request_extensions']:
name = extension['extn_id'].native
attribute_name = '_%s_value' % name
if hasattr(self, attribute_name):
setattr(self, attribute_name, extension['extn_value'].parsed)
if extension['critical'].native:
self._critical_extensions.add(name)
self._processed_extensions = True
@property
def critical_extensions(self):
"""
Returns a set of the names (or OID if not a known extension) of the
extensions marked as critical
:return:
A set of unicode strings
"""
if not self._processed_extensions:
self._set_extensions()
return self._critical_extensions
@property
def nonce_value(self):
"""
This extension is used to prevent replay attacks by including a unique,
random value with each request/response pair
:return:
None or an OctetString object
"""
if self._processed_extensions is False:
self._set_extensions()
return self._nonce_value
@property
def acceptable_responses_value(self):
"""
This extension is used to allow the client and server to communicate
with alternative response formats other than just basic_ocsp_response,
although no other formats are defined in the standard.
:return:
None or an AcceptableResponses object
"""
if self._processed_extensions is False:
self._set_extensions()
return self._acceptable_responses_value
@property
def preferred_signature_algorithms_value(self):
"""
This extension is used by the client to define what signature algorithms
are preferred, including both the hash algorithm and the public key
algorithm, with a level of detail down to even the public key algorithm
parameters, such as curve name.
:return:
None or a PreferredSignatureAlgorithms object
"""
if self._processed_extensions is False:
self._set_extensions()
return self._preferred_signature_algorithms_value
class OCSPResponseStatus(Enumerated):
_map = {
0: 'successful',
1: 'malformed_request',
2: 'internal_error',
3: 'try_later',
5: 'sign_required',
6: 'unauthorized',
}
class ResponderId(Choice):
_alternatives = [
('by_name', Name, {'explicit': 1}),
('by_key', OctetString, {'explicit': 2}),
]
# Custom class to return a meaningful .native attribute from CertStatus()
class StatusGood(Null):
def set(self, value):
"""
Sets the value of the object
:param value:
None or 'good'
"""
if value is not None and value != 'good' and not isinstance(value, Null):
raise ValueError(unwrap(
'''
value must be one of None, "good", not %s
''',
repr(value)
))
self.contents = b''
@property
def native(self):
return 'good'
# Custom class to return a meaningful .native attribute from CertStatus()
class StatusUnknown(Null):
def set(self, value):
"""
Sets the value of the object
:param value:
None or 'unknown'
"""
if value is not None and value != 'unknown' and not isinstance(value, Null):
raise ValueError(unwrap(
'''
value must be one of None, "unknown", not %s
''',
repr(value)
))
self.contents = b''
@property
def native(self):
return 'unknown'
class RevokedInfo(Sequence):
_fields = [
('revocation_time', GeneralizedTime),
('revocation_reason', CRLReason, {'explicit': 0, 'optional': True}),
]
class CertStatus(Choice):
_alternatives = [
('good', StatusGood, {'implicit': 0}),
('revoked', RevokedInfo, {'implicit': 1}),
('unknown', StatusUnknown, {'implicit': 2}),
]
class CrlId(Sequence):
_fields = [
('crl_url', IA5String, {'explicit': 0, 'optional': True}),
('crl_num', Integer, {'explicit': 1, 'optional': True}),
('crl_time', GeneralizedTime, {'explicit': 2, 'optional': True}),
]
class SingleResponseExtensionId(ObjectIdentifier):
_map = {
'1.3.6.1.5.5.7.48.1.3': 'crl',
'1.3.6.1.5.5.7.48.1.6': 'archive_cutoff',
# These are CRLEntryExtension values from
# https://tools.ietf.org/html/rfc5280
'2.5.29.21': 'crl_reason',
'2.5.29.24': 'invalidity_date',
'2.5.29.29': 'certificate_issuer',
# https://tools.ietf.org/html/rfc6962.html#page-13
'1.3.6.1.4.1.11129.2.4.5': 'signed_certificate_timestamp_list',
}
class SingleResponseExtension(Sequence):
_fields = [
('extn_id', SingleResponseExtensionId),
('critical', Boolean, {'default': False}),
('extn_value', ParsableOctetString),
]
_oid_pair = ('extn_id', 'extn_value')
_oid_specs = {
'crl': CrlId,
'archive_cutoff': GeneralizedTime,
'crl_reason': CRLReason,
'invalidity_date': GeneralizedTime,
'certificate_issuer': GeneralNames,
'signed_certificate_timestamp_list': OctetString,
}
class SingleResponseExtensions(SequenceOf):
_child_spec = SingleResponseExtension
class SingleResponse(Sequence):
_fields = [
('cert_id', CertId),
('cert_status', CertStatus),
('this_update', GeneralizedTime),
('next_update', GeneralizedTime, {'explicit': 0, 'optional': True}),
('single_extensions', SingleResponseExtensions, {'explicit': 1, 'optional': True}),
]
_processed_extensions = False
_critical_extensions = None
_crl_value = None
_archive_cutoff_value = None
_crl_reason_value = None
_invalidity_date_value = None
_certificate_issuer_value = None
def _set_extensions(self):
"""
Sets common named extensions to private attributes and creates a list
of critical extensions
"""
self._critical_extensions = set()
for extension in self['single_extensions']:
name = extension['extn_id'].native
attribute_name = '_%s_value' % name
if hasattr(self, attribute_name):
setattr(self, attribute_name, extension['extn_value'].parsed)
if extension['critical'].native:
self._critical_extensions.add(name)
self._processed_extensions = True
@property
def critical_extensions(self):
"""
Returns a set of the names (or OID if not a known extension) of the
extensions marked as critical
:return:
A set of unicode strings
"""
if not self._processed_extensions:
self._set_extensions()
return self._critical_extensions
@property
def crl_value(self):
"""
This extension is used to locate the CRL that a certificate's revocation
is contained within.
:return:
None or a CrlId object
"""
if self._processed_extensions is False:
self._set_extensions()
return self._crl_value
@property
def archive_cutoff_value(self):
"""
This extension is used to indicate the date at which an archived
(historical) certificate status entry will no longer be available.
:return:
None or a GeneralizedTime object
"""
if self._processed_extensions is False:
self._set_extensions()
return self._archive_cutoff_value
@property
def crl_reason_value(self):
"""
This extension indicates the reason that a certificate was revoked.
:return:
None or a CRLReason object
"""
if self._processed_extensions is False:
self._set_extensions()
return self._crl_reason_value
@property
def invalidity_date_value(self):
"""
This extension indicates the suspected date/time the private key was
compromised or the certificate became invalid. This would usually be
before the revocation date, which is when the CA processed the
revocation.
:return:
None or a GeneralizedTime object
"""
if self._processed_extensions is False:
self._set_extensions()
return self._invalidity_date_value
@property
def certificate_issuer_value(self):
"""
This extension indicates the issuer of the certificate in question.
:return:
None or an x509.GeneralNames object
"""
if self._processed_extensions is False:
self._set_extensions()
return self._certificate_issuer_value
class Responses(SequenceOf):
_child_spec = SingleResponse
class ResponseDataExtensionId(ObjectIdentifier):
_map = {
'1.3.6.1.5.5.7.48.1.2': 'nonce',
'1.3.6.1.5.5.7.48.1.9': 'extended_revoke',
}
class ResponseDataExtension(Sequence):
_fields = [
('extn_id', ResponseDataExtensionId),
('critical', Boolean, {'default': False}),
('extn_value', ParsableOctetString),
]
_oid_pair = ('extn_id', 'extn_value')
_oid_specs = {
'nonce': OctetString,
'extended_revoke': Null,
}
class ResponseDataExtensions(SequenceOf):
_child_spec = ResponseDataExtension
class ResponseData(Sequence):
_fields = [
('version', Version, {'explicit': 0, 'default': 'v1'}),
('responder_id', ResponderId),
('produced_at', GeneralizedTime),
('responses', Responses),
('response_extensions', ResponseDataExtensions, {'explicit': 1, 'optional': True}),
]
class BasicOCSPResponse(Sequence):
_fields = [
('tbs_response_data', ResponseData),
('signature_algorithm', SignedDigestAlgorithm),
('signature', OctetBitString),
('certs', Certificates, {'explicit': 0, 'optional': True}),
]
class ResponseBytes(Sequence):
_fields = [
('response_type', ResponseType),
('response', ParsableOctetString),
]
_oid_pair = ('response_type', 'response')
_oid_specs = {
'basic_ocsp_response': BasicOCSPResponse,
}
class OCSPResponse(Sequence):
_fields = [
('response_status', OCSPResponseStatus),
('response_bytes', ResponseBytes, {'explicit': 0, 'optional': True}),
]
_processed_extensions = False
_critical_extensions = None
_nonce_value = None
_extended_revoke_value = None
def _set_extensions(self):
"""
Sets common named extensions to private attributes and creates a list
of critical extensions
"""
self._critical_extensions = set()
for extension in self['response_bytes']['response'].parsed['tbs_response_data']['response_extensions']:
name = extension['extn_id'].native
attribute_name = '_%s_value' % name
if hasattr(self, attribute_name):
setattr(self, attribute_name, extension['extn_value'].parsed)
if extension['critical'].native:
self._critical_extensions.add(name)
self._processed_extensions = True
@property
def critical_extensions(self):
"""
Returns a set of the names (or OID if not a known extension) of the
extensions marked as critical
:return:
A set of unicode strings
"""
if not self._processed_extensions:
self._set_extensions()
return self._critical_extensions
@property
def nonce_value(self):
"""
This extension is used to prevent replay attacks on the request/response
exchange
:return:
None or an OctetString object
"""
if self._processed_extensions is False:
self._set_extensions()
return self._nonce_value
@property
def extended_revoke_value(self):
"""
This extension is used to signal that the responder will return a
"revoked" status for non-issued certificates.
:return:
None or a Null object (if present)
"""
if self._processed_extensions is False:
self._set_extensions()
return self._extended_revoke_value
@property
def basic_ocsp_response(self):
"""
A shortcut into the BasicOCSPResponse sequence
:return:
None or an asn1crypto.ocsp.BasicOCSPResponse object
"""
return self['response_bytes']['response'].parsed
@property
def response_data(self):
"""
A shortcut into the parsed, ResponseData sequence
:return:
None or an asn1crypto.ocsp.ResponseData object
"""
return self['response_bytes']['response'].parsed['tbs_response_data']

View File

@@ -0,0 +1,292 @@
# coding: utf-8
"""
Functions for parsing and dumping using the ASN.1 DER encoding. Exports the
following items:
- emit()
- parse()
- peek()
Other type classes are defined that help compose the types listed above.
"""
from __future__ import unicode_literals, division, absolute_import, print_function
import sys
from ._types import byte_cls, chr_cls, type_name
from .util import int_from_bytes, int_to_bytes
_PY2 = sys.version_info <= (3,)
_INSUFFICIENT_DATA_MESSAGE = 'Insufficient data - %s bytes requested but only %s available'
_MAX_DEPTH = 10
def emit(class_, method, tag, contents):
"""
Constructs a byte string of an ASN.1 DER-encoded value
This is typically not useful. Instead, use one of the standard classes from
asn1crypto.core, or construct a new class with specific fields, and call the
.dump() method.
:param class_:
An integer ASN.1 class value: 0 (universal), 1 (application),
2 (context), 3 (private)
:param method:
An integer ASN.1 method value: 0 (primitive), 1 (constructed)
:param tag:
An integer ASN.1 tag value
:param contents:
A byte string of the encoded byte contents
:return:
A byte string of the ASN.1 DER value (header and contents)
"""
if not isinstance(class_, int):
raise TypeError('class_ must be an integer, not %s' % type_name(class_))
if class_ < 0 or class_ > 3:
raise ValueError('class_ must be one of 0, 1, 2 or 3, not %s' % class_)
if not isinstance(method, int):
raise TypeError('method must be an integer, not %s' % type_name(method))
if method < 0 or method > 1:
raise ValueError('method must be 0 or 1, not %s' % method)
if not isinstance(tag, int):
raise TypeError('tag must be an integer, not %s' % type_name(tag))
if tag < 0:
raise ValueError('tag must be greater than zero, not %s' % tag)
if not isinstance(contents, byte_cls):
raise TypeError('contents must be a byte string, not %s' % type_name(contents))
return _dump_header(class_, method, tag, contents) + contents
def parse(contents, strict=False):
"""
Parses a byte string of ASN.1 BER/DER-encoded data.
This is typically not useful. Instead, use one of the standard classes from
asn1crypto.core, or construct a new class with specific fields, and call the
.load() class method.
:param contents:
A byte string of BER/DER-encoded data
:param strict:
A boolean indicating if trailing data should be forbidden - if so, a
ValueError will be raised when trailing data exists
:raises:
ValueError - when the contents do not contain an ASN.1 header or are truncated in some way
TypeError - when contents is not a byte string
:return:
A 6-element tuple:
- 0: integer class (0 to 3)
- 1: integer method
- 2: integer tag
- 3: byte string header
- 4: byte string content
- 5: byte string trailer
"""
if not isinstance(contents, byte_cls):
raise TypeError('contents must be a byte string, not %s' % type_name(contents))
contents_len = len(contents)
info, consumed = _parse(contents, contents_len)
if strict and consumed != contents_len:
raise ValueError('Extra data - %d bytes of trailing data were provided' % (contents_len - consumed))
return info
def peek(contents):
"""
Parses a byte string of ASN.1 BER/DER-encoded data to find the length
This is typically used to look into an encoded value to see how long the
next chunk of ASN.1-encoded data is. Primarily it is useful when a
value is a concatenation of multiple values.
:param contents:
A byte string of BER/DER-encoded data
:raises:
ValueError - when the contents do not contain an ASN.1 header or are truncated in some way
TypeError - when contents is not a byte string
:return:
An integer with the number of bytes occupied by the ASN.1 value
"""
if not isinstance(contents, byte_cls):
raise TypeError('contents must be a byte string, not %s' % type_name(contents))
info, consumed = _parse(contents, len(contents))
return consumed
def _parse(encoded_data, data_len, pointer=0, lengths_only=False, depth=0):
"""
Parses a byte string into component parts
:param encoded_data:
A byte string that contains BER-encoded data
:param data_len:
The integer length of the encoded data
:param pointer:
The index in the byte string to parse from
:param lengths_only:
A boolean to cause the call to return a 2-element tuple of the integer
number of bytes in the header and the integer number of bytes in the
contents. Internal use only.
:param depth:
The recursion depth when evaluating indefinite-length encoding.
:return:
A 2-element tuple:
- 0: A tuple of (class_, method, tag, header, content, trailer)
- 1: An integer indicating how many bytes were consumed
"""
if depth > _MAX_DEPTH:
raise ValueError('Indefinite-length recursion limit exceeded')
start = pointer
if data_len < pointer + 1:
raise ValueError(_INSUFFICIENT_DATA_MESSAGE % (1, data_len - pointer))
first_octet = ord(encoded_data[pointer]) if _PY2 else encoded_data[pointer]
pointer += 1
tag = first_octet & 31
constructed = (first_octet >> 5) & 1
# Base 128 length using 8th bit as continuation indicator
if tag == 31:
tag = 0
while True:
if data_len < pointer + 1:
raise ValueError(_INSUFFICIENT_DATA_MESSAGE % (1, data_len - pointer))
num = ord(encoded_data[pointer]) if _PY2 else encoded_data[pointer]
pointer += 1
if num == 0x80 and tag == 0:
raise ValueError('Non-minimal tag encoding')
tag *= 128
tag += num & 127
if num >> 7 == 0:
break
if tag < 31:
raise ValueError('Non-minimal tag encoding')
if data_len < pointer + 1:
raise ValueError(_INSUFFICIENT_DATA_MESSAGE % (1, data_len - pointer))
length_octet = ord(encoded_data[pointer]) if _PY2 else encoded_data[pointer]
pointer += 1
trailer = b''
if length_octet >> 7 == 0:
contents_end = pointer + (length_octet & 127)
else:
length_octets = length_octet & 127
if length_octets:
if data_len < pointer + length_octets:
raise ValueError(_INSUFFICIENT_DATA_MESSAGE % (length_octets, data_len - pointer))
pointer += length_octets
contents_end = pointer + int_from_bytes(encoded_data[pointer - length_octets:pointer], signed=False)
else:
# To properly parse indefinite length values, we need to scan forward
# parsing headers until we find a value with a length of zero. If we
# just scanned looking for \x00\x00, nested indefinite length values
# would not work.
if not constructed:
raise ValueError('Indefinite-length element must be constructed')
contents_end = pointer
while data_len < contents_end + 2 or encoded_data[contents_end:contents_end+2] != b'\x00\x00':
_, contents_end = _parse(encoded_data, data_len, contents_end, lengths_only=True, depth=depth+1)
contents_end += 2
trailer = b'\x00\x00'
if contents_end > data_len:
raise ValueError(_INSUFFICIENT_DATA_MESSAGE % (contents_end - pointer, data_len - pointer))
if lengths_only:
return (pointer, contents_end)
return (
(
first_octet >> 6,
constructed,
tag,
encoded_data[start:pointer],
encoded_data[pointer:contents_end-len(trailer)],
trailer
),
contents_end
)
def _dump_header(class_, method, tag, contents):
"""
Constructs the header bytes for an ASN.1 object
:param class_:
An integer ASN.1 class value: 0 (universal), 1 (application),
2 (context), 3 (private)
:param method:
An integer ASN.1 method value: 0 (primitive), 1 (constructed)
:param tag:
An integer ASN.1 tag value
:param contents:
A byte string of the encoded byte contents
:return:
A byte string of the ASN.1 DER header
"""
header = b''
id_num = 0
id_num |= class_ << 6
id_num |= method << 5
if tag >= 31:
cont_bit = 0
while tag > 0:
header = chr_cls(cont_bit | (tag & 0x7f)) + header
if not cont_bit:
cont_bit = 0x80
tag = tag >> 7
header = chr_cls(id_num | 31) + header
else:
header += chr_cls(id_num | tag)
length = len(contents)
if length <= 127:
header += chr_cls(length)
else:
length_bytes = int_to_bytes(length)
header += chr_cls(0x80 | len(length_bytes))
header += length_bytes
return header

View File

@@ -0,0 +1,84 @@
# coding: utf-8
"""
ASN.1 type classes for PDF signature structures. Adds extra oid mapping and
value parsing to asn1crypto.x509.Extension() and asn1crypto.xms.CMSAttribute().
"""
from __future__ import unicode_literals, division, absolute_import, print_function
from .cms import CMSAttributeType, CMSAttribute
from .core import (
Boolean,
Integer,
Null,
ObjectIdentifier,
OctetString,
Sequence,
SequenceOf,
SetOf,
)
from .crl import CertificateList
from .ocsp import OCSPResponse
from .x509 import (
Extension,
ExtensionId,
GeneralName,
KeyPurposeId,
)
class AdobeArchiveRevInfo(Sequence):
_fields = [
('version', Integer)
]
class AdobeTimestamp(Sequence):
_fields = [
('version', Integer),
('location', GeneralName),
('requires_auth', Boolean, {'optional': True, 'default': False}),
]
class OtherRevInfo(Sequence):
_fields = [
('type', ObjectIdentifier),
('value', OctetString),
]
class SequenceOfCertificateList(SequenceOf):
_child_spec = CertificateList
class SequenceOfOCSPResponse(SequenceOf):
_child_spec = OCSPResponse
class SequenceOfOtherRevInfo(SequenceOf):
_child_spec = OtherRevInfo
class RevocationInfoArchival(Sequence):
_fields = [
('crl', SequenceOfCertificateList, {'explicit': 0, 'optional': True}),
('ocsp', SequenceOfOCSPResponse, {'explicit': 1, 'optional': True}),
('other_rev_info', SequenceOfOtherRevInfo, {'explicit': 2, 'optional': True}),
]
class SetOfRevocationInfoArchival(SetOf):
_child_spec = RevocationInfoArchival
ExtensionId._map['1.2.840.113583.1.1.9.2'] = 'adobe_archive_rev_info'
ExtensionId._map['1.2.840.113583.1.1.9.1'] = 'adobe_timestamp'
ExtensionId._map['1.2.840.113583.1.1.10'] = 'adobe_ppklite_credential'
Extension._oid_specs['adobe_archive_rev_info'] = AdobeArchiveRevInfo
Extension._oid_specs['adobe_timestamp'] = AdobeTimestamp
Extension._oid_specs['adobe_ppklite_credential'] = Null
KeyPurposeId._map['1.2.840.113583.1.1.5'] = 'pdf_signing'
CMSAttributeType._map['1.2.840.113583.1.1.8'] = 'adobe_revocation_info_archival'
CMSAttribute._oid_specs['adobe_revocation_info_archival'] = SetOfRevocationInfoArchival

View File

@@ -0,0 +1,222 @@
# coding: utf-8
"""
Encoding DER to PEM and decoding PEM to DER. Exports the following items:
- armor()
- detect()
- unarmor()
"""
from __future__ import unicode_literals, division, absolute_import, print_function
import base64
import re
import sys
from ._errors import unwrap
from ._types import type_name as _type_name, str_cls, byte_cls
if sys.version_info < (3,):
from cStringIO import StringIO as BytesIO
else:
from io import BytesIO
def detect(byte_string):
"""
Detect if a byte string seems to contain a PEM-encoded block
:param byte_string:
A byte string to look through
:return:
A boolean, indicating if a PEM-encoded block is contained in the byte
string
"""
if not isinstance(byte_string, byte_cls):
raise TypeError(unwrap(
'''
byte_string must be a byte string, not %s
''',
_type_name(byte_string)
))
return byte_string.find(b'-----BEGIN') != -1 or byte_string.find(b'---- BEGIN') != -1
def armor(type_name, der_bytes, headers=None):
"""
Armors a DER-encoded byte string in PEM
:param type_name:
A unicode string that will be capitalized and placed in the header
and footer of the block. E.g. "CERTIFICATE", "PRIVATE KEY", etc. This
will appear as "-----BEGIN CERTIFICATE-----" and
"-----END CERTIFICATE-----".
:param der_bytes:
A byte string to be armored
:param headers:
An OrderedDict of the header lines to write after the BEGIN line
:return:
A byte string of the PEM block
"""
if not isinstance(der_bytes, byte_cls):
raise TypeError(unwrap(
'''
der_bytes must be a byte string, not %s
''' % _type_name(der_bytes)
))
if not isinstance(type_name, str_cls):
raise TypeError(unwrap(
'''
type_name must be a unicode string, not %s
''',
_type_name(type_name)
))
type_name = type_name.upper().encode('ascii')
output = BytesIO()
output.write(b'-----BEGIN ')
output.write(type_name)
output.write(b'-----\n')
if headers:
for key in headers:
output.write(key.encode('ascii'))
output.write(b': ')
output.write(headers[key].encode('ascii'))
output.write(b'\n')
output.write(b'\n')
b64_bytes = base64.b64encode(der_bytes)
b64_len = len(b64_bytes)
i = 0
while i < b64_len:
output.write(b64_bytes[i:i + 64])
output.write(b'\n')
i += 64
output.write(b'-----END ')
output.write(type_name)
output.write(b'-----\n')
return output.getvalue()
def _unarmor(pem_bytes):
"""
Convert a PEM-encoded byte string into one or more DER-encoded byte strings
:param pem_bytes:
A byte string of the PEM-encoded data
:raises:
ValueError - when the pem_bytes do not appear to be PEM-encoded bytes
:return:
A generator of 3-element tuples in the format: (object_type, headers,
der_bytes). The object_type is a unicode string of what is between
"-----BEGIN " and "-----". Examples include: "CERTIFICATE",
"PUBLIC KEY", "PRIVATE KEY". The headers is a dict containing any lines
in the form "Name: Value" that are right after the begin line.
"""
if not isinstance(pem_bytes, byte_cls):
raise TypeError(unwrap(
'''
pem_bytes must be a byte string, not %s
''',
_type_name(pem_bytes)
))
# Valid states include: "trash", "headers", "body"
state = 'trash'
headers = {}
base64_data = b''
object_type = None
found_start = False
found_end = False
for line in pem_bytes.splitlines(False):
if line == b'':
continue
if state == "trash":
# Look for a starting line since some CA cert bundle show the cert
# into in a parsed format above each PEM block
type_name_match = re.match(b'^(?:---- |-----)BEGIN ([A-Z0-9 ]+)(?: ----|-----)', line)
if not type_name_match:
continue
object_type = type_name_match.group(1).decode('ascii')
found_start = True
state = 'headers'
continue
if state == 'headers':
if line.find(b':') == -1:
state = 'body'
else:
decoded_line = line.decode('ascii')
name, value = decoded_line.split(':', 1)
headers[name] = value.strip()
continue
if state == 'body':
if line[0:5] in (b'-----', b'---- '):
der_bytes = base64.b64decode(base64_data)
yield (object_type, headers, der_bytes)
state = 'trash'
headers = {}
base64_data = b''
object_type = None
found_end = True
continue
base64_data += line
if not found_start or not found_end:
raise ValueError(unwrap(
'''
pem_bytes does not appear to contain PEM-encoded data - no
BEGIN/END combination found
'''
))
def unarmor(pem_bytes, multiple=False):
"""
Convert a PEM-encoded byte string into a DER-encoded byte string
:param pem_bytes:
A byte string of the PEM-encoded data
:param multiple:
If True, function will return a generator
:raises:
ValueError - when the pem_bytes do not appear to be PEM-encoded bytes
:return:
A 3-element tuple (object_name, headers, der_bytes). The object_name is
a unicode string of what is between "-----BEGIN " and "-----". Examples
include: "CERTIFICATE", "PUBLIC KEY", "PRIVATE KEY". The headers is a
dict containing any lines in the form "Name: Value" that are right
after the begin line.
"""
generator = _unarmor(pem_bytes)
if not multiple:
return next(generator)
return generator

View File

@@ -0,0 +1,193 @@
# coding: utf-8
"""
ASN.1 type classes for PKCS#12 files. Exports the following items:
- CertBag()
- CrlBag()
- Pfx()
- SafeBag()
- SecretBag()
Other type classes are defined that help compose the types listed above.
"""
from __future__ import unicode_literals, division, absolute_import, print_function
from .algos import DigestInfo
from .cms import ContentInfo, SignedData
from .core import (
Any,
BMPString,
Integer,
ObjectIdentifier,
OctetString,
ParsableOctetString,
Sequence,
SequenceOf,
SetOf,
)
from .keys import PrivateKeyInfo, EncryptedPrivateKeyInfo
from .x509 import Certificate, KeyPurposeId
# The structures in this file are taken from https://tools.ietf.org/html/rfc7292
class MacData(Sequence):
_fields = [
('mac', DigestInfo),
('mac_salt', OctetString),
('iterations', Integer, {'default': 1}),
]
class Version(Integer):
_map = {
3: 'v3'
}
class AttributeType(ObjectIdentifier):
_map = {
# https://tools.ietf.org/html/rfc2985#page-18
'1.2.840.113549.1.9.20': 'friendly_name',
'1.2.840.113549.1.9.21': 'local_key_id',
# https://support.microsoft.com/en-us/kb/287547
'1.3.6.1.4.1.311.17.1': 'microsoft_local_machine_keyset',
# https://github.com/frohoff/jdk8u-dev-jdk/blob/master/src/share/classes/sun/security/pkcs12/PKCS12KeyStore.java
# this is a set of OIDs, representing key usage, the usual value is a SET of one element OID 2.5.29.37.0
'2.16.840.1.113894.746875.1.1': 'trusted_key_usage',
}
class SetOfAny(SetOf):
_child_spec = Any
class SetOfBMPString(SetOf):
_child_spec = BMPString
class SetOfOctetString(SetOf):
_child_spec = OctetString
class SetOfKeyPurposeId(SetOf):
_child_spec = KeyPurposeId
class Attribute(Sequence):
_fields = [
('type', AttributeType),
('values', None),
]
_oid_specs = {
'friendly_name': SetOfBMPString,
'local_key_id': SetOfOctetString,
'microsoft_csp_name': SetOfBMPString,
'trusted_key_usage': SetOfKeyPurposeId,
}
def _values_spec(self):
return self._oid_specs.get(self['type'].native, SetOfAny)
_spec_callbacks = {
'values': _values_spec
}
class Attributes(SetOf):
_child_spec = Attribute
class Pfx(Sequence):
_fields = [
('version', Version),
('auth_safe', ContentInfo),
('mac_data', MacData, {'optional': True})
]
_authenticated_safe = None
@property
def authenticated_safe(self):
if self._authenticated_safe is None:
content = self['auth_safe']['content']
if isinstance(content, SignedData):
content = content['content_info']['content']
self._authenticated_safe = AuthenticatedSafe.load(content.native)
return self._authenticated_safe
class AuthenticatedSafe(SequenceOf):
_child_spec = ContentInfo
class BagId(ObjectIdentifier):
_map = {
'1.2.840.113549.1.12.10.1.1': 'key_bag',
'1.2.840.113549.1.12.10.1.2': 'pkcs8_shrouded_key_bag',
'1.2.840.113549.1.12.10.1.3': 'cert_bag',
'1.2.840.113549.1.12.10.1.4': 'crl_bag',
'1.2.840.113549.1.12.10.1.5': 'secret_bag',
'1.2.840.113549.1.12.10.1.6': 'safe_contents',
}
class CertId(ObjectIdentifier):
_map = {
'1.2.840.113549.1.9.22.1': 'x509',
'1.2.840.113549.1.9.22.2': 'sdsi',
}
class CertBag(Sequence):
_fields = [
('cert_id', CertId),
('cert_value', ParsableOctetString, {'explicit': 0}),
]
_oid_pair = ('cert_id', 'cert_value')
_oid_specs = {
'x509': Certificate,
}
class CrlBag(Sequence):
_fields = [
('crl_id', ObjectIdentifier),
('crl_value', OctetString, {'explicit': 0}),
]
class SecretBag(Sequence):
_fields = [
('secret_type_id', ObjectIdentifier),
('secret_value', OctetString, {'explicit': 0}),
]
class SafeContents(SequenceOf):
pass
class SafeBag(Sequence):
_fields = [
('bag_id', BagId),
('bag_value', Any, {'explicit': 0}),
('bag_attributes', Attributes, {'optional': True}),
]
_oid_pair = ('bag_id', 'bag_value')
_oid_specs = {
'key_bag': PrivateKeyInfo,
'pkcs8_shrouded_key_bag': EncryptedPrivateKeyInfo,
'cert_bag': CertBag,
'crl_bag': CrlBag,
'secret_bag': SecretBag,
'safe_contents': SafeContents
}
SafeContents._child_spec = SafeBag

View File

@@ -0,0 +1,310 @@
# coding: utf-8
"""
ASN.1 type classes for the time stamp protocol (TSP). Exports the following
items:
- TimeStampReq()
- TimeStampResp()
Also adds TimeStampedData() support to asn1crypto.cms.ContentInfo(),
TimeStampedData() and TSTInfo() support to
asn1crypto.cms.EncapsulatedContentInfo() and some oids and value parsers to
asn1crypto.cms.CMSAttribute().
Other type classes are defined that help compose the types listed above.
"""
from __future__ import unicode_literals, division, absolute_import, print_function
from .algos import DigestAlgorithm
from .cms import (
CMSAttribute,
CMSAttributeType,
ContentInfo,
ContentType,
EncapsulatedContentInfo,
)
from .core import (
Any,
BitString,
Boolean,
Choice,
GeneralizedTime,
IA5String,
Integer,
ObjectIdentifier,
OctetString,
Sequence,
SequenceOf,
SetOf,
UTF8String,
)
from .crl import CertificateList
from .x509 import (
Attributes,
CertificatePolicies,
GeneralName,
GeneralNames,
)
# The structures in this file are based on https://tools.ietf.org/html/rfc3161,
# https://tools.ietf.org/html/rfc4998, https://tools.ietf.org/html/rfc5544,
# https://tools.ietf.org/html/rfc5035, https://tools.ietf.org/html/rfc2634
class Version(Integer):
_map = {
0: 'v0',
1: 'v1',
2: 'v2',
3: 'v3',
4: 'v4',
5: 'v5',
}
class MessageImprint(Sequence):
_fields = [
('hash_algorithm', DigestAlgorithm),
('hashed_message', OctetString),
]
class Accuracy(Sequence):
_fields = [
('seconds', Integer, {'optional': True}),
('millis', Integer, {'implicit': 0, 'optional': True}),
('micros', Integer, {'implicit': 1, 'optional': True}),
]
class Extension(Sequence):
_fields = [
('extn_id', ObjectIdentifier),
('critical', Boolean, {'default': False}),
('extn_value', OctetString),
]
class Extensions(SequenceOf):
_child_spec = Extension
class TSTInfo(Sequence):
_fields = [
('version', Version),
('policy', ObjectIdentifier),
('message_imprint', MessageImprint),
('serial_number', Integer),
('gen_time', GeneralizedTime),
('accuracy', Accuracy, {'optional': True}),
('ordering', Boolean, {'default': False}),
('nonce', Integer, {'optional': True}),
('tsa', GeneralName, {'explicit': 0, 'optional': True}),
('extensions', Extensions, {'implicit': 1, 'optional': True}),
]
class TimeStampReq(Sequence):
_fields = [
('version', Version),
('message_imprint', MessageImprint),
('req_policy', ObjectIdentifier, {'optional': True}),
('nonce', Integer, {'optional': True}),
('cert_req', Boolean, {'default': False}),
('extensions', Extensions, {'implicit': 0, 'optional': True}),
]
class PKIStatus(Integer):
_map = {
0: 'granted',
1: 'granted_with_mods',
2: 'rejection',
3: 'waiting',
4: 'revocation_warning',
5: 'revocation_notification',
}
class PKIFreeText(SequenceOf):
_child_spec = UTF8String
class PKIFailureInfo(BitString):
_map = {
0: 'bad_alg',
2: 'bad_request',
5: 'bad_data_format',
14: 'time_not_available',
15: 'unaccepted_policy',
16: 'unaccepted_extensions',
17: 'add_info_not_available',
25: 'system_failure',
}
class PKIStatusInfo(Sequence):
_fields = [
('status', PKIStatus),
('status_string', PKIFreeText, {'optional': True}),
('fail_info', PKIFailureInfo, {'optional': True}),
]
class TimeStampResp(Sequence):
_fields = [
('status', PKIStatusInfo),
('time_stamp_token', ContentInfo),
]
class MetaData(Sequence):
_fields = [
('hash_protected', Boolean),
('file_name', UTF8String, {'optional': True}),
('media_type', IA5String, {'optional': True}),
('other_meta_data', Attributes, {'optional': True}),
]
class TimeStampAndCRL(Sequence):
_fields = [
('time_stamp', EncapsulatedContentInfo),
('crl', CertificateList, {'optional': True}),
]
class TimeStampTokenEvidence(SequenceOf):
_child_spec = TimeStampAndCRL
class DigestAlgorithms(SequenceOf):
_child_spec = DigestAlgorithm
class EncryptionInfo(Sequence):
_fields = [
('encryption_info_type', ObjectIdentifier),
('encryption_info_value', Any),
]
class PartialHashtree(SequenceOf):
_child_spec = OctetString
class PartialHashtrees(SequenceOf):
_child_spec = PartialHashtree
class ArchiveTimeStamp(Sequence):
_fields = [
('digest_algorithm', DigestAlgorithm, {'implicit': 0, 'optional': True}),
('attributes', Attributes, {'implicit': 1, 'optional': True}),
('reduced_hashtree', PartialHashtrees, {'implicit': 2, 'optional': True}),
('time_stamp', ContentInfo),
]
class ArchiveTimeStampSequence(SequenceOf):
_child_spec = ArchiveTimeStamp
class EvidenceRecord(Sequence):
_fields = [
('version', Version),
('digest_algorithms', DigestAlgorithms),
('crypto_infos', Attributes, {'implicit': 0, 'optional': True}),
('encryption_info', EncryptionInfo, {'implicit': 1, 'optional': True}),
('archive_time_stamp_sequence', ArchiveTimeStampSequence),
]
class OtherEvidence(Sequence):
_fields = [
('oe_type', ObjectIdentifier),
('oe_value', Any),
]
class Evidence(Choice):
_alternatives = [
('tst_evidence', TimeStampTokenEvidence, {'implicit': 0}),
('ers_evidence', EvidenceRecord, {'implicit': 1}),
('other_evidence', OtherEvidence, {'implicit': 2}),
]
class TimeStampedData(Sequence):
_fields = [
('version', Version),
('data_uri', IA5String, {'optional': True}),
('meta_data', MetaData, {'optional': True}),
('content', OctetString, {'optional': True}),
('temporal_evidence', Evidence),
]
class IssuerSerial(Sequence):
_fields = [
('issuer', GeneralNames),
('serial_number', Integer),
]
class ESSCertID(Sequence):
_fields = [
('cert_hash', OctetString),
('issuer_serial', IssuerSerial, {'optional': True}),
]
class ESSCertIDs(SequenceOf):
_child_spec = ESSCertID
class SigningCertificate(Sequence):
_fields = [
('certs', ESSCertIDs),
('policies', CertificatePolicies, {'optional': True}),
]
class SetOfSigningCertificates(SetOf):
_child_spec = SigningCertificate
class ESSCertIDv2(Sequence):
_fields = [
('hash_algorithm', DigestAlgorithm, {'default': {'algorithm': 'sha256'}}),
('cert_hash', OctetString),
('issuer_serial', IssuerSerial, {'optional': True}),
]
class ESSCertIDv2s(SequenceOf):
_child_spec = ESSCertIDv2
class SigningCertificateV2(Sequence):
_fields = [
('certs', ESSCertIDv2s),
('policies', CertificatePolicies, {'optional': True}),
]
class SetOfSigningCertificatesV2(SetOf):
_child_spec = SigningCertificateV2
EncapsulatedContentInfo._oid_specs['tst_info'] = TSTInfo
EncapsulatedContentInfo._oid_specs['timestamped_data'] = TimeStampedData
ContentInfo._oid_specs['timestamped_data'] = TimeStampedData
ContentType._map['1.2.840.113549.1.9.16.1.4'] = 'tst_info'
ContentType._map['1.2.840.113549.1.9.16.1.31'] = 'timestamped_data'
CMSAttributeType._map['1.2.840.113549.1.9.16.2.12'] = 'signing_certificate'
CMSAttribute._oid_specs['signing_certificate'] = SetOfSigningCertificates
CMSAttributeType._map['1.2.840.113549.1.9.16.2.47'] = 'signing_certificate_v2'
CMSAttribute._oid_specs['signing_certificate_v2'] = SetOfSigningCertificatesV2

View File

@@ -0,0 +1,878 @@
# coding: utf-8
"""
Miscellaneous data helpers, including functions for converting integers to and
from bytes and UTC timezone. Exports the following items:
- OrderedDict()
- int_from_bytes()
- int_to_bytes()
- timezone.utc
- utc_with_dst
- create_timezone()
- inet_ntop()
- inet_pton()
- uri_to_iri()
- iri_to_uri()
"""
from __future__ import unicode_literals, division, absolute_import, print_function
import math
import sys
from datetime import datetime, date, timedelta, tzinfo
from ._errors import unwrap
from ._iri import iri_to_uri, uri_to_iri # noqa
from ._ordereddict import OrderedDict # noqa
from ._types import type_name
if sys.platform == 'win32':
from ._inet import inet_ntop, inet_pton
else:
from socket import inet_ntop, inet_pton # noqa
# Python 2
if sys.version_info <= (3,):
def int_to_bytes(value, signed=False, width=None):
"""
Converts an integer to a byte string
:param value:
The integer to convert
:param signed:
If the byte string should be encoded using two's complement
:param width:
If None, the minimal possible size (but at least 1),
otherwise an integer of the byte width for the return value
:return:
A byte string
"""
if value == 0 and width == 0:
return b''
# Handle negatives in two's complement
is_neg = False
if signed and value < 0:
is_neg = True
bits = int(math.ceil(len('%x' % abs(value)) / 2.0) * 8)
value = (value + (1 << bits)) % (1 << bits)
hex_str = '%x' % value
if len(hex_str) & 1:
hex_str = '0' + hex_str
output = hex_str.decode('hex')
if signed and not is_neg and ord(output[0:1]) & 0x80:
output = b'\x00' + output
if width is not None:
if len(output) > width:
raise OverflowError('int too big to convert')
if is_neg:
pad_char = b'\xFF'
else:
pad_char = b'\x00'
output = (pad_char * (width - len(output))) + output
elif is_neg and ord(output[0:1]) & 0x80 == 0:
output = b'\xFF' + output
return output
def int_from_bytes(value, signed=False):
"""
Converts a byte string to an integer
:param value:
The byte string to convert
:param signed:
If the byte string should be interpreted using two's complement
:return:
An integer
"""
if value == b'':
return 0
num = long(value.encode("hex"), 16) # noqa
if not signed:
return num
# Check for sign bit and handle two's complement
if ord(value[0:1]) & 0x80:
bit_len = len(value) * 8
return num - (1 << bit_len)
return num
class timezone(tzinfo): # noqa
"""
Implements datetime.timezone for py2.
Only full minute offsets are supported.
DST is not supported.
"""
def __init__(self, offset, name=None):
"""
:param offset:
A timedelta with this timezone's offset from UTC
:param name:
Name of the timezone; if None, generate one.
"""
if not timedelta(hours=-24) < offset < timedelta(hours=24):
raise ValueError('Offset must be in [-23:59, 23:59]')
if offset.seconds % 60 or offset.microseconds:
raise ValueError('Offset must be full minutes')
self._offset = offset
if name is not None:
self._name = name
elif not offset:
self._name = 'UTC'
else:
self._name = 'UTC' + _format_offset(offset)
def __eq__(self, other):
"""
Compare two timezones
:param other:
The other timezone to compare to
:return:
A boolean
"""
if type(other) != timezone:
return False
return self._offset == other._offset
def __getinitargs__(self):
"""
Called by tzinfo.__reduce__ to support pickle and copy.
:return:
offset and name, to be used for __init__
"""
return self._offset, self._name
def tzname(self, dt):
"""
:param dt:
A datetime object; ignored.
:return:
Name of this timezone
"""
return self._name
def utcoffset(self, dt):
"""
:param dt:
A datetime object; ignored.
:return:
A timedelta object with the offset from UTC
"""
return self._offset
def dst(self, dt):
"""
:param dt:
A datetime object; ignored.
:return:
Zero timedelta
"""
return timedelta(0)
timezone.utc = timezone(timedelta(0))
# Python 3
else:
from datetime import timezone # noqa
def int_to_bytes(value, signed=False, width=None):
"""
Converts an integer to a byte string
:param value:
The integer to convert
:param signed:
If the byte string should be encoded using two's complement
:param width:
If None, the minimal possible size (but at least 1),
otherwise an integer of the byte width for the return value
:return:
A byte string
"""
if width is None:
if signed:
if value < 0:
bits_required = abs(value + 1).bit_length()
else:
bits_required = value.bit_length()
if bits_required % 8 == 0:
bits_required += 1
else:
bits_required = value.bit_length()
width = math.ceil(bits_required / 8) or 1
return value.to_bytes(width, byteorder='big', signed=signed)
def int_from_bytes(value, signed=False):
"""
Converts a byte string to an integer
:param value:
The byte string to convert
:param signed:
If the byte string should be interpreted using two's complement
:return:
An integer
"""
return int.from_bytes(value, 'big', signed=signed)
def _format_offset(off):
"""
Format a timedelta into "[+-]HH:MM" format or "" for None
"""
if off is None:
return ''
mins = off.days * 24 * 60 + off.seconds // 60
sign = '-' if mins < 0 else '+'
return sign + '%02d:%02d' % divmod(abs(mins), 60)
class _UtcWithDst(tzinfo):
"""
Utc class where dst does not return None; required for astimezone
"""
def tzname(self, dt):
return 'UTC'
def utcoffset(self, dt):
return timedelta(0)
def dst(self, dt):
return timedelta(0)
utc_with_dst = _UtcWithDst()
_timezone_cache = {}
def create_timezone(offset):
"""
Returns a new datetime.timezone object with the given offset.
Uses cached objects if possible.
:param offset:
A datetime.timedelta object; It needs to be in full minutes and between -23:59 and +23:59.
:return:
A datetime.timezone object
"""
try:
tz = _timezone_cache[offset]
except KeyError:
tz = _timezone_cache[offset] = timezone(offset)
return tz
class extended_date(object):
"""
A datetime.datetime-like object that represents the year 0. This is just
to handle 0000-01-01 found in some certificates. Python's datetime does
not support year 0.
The proleptic gregorian calendar repeats itself every 400 years. Therefore,
the simplest way to format is to substitute year 2000.
"""
def __init__(self, year, month, day):
"""
:param year:
The integer 0
:param month:
An integer from 1 to 12
:param day:
An integer from 1 to 31
"""
if year != 0:
raise ValueError('year must be 0')
self._y2k = date(2000, month, day)
@property
def year(self):
"""
:return:
The integer 0
"""
return 0
@property
def month(self):
"""
:return:
An integer from 1 to 12
"""
return self._y2k.month
@property
def day(self):
"""
:return:
An integer from 1 to 31
"""
return self._y2k.day
def strftime(self, format):
"""
Formats the date using strftime()
:param format:
A strftime() format string
:return:
A str, the formatted date as a unicode string
in Python 3 and a byte string in Python 2
"""
# Format the date twice, once with year 2000, once with year 4000.
# The only differences in the result will be in the millennium. Find them and replace by zeros.
y2k = self._y2k.strftime(format)
y4k = self._y2k.replace(year=4000).strftime(format)
return ''.join('0' if (c2, c4) == ('2', '4') else c2 for c2, c4 in zip(y2k, y4k))
def isoformat(self):
"""
Formats the date as %Y-%m-%d
:return:
The date formatted to %Y-%m-%d as a unicode string in Python 3
and a byte string in Python 2
"""
return self.strftime('0000-%m-%d')
def replace(self, year=None, month=None, day=None):
"""
Returns a new datetime.date or asn1crypto.util.extended_date
object with the specified components replaced
:return:
A datetime.date or asn1crypto.util.extended_date object
"""
if year is None:
year = self.year
if month is None:
month = self.month
if day is None:
day = self.day
if year > 0:
cls = date
else:
cls = extended_date
return cls(
year,
month,
day
)
def __str__(self):
"""
:return:
A str representing this extended_date, e.g. "0000-01-01"
"""
return self.strftime('%Y-%m-%d')
def __eq__(self, other):
"""
Compare two extended_date objects
:param other:
The other extended_date to compare to
:return:
A boolean
"""
# datetime.date object wouldn't compare equal because it can't be year 0
if not isinstance(other, self.__class__):
return False
return self.__cmp__(other) == 0
def __ne__(self, other):
"""
Compare two extended_date objects
:param other:
The other extended_date to compare to
:return:
A boolean
"""
return not self.__eq__(other)
def _comparison_error(self, other):
raise TypeError(unwrap(
'''
An asn1crypto.util.extended_date object can only be compared to
an asn1crypto.util.extended_date or datetime.date object, not %s
''',
type_name(other)
))
def __cmp__(self, other):
"""
Compare two extended_date or datetime.date objects
:param other:
The other extended_date object to compare to
:return:
An integer smaller than, equal to, or larger than 0
"""
# self is year 0, other is >= year 1
if isinstance(other, date):
return -1
if not isinstance(other, self.__class__):
self._comparison_error(other)
if self._y2k < other._y2k:
return -1
if self._y2k > other._y2k:
return 1
return 0
def __lt__(self, other):
return self.__cmp__(other) < 0
def __le__(self, other):
return self.__cmp__(other) <= 0
def __gt__(self, other):
return self.__cmp__(other) > 0
def __ge__(self, other):
return self.__cmp__(other) >= 0
class extended_datetime(object):
"""
A datetime.datetime-like object that represents the year 0. This is just
to handle 0000-01-01 found in some certificates. Python's datetime does
not support year 0.
The proleptic gregorian calendar repeats itself every 400 years. Therefore,
the simplest way to format is to substitute year 2000.
"""
# There are 97 leap days during 400 years.
DAYS_IN_400_YEARS = 400 * 365 + 97
DAYS_IN_2000_YEARS = 5 * DAYS_IN_400_YEARS
def __init__(self, year, *args, **kwargs):
"""
:param year:
The integer 0
:param args:
Other positional arguments; see datetime.datetime.
:param kwargs:
Other keyword arguments; see datetime.datetime.
"""
if year != 0:
raise ValueError('year must be 0')
self._y2k = datetime(2000, *args, **kwargs)
@property
def year(self):
"""
:return:
The integer 0
"""
return 0
@property
def month(self):
"""
:return:
An integer from 1 to 12
"""
return self._y2k.month
@property
def day(self):
"""
:return:
An integer from 1 to 31
"""
return self._y2k.day
@property
def hour(self):
"""
:return:
An integer from 1 to 24
"""
return self._y2k.hour
@property
def minute(self):
"""
:return:
An integer from 1 to 60
"""
return self._y2k.minute
@property
def second(self):
"""
:return:
An integer from 1 to 60
"""
return self._y2k.second
@property
def microsecond(self):
"""
:return:
An integer from 0 to 999999
"""
return self._y2k.microsecond
@property
def tzinfo(self):
"""
:return:
If object is timezone aware, a datetime.tzinfo object, else None.
"""
return self._y2k.tzinfo
def utcoffset(self):
"""
:return:
If object is timezone aware, a datetime.timedelta object, else None.
"""
return self._y2k.utcoffset()
def time(self):
"""
:return:
A datetime.time object
"""
return self._y2k.time()
def date(self):
"""
:return:
An asn1crypto.util.extended_date of the date
"""
return extended_date(0, self.month, self.day)
def strftime(self, format):
"""
Performs strftime(), always returning a str
:param format:
A strftime() format string
:return:
A str of the formatted datetime
"""
# Format the datetime twice, once with year 2000, once with year 4000.
# The only differences in the result will be in the millennium. Find them and replace by zeros.
y2k = self._y2k.strftime(format)
y4k = self._y2k.replace(year=4000).strftime(format)
return ''.join('0' if (c2, c4) == ('2', '4') else c2 for c2, c4 in zip(y2k, y4k))
def isoformat(self, sep='T'):
"""
Formats the date as "%Y-%m-%d %H:%M:%S" with the sep param between the
date and time portions
:param set:
A single character of the separator to place between the date and
time
:return:
The formatted datetime as a unicode string in Python 3 and a byte
string in Python 2
"""
s = '0000-%02d-%02d%c%02d:%02d:%02d' % (self.month, self.day, sep, self.hour, self.minute, self.second)
if self.microsecond:
s += '.%06d' % self.microsecond
return s + _format_offset(self.utcoffset())
def replace(self, year=None, *args, **kwargs):
"""
Returns a new datetime.datetime or asn1crypto.util.extended_datetime
object with the specified components replaced
:param year:
The new year to substitute. None to keep it.
:param args:
Other positional arguments; see datetime.datetime.replace.
:param kwargs:
Other keyword arguments; see datetime.datetime.replace.
:return:
A datetime.datetime or asn1crypto.util.extended_datetime object
"""
if year:
return self._y2k.replace(year, *args, **kwargs)
return extended_datetime.from_y2k(self._y2k.replace(2000, *args, **kwargs))
def astimezone(self, tz):
"""
Convert this extended_datetime to another timezone.
:param tz:
A datetime.tzinfo object.
:return:
A new extended_datetime or datetime.datetime object
"""
return extended_datetime.from_y2k(self._y2k.astimezone(tz))
def timestamp(self):
"""
Return POSIX timestamp. Only supported in python >= 3.3
:return:
A float representing the seconds since 1970-01-01 UTC. This will be a negative value.
"""
return self._y2k.timestamp() - self.DAYS_IN_2000_YEARS * 86400
def __str__(self):
"""
:return:
A str representing this extended_datetime, e.g. "0000-01-01 00:00:00.000001-10:00"
"""
return self.isoformat(sep=' ')
def __eq__(self, other):
"""
Compare two extended_datetime objects
:param other:
The other extended_datetime to compare to
:return:
A boolean
"""
# Only compare against other datetime or extended_datetime objects
if not isinstance(other, (self.__class__, datetime)):
return False
# Offset-naive and offset-aware datetimes are never the same
if (self.tzinfo is None) != (other.tzinfo is None):
return False
return self.__cmp__(other) == 0
def __ne__(self, other):
"""
Compare two extended_datetime objects
:param other:
The other extended_datetime to compare to
:return:
A boolean
"""
return not self.__eq__(other)
def _comparison_error(self, other):
"""
Raises a TypeError about the other object not being suitable for
comparison
:param other:
The object being compared to
"""
raise TypeError(unwrap(
'''
An asn1crypto.util.extended_datetime object can only be compared to
an asn1crypto.util.extended_datetime or datetime.datetime object,
not %s
''',
type_name(other)
))
def __cmp__(self, other):
"""
Compare two extended_datetime or datetime.datetime objects
:param other:
The other extended_datetime or datetime.datetime object to compare to
:return:
An integer smaller than, equal to, or larger than 0
"""
if not isinstance(other, (self.__class__, datetime)):
self._comparison_error(other)
if (self.tzinfo is None) != (other.tzinfo is None):
raise TypeError("can't compare offset-naive and offset-aware datetimes")
diff = self - other
zero = timedelta(0)
if diff < zero:
return -1
if diff > zero:
return 1
return 0
def __lt__(self, other):
return self.__cmp__(other) < 0
def __le__(self, other):
return self.__cmp__(other) <= 0
def __gt__(self, other):
return self.__cmp__(other) > 0
def __ge__(self, other):
return self.__cmp__(other) >= 0
def __add__(self, other):
"""
Adds a timedelta
:param other:
A datetime.timedelta object to add.
:return:
A new extended_datetime or datetime.datetime object.
"""
return extended_datetime.from_y2k(self._y2k + other)
def __sub__(self, other):
"""
Subtracts a timedelta or another datetime.
:param other:
A datetime.timedelta or datetime.datetime or extended_datetime object to subtract.
:return:
If a timedelta is passed, a new extended_datetime or datetime.datetime object.
Else a datetime.timedelta object.
"""
if isinstance(other, timedelta):
return extended_datetime.from_y2k(self._y2k - other)
if isinstance(other, extended_datetime):
return self._y2k - other._y2k
if isinstance(other, datetime):
return self._y2k - other - timedelta(days=self.DAYS_IN_2000_YEARS)
return NotImplemented
def __rsub__(self, other):
return -(self - other)
@classmethod
def from_y2k(cls, value):
"""
Revert substitution of year 2000.
:param value:
A datetime.datetime object which is 2000 years in the future.
:return:
A new extended_datetime or datetime.datetime object.
"""
year = value.year - 2000
if year > 0:
new_cls = datetime
else:
new_cls = cls
return new_cls(
year,
value.month,
value.day,
value.hour,
value.minute,
value.second,
value.microsecond,
value.tzinfo
)

View File

@@ -0,0 +1,6 @@
# coding: utf-8
from __future__ import unicode_literals, division, absolute_import, print_function
__version__ = '1.5.1'
__version_info__ = (1, 5, 1)

File diff suppressed because it is too large Load Diff

151
jc/parsers/chage.py Normal file
View File

@@ -0,0 +1,151 @@
"""jc - JSON Convert `chage --list` command output parser
Supports `chage -l <username>` or `chage --list <username>`
Usage (cli):
$ chage -l johndoe | jc --chage
or
$ jc chage -l johndoe
Usage (module):
import jc
result = jc.parse('chage', chage_command_output)
Schema:
{
"password_last_changed": string,
"password_expires": string,
"password_inactive": string,
"account_expires": string,
"min_days_between_password_change": integer,
"max_days_between_password_change": integer,
"warning_days_before_password_expires": integer
}
Examples:
$ chage --list joeuser | jc --chage -p
{
"password_last_changed": "never",
"password_expires": "never",
"password_inactive": "never",
"account_expires": "never",
"min_days_between_password_change": 0,
"max_days_between_password_change": 99999,
"warning_days_before_password_expires": 7
}
$ chage --list joeuser | jc --chage -p -r
{
"password_last_changed": "never",
"password_expires": "never",
"password_inactive": "never",
"account_expires": "never",
"min_days_between_password_change": "0",
"max_days_between_password_change": "99999",
"warning_days_before_password_expires": "7"
}
"""
from typing import List, Dict
import jc.utils
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.0'
description = '`chage --list` command parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
compatible = ['linux']
magic_commands = ['chage --list', 'chage -l']
__version__ = info.version
def _process(proc_data: Dict) -> Dict:
"""
Final processing to conform to the schema.
Parameters:
proc_data: (Dictionary) raw structured data to process
Returns:
Dictionary. Structured to conform to the schema.
"""
int_list = ['min_days_between_password_change', 'max_days_between_password_change',
'warning_days_before_password_expires']
for key in proc_data:
if key in int_list:
proc_data[key] = jc.utils.convert_to_int(proc_data[key])
return proc_data
def parse(
data: str,
raw: bool = False,
quiet: bool = False
) -> Dict:
"""
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
Returns:
Dictionary. Raw or processed structured data.
"""
jc.utils.compatibility(__name__, info.compatible, quiet)
jc.utils.input_type_check(data)
raw_output: Dict = {}
if jc.utils.has_data(data):
for line in filter(None, data.splitlines()):
key, val = line.split(':', maxsplit=1)
key = key.strip()
val = val.strip()
if key == 'Last password change':
raw_output['password_last_changed'] = val
continue
if key == 'Password expires':
raw_output['password_expires'] = val
continue
if key == 'Password inactive':
raw_output['password_inactive'] = val
continue
if key == 'Account expires':
raw_output['account_expires'] = val
continue
if key == 'Minimum number of days between password change':
raw_output['min_days_between_password_change'] = val
continue
if key == 'Maximum number of days between password change':
raw_output['max_days_between_password_change'] = val
continue
if key == 'Number of days of warning before password expires':
raw_output['warning_days_before_password_expires'] = val
continue
return raw_output if raw else _process(raw_output)

View File

@@ -1,14 +1,14 @@
"""jc - JSON Convert `csv` file streaming parser
> This streaming parser outputs JSON Lines (cli) or returns a Generator
iterator of Dictionaries (module)
> This streaming parser outputs JSON Lines (cli) or returns an Iterable of
> Dictionaries (module)
The `csv` streaming parser will attempt to automatically detect the
delimiter character. If the delimiter cannot be detected it will default
to comma. The first row of the file must be a header row.
Note: The first 100 rows are read into memory to enable delimiter detection,
then the rest of the rows are loaded lazily.
> Note: The first 100 rows are read into memory to enable delimiter
> detection, then the rest of the rows are loaded lazily.
Usage (cli):
@@ -95,7 +95,7 @@ def _process(proc_data):
@add_jc_meta
def parse(data, raw=False, quiet=False, ignore_exceptions=False):
"""
Main text parsing generator function. Returns an iterator object.
Main text parsing generator function. Returns an iterable object.
Parameters:
@@ -106,13 +106,9 @@ def parse(data, raw=False, quiet=False, ignore_exceptions=False):
quiet: (boolean) suppress warning messages if True
ignore_exceptions: (boolean) ignore parsing exceptions if True
Yields:
Dictionary. Raw or processed structured data.
Returns:
Iterator object (generator)
Iterable of Dictionaries
"""
jc.utils.compatibility(__name__, info.compatible, quiet)
streaming_input_type_check(data)

View File

@@ -99,7 +99,7 @@ import jc.parsers.universal
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.9'
version = '1.10'
description = '`df` command parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
@@ -208,7 +208,9 @@ def parse(data, raw=False, quiet=False):
jc.utils.compatibility(__name__, info.compatible, quiet)
jc.utils.input_type_check(data)
cleandata = data.splitlines()
# remove blank lines
cleandata = list(filter(None, data.splitlines()))
fix_data = []
raw_output = []
filesystem_map = {}

View File

@@ -14,7 +14,7 @@ time of the system the parser is run on)
Usage (cli):
C:> dir | jc --dir
C:\> dir | jc --dir
Usage (module):
@@ -37,7 +37,7 @@ Schema:
Examples:
C:> dir | jc --dir -p
C:\> dir | jc --dir -p
[
{
"date": "03/24/2021",
@@ -78,7 +78,7 @@ Examples:
...
]
C:> dir | jc --dir -p -r
C:\> dir | jc --dir -p -r
[
{
"date": "03/24/2021",

View File

@@ -1,7 +1,7 @@
"""jc - JSON Convert `foo` command output streaming parser
> This streaming parser outputs JSON Lines (cli) or returns a Generator
iterator of Dictionaries (module)
> This streaming parser outputs JSON Lines (cli) or returns an Iterable of
> Dictionaries (module)
<<Short foo description and caveats>>
@@ -40,7 +40,7 @@ Examples:
{example output}
...
"""
from typing import Dict, Iterable, Generator, Union
from typing import Dict, Iterable, Union
import jc.utils
from jc.streaming import (
add_jc_meta, streaming_input_type_check, streaming_line_input_type_check, raise_or_yield
@@ -90,9 +90,9 @@ def parse(
raw: bool = False,
quiet: bool = False,
ignore_exceptions: bool = False
) -> Union[Generator[Dict, None, None], tuple]:
) -> Union[Iterable[Dict], tuple]:
"""
Main text parsing generator function. Returns an iterator object.
Main text parsing generator function. Returns an iterable object.
Parameters:
@@ -103,13 +103,10 @@ def parse(
quiet: (boolean) suppress warning messages if True
ignore_exceptions: (boolean) ignore parsing exceptions if True
Yields:
Dictionary. Raw or processed structured data.
Returns:
Iterator object (generator)
Iterable of Dictionaries
"""
jc.utils.compatibility(__name__, info.compatible, quiet)
streaming_input_type_check(data)

View File

@@ -38,8 +38,8 @@ Schema:
"author": string,
"author_email": string,
"date": string,
"epoch": integer, [0]
"epoch_utc": integer, [1]
"epoch": integer, # [0]
"epoch_utc": integer, # [1]
"commit_by": string,
"commit_by_email": string,
"commit_by_date": string,
@@ -148,12 +148,12 @@ import re
from typing import List, Dict
import jc.utils
hash_pattern = re.compile(r'([0-9]|[a-f])+')
hash_pattern = re.compile(r'(?:[0-9]|[a-f]){40}')
changes_pattern = re.compile(r'\s(?P<files>\d+)\s+(files? changed),\s+(?P<insertions>\d+)\s(insertions?\(\+\))?(,\s+)?(?P<deletions>\d+)?(\s+deletions?\(\-\))?')
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.0'
version = '1.1'
description = '`git log` command parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
@@ -234,7 +234,7 @@ def parse(
line_list = line.split(maxsplit=1)
# oneline style
if line_list and _is_commit_hash(line_list[0]):
if not line.startswith(' ') and line_list and _is_commit_hash(line_list[0]):
if output_line:
if file_list:
output_line['stats']['files'] = file_list

282
jc/parsers/git_log_s.py Normal file
View File

@@ -0,0 +1,282 @@
"""jc - JSON Convert `git log` command output streaming parser
> This streaming parser outputs JSON Lines (cli) or returns an Iterable of
> Dictionaries (module)
Can be used with the following format options:
- `oneline`
- `short`
- `medium`
- `full`
- `fuller`
Additional options supported:
- `--stat`
- `--shortstat`
The `epoch` calculated timestamp field is naive. (i.e. based on the
local time of the system the parser is run on)
The `epoch_utc` calculated timestamp field is timezone-aware and is
only available if the timezone field is UTC.
Usage (cli):
$ git log | jc --git-log-s
Usage (module):
import jc
result = jc.parse('git_log_s', git_log_command_output.splitlines())
for item in result:
# do something
Schema:
{
"commit": string,
"author": string,
"author_email": string,
"date": string,
"epoch": integer, # [0]
"epoch_utc": integer, # [1]
"commit_by": string,
"commit_by_email": string,
"commit_by_date": string,
"message": string,
"stats" : {
"files_changed": integer,
"insertions": integer,
"deletions": integer,
"files": [
string
]
}
# below object only exists if using -qq or ignore_exceptions=True
"_jc_meta": {
"success": boolean, # false if error parsing
"error": string, # exists if "success" is false
"line": string # exists if "success" is false
}
}
[0] naive timestamp if "date" field is parsable, else null
[1] timezone aware timestamp availabe for UTC, else null
Examples:
$ git log | jc --git-log-s
{"commit":"a730ae18c8e81c5261db132df73cd74f272a0a26","author":"Kelly...}
{"commit":"930bf439c06c48a952baec05a9896c8d92b7693e","author":"Kelly...}
...
"""
import re
from typing import List, Dict, Iterable, Union
import jc.utils
from jc.streaming import (
add_jc_meta, streaming_input_type_check, streaming_line_input_type_check, raise_or_yield
)
from jc.exceptions import ParseError
hash_pattern = re.compile(r'(?:[0-9]|[a-f]){40}')
changes_pattern = re.compile(r'\s(?P<files>\d+)\s+(files? changed),\s+(?P<insertions>\d+)\s(insertions?\(\+\))?(,\s+)?(?P<deletions>\d+)?(\s+deletions?\(\-\))?')
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.1'
description = '`git log` command streaming parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
compatible = ['linux', 'darwin', 'cygwin', 'win32', 'aix', 'freebsd']
streaming = True
__version__ = info.version
def _process(proc_data: Dict) -> Dict:
"""
Final processing to conform to the schema.
Parameters:
proc_data: (Dictionary) raw structured data to process
Returns:
Dictionary. Structured data to conform to the schema.
"""
int_list = ['files_changed', 'insertions', 'deletions']
if 'date' in proc_data:
ts = jc.utils.timestamp(proc_data['date'], format_hint=(1100,))
proc_data['epoch'] = ts.naive
proc_data['epoch_utc'] = ts.utc
if 'stats' in proc_data:
for key in proc_data['stats']:
if key in int_list:
proc_data['stats'][key] = jc.utils.convert_to_int(proc_data['stats'][key])
return proc_data
def _is_commit_hash(hash_string: str) -> bool:
# 0c55240e9da30ac4293dc324f1094de2abd3da91
if len(hash_string) != 40:
return False
if hash_pattern.match(hash_string):
return True
return False
@add_jc_meta
def parse(
data: Iterable[str],
raw: bool = False,
quiet: bool = False,
ignore_exceptions: bool = False
) -> Union[Iterable[Dict], tuple]:
"""
Main text parsing generator function. Returns an iterable object.
Parameters:
data: (iterable) line-based text data to parse
(e.g. sys.stdin or str.splitlines())
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
ignore_exceptions: (boolean) ignore parsing exceptions if True
Returns:
Iterable of Dictionaries
"""
jc.utils.compatibility(__name__, info.compatible, quiet)
streaming_input_type_check(data)
output_line: Dict = {}
message_lines: List[str] = []
file_list: List[str] = []
for line in data:
try:
streaming_line_input_type_check(line)
if line == '' or line == '\n':
continue
line_list = line.rstrip().split(maxsplit=1)
# oneline style
if not line.startswith(' ') and line_list and _is_commit_hash(line_list[0]):
if output_line:
if file_list:
output_line['stats']['files'] = file_list
yield output_line if raw else _process(output_line)
output_line = {}
message_lines = []
file_list = []
output_line = {
'commit': line_list[0],
'message': line_list[1]
}
continue
# all other styles
if line.startswith('commit '):
if output_line:
if message_lines:
output_line['message'] = '\n'.join(message_lines)
if file_list:
output_line['stats']['files'] = file_list
yield output_line if raw else _process(output_line)
output_line = {}
message_lines = []
file_list = []
output_line['commit'] = line_list[1]
continue
if line.startswith('Merge: '):
output_line['merge'] = line_list[1]
continue
if line.startswith('Author: '):
values = line_list[1].rsplit(maxsplit=1)
output_line['author'] = values[0]
output_line['author_email'] = values[1].strip('<').strip('>')
continue
if line.startswith('Date: '):
output_line['date'] = line_list[1]
continue
if line.startswith('AuthorDate: '):
output_line['date'] = line_list[1]
continue
if line.startswith('CommitDate: '):
output_line['commit_by_date'] = line_list[1]
continue
if line.startswith('Commit: '):
values = line_list[1].rsplit(maxsplit=1)
output_line['commit_by'] = values[0]
output_line['commit_by_email'] = values[1].strip('<').strip('>')
continue
if line.startswith(' '):
message_lines.append(line.strip())
continue
if line.startswith(' ') and 'changed, ' not in line:
# this is a file name
file_name = line.split('|')[0].strip()
file_list.append(file_name)
continue
if line.startswith(' ') and 'changed, ' in line:
# this is the stat summary
changes = changes_pattern.match(line)
if changes:
files = changes['files']
insertions = changes['insertions']
deletions = changes['deletions']
output_line['stats'] = {
'files_changed': files or '0',
'insertions': insertions or '0',
'deletions': deletions or '0'
}
continue
raise ParseError('Not git_log_s data')
except Exception as e:
yield raise_or_yield(ignore_exceptions, e, line)
try:
if output_line:
if message_lines:
output_line['message'] = '\n'.join(message_lines)
if file_list:
output_line['stats']['files'] = file_list
yield output_line if raw else _process(output_line)
except Exception as e:
yield raise_or_yield(ignore_exceptions, e, line)

337
jc/parsers/gpg.py Normal file
View File

@@ -0,0 +1,337 @@
"""jc - JSON Convert `gpg --with-colons` command output parser
Usage (cli):
$ gpg --with-colons --show-keys file.gpg | jc --gpg
or
$ jc gpg --with-colons --show-keys file.gpg
Usage (module):
import jc
result = jc.parse('gpg', gpg_command_output)
Schema:
Field definitions from https://git.gnupg.org/cgi-bin/gitweb.cgi?p=gnupg.git;a=blob_plain;f=doc/DETAILS
> Note: Number values are not converted to integers because many field
> specifications are overloaded and future augmentations are implied in the
> documentation.
[
{
"type": string,
"validity": string,
"key_length": string,
"pub_key_alg": string,
"key_id": string,
"creation_date": string,
"expiration_date": string,
"certsn_uidhash_trustinfo": string,
"owner_trust": string,
"user_id": string,
"signature_class": string,
"key_capabilities": string,
"cert_fingerprint_other": string,
"flag": string,
"token_sn": string,
"hash_alg": string,
"curve_name": string,
"compliance_flags": string,
"last_update_date": string,
"origin": string,
"comment": string,
"index": string, # [0]
"bits": string, # [0]
"value": string, # [0]
"version": string, # [1], [4]
"signature_count": string, # [1]
"encryption_count": string, # [1]
"policy": string, # [1]
"signature_first_seen": string, # [1]
"signature_most_recent_seen": string, # [1]
"encryption_first_done": string, # [1]
"encryption_most_recent_done": string, # [1]
"staleness_reason": string, # [2]
"trust_model": string, # [2]
"trust_db_created": string, # [2]
"trust_db_expires": string, # [2]
"marginally_trusted_users": string, # [2]
"completely_trusted_users": string, # [2]
"cert_chain_max_depth": string, # [2]
"subpacket_number": string, # [3]
"hex_flags": string, # [3]
"subpacket_length": string, # [3]
"subpacket_data": string, # [3]
"pubkey": string, # [4]
"cipher": string, # [4]
"digest": string, # [4]
"compress": string, # [4]
"group": string, # [4]
"members": string, # [4]
"curve_names": string, # [4]
}
]
All blank values are converted to null/None.
[0] for 'pkd' type
[1] for 'tfs' type
[2] for 'tru' type
[3] for 'skp' type
[4] for 'cfg' type
Examples:
$ gpg --with-colons --show-keys file.gpg | jc --gpg -p
[
{
"type": "pub",
"validity": "f",
"key_length": "1024",
"pub_key_alg": "17",
"key_id": "6C7EE1B8621CC013",
"creation_date": "899817715",
"expiration_date": "1055898235",
"certsn_uidhash_trustinfo": null,
"owner_trust": "m",
"user_id": null,
"signature_class": null,
"key_capabilities": "scESC",
"cert_fingerprint_other": null,
"flag": null,
"token_sn": null,
"hash_alg": null,
"curve_name": null,
"compliance_flags": null,
"last_update_date": null,
"origin": null,
"comment": null
},
...
]
"""
from typing import List, Dict, Optional
import jc.utils
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.0'
description = '`gpg --with-colons` command parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
compatible = ['linux']
magic_commands = ['gpg --with-colons']
__version__ = info.version
def _process(proc_data: List[Dict]) -> List[Dict]:
"""
Final processing to conform to the schema.
Parameters:
proc_data: (List of Dictionaries) raw structured data to process
Returns:
List of Dictionaries. Structured to conform to the schema.
"""
return proc_data
def _list_get(my_list: List, index: int, default_val=None) -> Optional[str]:
"""get a list value or return None/default value if out of range."""
if index <= len(my_list) - 1:
return my_list[index] or None
return default_val
def parse(
data: str,
raw: bool = False,
quiet: bool = False
) -> List[Dict]:
"""
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) unprocessed output if True
quiet: (boolean) suppress warning messages if True
Returns:
List of Dictionaries. Raw or processed structured data.
"""
jc.utils.compatibility(__name__, info.compatible, quiet)
jc.utils.input_type_check(data)
raw_output: List = []
if jc.utils.has_data(data):
for line in filter(None, data.splitlines()):
values = line.split(':')
temp_obj = {
'type': _list_get(values, 0),
'validity': _list_get(values, 1),
'key_length': _list_get(values, 2),
'pub_key_alg': _list_get(values, 3),
'key_id': _list_get(values, 4),
'creation_date': _list_get(values, 5),
'expiration_date': _list_get(values, 6),
'certsn_uidhash_trustinfo': _list_get(values, 7),
'owner_trust': _list_get(values, 8),
'user_id': _list_get(values, 9),
'signature_class': _list_get(values, 10),
'key_capabilities': _list_get(values, 11),
'cert_fingerprint_other': _list_get(values, 12),
'flag': _list_get(values, 13),
'token_sn': _list_get(values, 14),
'hash_alg': _list_get(values, 15),
'curve_name': _list_get(values, 16),
'compliance_flags': _list_get(values, 17),
'last_update_date': _list_get(values, 18),
'origin': _list_get(values, 19),
'comment': _list_get(values, 20)
}
# field mappings change for special types: pkd, tfs, tru, skp, cfg
if temp_obj['type'] == 'pkd':
# pkd:0:1024:B665B1435F4C2 .... FF26ABB:
# ! ! !-- the value
# ! !------ for information number of bits in the value
# !--------- index (eg. DSA goes from 0 to 3: p,q,g,y)
line_obj = {
'type': temp_obj['type'],
'index': temp_obj['validity'],
'bits': temp_obj['key_length'],
'value': temp_obj['pub_key_alg']
}
elif temp_obj['type'] == 'tfs':
# - Field 2 :: tfs record version (must be 1)
# - Field 3 :: validity - A number with validity code.
# - Field 4 :: signcount - The number of signatures seen.
# - Field 5 :: encrcount - The number of encryptions done.
# - Field 6 :: policy - A string with the policy
# - Field 7 :: signture-first-seen - a timestamp or 0 if not known.
# - Field 8 :: signature-most-recent-seen - a timestamp or 0 if not known.
# - Field 9 :: encryption-first-done - a timestamp or 0 if not known.
# - Field 10 :: encryption-most-recent-done - a timestamp or 0 if not known.
line_obj = {
'type': temp_obj['type'],
'version': temp_obj['validity'],
'validity': temp_obj['key_length'],
'signature_count': temp_obj['pub_key_alg'],
'encryption_count': temp_obj['key_id'],
'policy': temp_obj['creation_date'],
'signature_first_seen': temp_obj['expiration_date'],
'signature_most_recent_seen': temp_obj['certsn_uidhash_trustinfo'],
'encryption_first_done': temp_obj['owner_trust'],
'encryption_most_recent_done': temp_obj['user_id']
}
elif temp_obj['type'] == 'tru':
# tru:o:0:1166697654:1:3:1:5
# - Field 2 :: Reason for staleness of trust.
# - Field 3 :: Trust model
# - Field 4 :: Date trustdb was created in seconds since Epoch.
# - Field 5 :: Date trustdb will expire in seconds since Epoch.
# - Field 6 :: Number of marginally trusted users to introduce a new key signer.
# - Field 7 :: Number of completely trusted users to introduce a new key signer.
# - Field 8 :: Maximum depth of a certification chain.
line_obj = {
'type': temp_obj['type'],
'staleness_reason': temp_obj['validity'],
'trust_model': temp_obj['key_length'],
'trust_db_created': temp_obj['pub_key_alg'],
'trust_db_expires': temp_obj['key_id'],
'marginally_trusted_users': temp_obj['creation_date'],
'completely_trusted_users': temp_obj['expiration_date'],
'cert_chain_max_depth': temp_obj['certsn_uidhash_trustinfo']
}
elif temp_obj['type'] == 'skp':
# - Field 2 :: Subpacket number as per RFC-4880 and later.
# - Field 3 :: Flags in hex.
# - Field 4 :: Length of the subpacket.
# - Field 5 :: The subpacket data.
line_obj = {
'type': temp_obj['type'],
'subpacket_number': temp_obj['validity'],
'hex_flags': temp_obj['key_length'],
'subpacket_length': temp_obj['pub_key_alg'],
'subpacket_data': temp_obj['key_id']
}
elif temp_obj['type'] == 'cfg':
# there are several 'cfg' formats
if temp_obj['validity'] == 'version':
# cfg:version:1.3.5
line_obj = {
'type': temp_obj['type'],
'version': temp_obj['key_length']
}
elif temp_obj['validity'] == 'pubkey':
# cfg:pubkey:1;2;3;16;17
line_obj = {
'type': temp_obj['type'],
'pubkey': temp_obj['key_length']
}
elif temp_obj['validity'] == 'cipher':
# cfg:cipher:2;3;4;7;8;9;10
line_obj = {
'type': temp_obj['type'],
'cipher': temp_obj['key_length']
}
elif temp_obj['validity'] == 'digest':
# cfg:digest:1;2;3;8;9;10
line_obj = {
'type': temp_obj['type'],
'digest': temp_obj['key_length']
}
elif temp_obj['validity'] == 'compress':
# cfg:compress:0;1;2;3
line_obj = {
'type': temp_obj['type'],
'compress': temp_obj['key_length']
}
elif temp_obj['validity'] == 'group':
# cfg:group:mynames:patti;joe;0x12345678;paige
line_obj = {
'type': temp_obj['type'],
'group': temp_obj['key_length'],
'members': temp_obj['pub_key_alg']
}
elif temp_obj['validity'] == 'curve':
# cfg:curve:ed25519;nistp256;nistp384;nistp521
line_obj = {
'type': temp_obj['type'],
'curve_names': temp_obj['key_length']
}
else:
line_obj = temp_obj
raw_output.append(line_obj)
return raw_output if raw else _process(raw_output)

View File

@@ -105,7 +105,7 @@ import jc.utils
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.4'
version = '1.5'
description = '`id` command parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
@@ -144,6 +144,13 @@ def _process(proc_data):
return proc_data
def _get_item(my_list, index, default=None):
if index < len(my_list):
return my_list[index]
return default
def parse(data, raw=False, quiet=False):
"""
Main text parsing function
@@ -174,14 +181,14 @@ def parse(data, raw=False, quiet=False):
uid_parsed = uid_parsed.split('=')
raw_output['uid'] = {}
raw_output['uid']['id'] = uid_parsed[1]
raw_output['uid']['name'] = uid_parsed[2]
raw_output['uid']['name'] = _get_item(uid_parsed, 2)
if section.startswith('gid'):
gid_parsed = section.replace('(', '=').replace(')', '=')
gid_parsed = gid_parsed.split('=')
raw_output['gid'] = {}
raw_output['gid']['id'] = gid_parsed[1]
raw_output['gid']['name'] = gid_parsed[2]
raw_output['gid']['name'] = _get_item(gid_parsed, 2)
if section.startswith('groups'):
groups_parsed = section.replace('(', '=').replace(')', '=')
@@ -193,7 +200,7 @@ def parse(data, raw=False, quiet=False):
group_dict = {}
grp_parsed = group.split('=')
group_dict['id'] = grp_parsed[0]
group_dict['name'] = grp_parsed[1]
group_dict['name'] = _get_item(grp_parsed, 1)
raw_output['groups'].append(group_dict)
if section.startswith('context'):

View File

@@ -1,6 +1,6 @@
"""jc - JSON Convert `ifconfig` command output parser
Note: No `ifconfig` options are supported.
> Note: No `ifconfig` options are supported.
Usage (cli):

View File

@@ -6,9 +6,10 @@ Parses standard `INI` files and files containing simple key/value pairs.
- Comment prefix can be `#` or `;`. Comments must be on their own line.
- If duplicate keys are found, only the last value will be used.
> Note: Values starting and ending with quotation marks will have the marks
removed. If you would like to keep the quotation marks, use the `-r`
command-line argument or the `raw=True` argument in `parse()`.
> Note: Values starting and ending with double or single quotation marks
> will have the marks removed. If you would like to keep the quotation
> marks, use the `-r` command-line argument or the `raw=True` argument in
> `parse()`.
Usage (cli):
@@ -21,8 +22,8 @@ Usage (module):
Schema:
ini or key/value document converted to a dictionary - see the
configparser standard library documentation for more details.
ini or key/value document converted to a dictionary - see the configparser
standard library documentation for more details.
{
"key1": string,
@@ -69,7 +70,7 @@ import configparser
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.6'
version = '1.7'
description = 'INI file parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
@@ -99,17 +100,21 @@ def _process(proc_data):
for key, value in proc_data[heading].items():
if value is not None and value.startswith('"') and value.endswith('"'):
proc_data[heading][key] = value.lstrip('"').rstrip('"')
elif value is not None and value.startswith("'") and value.endswith("'"):
proc_data[heading][key] = value.lstrip("'").rstrip("'")
elif value is None:
proc_data[heading][key] = ''
# simple key/value files with no headers
else:
if (proc_data[heading] is not None and
proc_data[heading].startswith('"') and
proc_data[heading].endswith('"')):
if proc_data[heading] is not None and proc_data[heading].startswith('"') and proc_data[heading].endswith('"'):
proc_data[heading] = proc_data[heading].lstrip('"').rstrip('"')
elif proc_data[heading] is not None and proc_data[heading].startswith("'") and proc_data[heading].endswith("'"):
proc_data[heading] = proc_data[heading].lstrip("'").rstrip("'")
elif proc_data[heading] is None:
proc_data[heading] = ''

View File

@@ -1,6 +1,6 @@
"""jc - JSON Convert `iostat` command output parser
Note: `iostat` version 11 and higher include a JSON output option
> Note: `iostat` version 11 and higher include a JSON output option
Usage (cli):

View File

@@ -1,20 +1,20 @@
"""jc - JSON Convert `iostat` command output streaming parser
> This streaming parser outputs JSON Lines (cli) or returns a Generator
iterator of Dictionaries (module)
> This streaming parser outputs JSON Lines (cli) or returns an Iterable of
> Dictionaries (module)
Note: `iostat` version 11 and higher include a JSON output option
> Note: `iostat` version 11 and higher include a JSON output option
Usage (cli):
$ iostat | jc --iostat-s
> Note: When piping `jc` converted `iostat` output to other processes it may
appear the output is hanging due to the OS pipe buffers. This is because
`iostat` output is too small to quickly fill up the buffer. Use the `-u`
option to unbuffer the `jc` output if you would like immediate output. See
the [readme](https://github.com/kellyjonbrazil/jc/tree/master#unbuffering-output)
for more information.
> appear the output is hanging due to the OS pipe buffers. This is because
> `iostat` output is too small to quickly fill up the buffer. Use the `-u`
> option to unbuffer the `jc` output if you would like immediate output. See
> the [readme](https://github.com/kellyjonbrazil/jc/tree/master#unbuffering-output)
> for more information.
Usage (module):
@@ -163,7 +163,7 @@ def _create_obj_list(section_list, section_name):
@add_jc_meta
def parse(data, raw=False, quiet=False, ignore_exceptions=False):
"""
Main text parsing generator function. Returns an iterator object.
Main text parsing generator function. Returns an iterable object.
Parameters:
@@ -174,13 +174,9 @@ def parse(data, raw=False, quiet=False, ignore_exceptions=False):
quiet: (boolean) suppress warning messages if True
ignore_exceptions: (boolean) ignore parsing exceptions if True
Yields:
Dictionary. Raw or processed structured data.
Returns:
Iterator object (generator)
Iterable of Dictionaries
"""
jc.utils.compatibility(__name__, info.compatible, quiet)
streaming_input_type_check(data)

View File

@@ -163,7 +163,7 @@ import jc.utils
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.7'
version = '1.8'
description = '`iptables` command parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
@@ -264,7 +264,6 @@ def parse(data, raw=False, quiet=False):
continue
elif line.startswith('target') or line.find('pkts') == 1 or line.startswith('num'):
headers = []
headers = [h for h in ' '.join(line.lower().strip().split()).split() if h]
headers.append("options")

View File

@@ -7,8 +7,8 @@ Supports files containing simple key/value pairs.
- If duplicate keys are found, only the last value will be used.
> Note: Values starting and ending with quotation marks will have the marks
removed. If you would like to keep the quotation marks, use the `-r`
command-line argument or the `raw=True` argument in `parse()`.
> removed. If you would like to keep the quotation marks, use the `-r`
> command-line argument or the `raw=True` argument in `parse()`.
Usage (cli):
@@ -21,8 +21,8 @@ Usage (module):
Schema:
key/value document converted to a dictionary - see the
configparser standard library documentation for more details.
key/value document converted to a dictionary - see the configparser standard
library documentation for more details.
{
"key1": string,

View File

@@ -4,11 +4,12 @@ Options supported:
- `lbaR1`
- `--time-style=full-iso`
Note: The `-1`, `-l`, or `-b` option of `ls` should be used to correctly
parse filenames that include newline characters. Since `ls` does not encode
newlines in filenames when outputting to a pipe it will cause `jc` to see
multiple files instead of a single file if `-1`, `-l`, or `-b` is not used.
Alternatively, `vdir` can be used, which is the same as running `ls -lb`.
> Note: The `-1`, `-l`, or `-b` option of `ls` should be used to correctly
> parse filenames that include newline characters. Since `ls` does not
> encode newlines in filenames when outputting to a pipe it will cause `jc`
> to see multiple files instead of a single file if `-1`, `-l`, or `-b` is
> not used. Alternatively, `vdir` can be used, which is the same as running
> `ls -lb`.
The `epoch` calculated timestamp field is naive. (i.e. based on the local
time of the system the parser is run on)

View File

@@ -1,7 +1,7 @@
"""jc - JSON Convert `ls` and `vdir` command output streaming parser
> This streaming parser outputs JSON Lines (cli) or returns a Generator
iterator of Dictionaries (module)
> This streaming parser outputs JSON Lines (cli) or returns an Iterable of
> Dictionaries (module)
Requires the `-l` option to be used on `ls`. If there are newline characters
in the filename, then make sure to use the `-b` option on `ls`.
@@ -118,7 +118,7 @@ def _process(proc_data):
@add_jc_meta
def parse(data, raw=False, quiet=False, ignore_exceptions=False):
"""
Main text parsing generator function. Returns an iterator object.
Main text parsing generator function. Returns an iterable object.
Parameters:
@@ -129,13 +129,9 @@ def parse(data, raw=False, quiet=False, ignore_exceptions=False):
quiet: (boolean) suppress warning messages if True
ignore_exceptions: (boolean) ignore parsing exceptions if True
Yields:
Dictionary. Raw or processed structured data.
Returns:
Iterator object (generator)
Iterable of Dictionaries
"""
jc.utils.compatibility(__name__, info.compatible, quiet)
streaming_input_type_check(data)

Some files were not shown because too many files have changed in this diff Show More