python/jc
1
0
Fork 0
mirror of https://github.com/kellyjonbrazil/jc.git synced 2025-06-17 00:07:37 +02:00
Code Issues Releases Activity

Merge pull request #127 from kellyjonbrazil/dev

Browse Source 
Dev v1.15.3
This commit is contained in:
Kelly Brazil
2021-04-26 12:12:02 -07:00
committed by GitHub
parent fcc7e52949 35464bbbfb
commit fd7861db11
45 changed files with 1784 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
CHANGELOG
View File

@ -1,5 +1,11 @@
jc changelog
20210426 v1.15.3
- Add ufw status command parser tested on linux
- Add ufw-appinfo command parser tested on linux
- Fix deb package name to conform to standard
- Add Caveats section to readme and manpage
20210418 v1.15.2
- Add systeminfo parser tested on Windows
- Update dig parser to fix an issue with IPv6 addresses in the server field

7
CONTRIBUTING.md
View File

@ -17,7 +17,7 @@ Pull requests are the best way to propose changes to the codebase (we use [Githu
2. Fork the repo and create your branch from `dev`, if available, otherwise `master`.
3. If you've added code that should be tested, add tests. All new parsers should have several sample outputs and tests.
4. Documentation is auto-generated from docstrings, so ensure they are clear and accurate.
5. Ensure the test suite passes.
5. Ensure the test suite passes. (Note: "**America/Los_Angeles**" timezone should be configured on the test system)
6. Make sure your code lints.
7. Issue that pull request!
@ -61,6 +61,11 @@ Good:
]
```
## Tests
It is essential to have good command output sample coverage and tests to keep the `jc` parser quality high.
Many parsers include calculated timestamp fields using the `jc.utils.timestamp` class. Naive timestamps created with this class should be generated on a system configured with the "**America/Los_Angeles**" timezone on linux/macOS/unix and "**Pacific Standard Time**" timezone on Windows for tests to pass on the Github Actions CI tests. This timezone should be configured on your local system before running the tests locally, as well.
## Any contributions you make will be under the MIT Software License
In short, when you submit code changes, your submissions are understood to be under the same [MIT License](http://choosealicense.com/licenses/mit/) that covers the project. Feel free to contact the maintainers if that's a concern.

109
EXAMPLES.md
View File

@ -3126,6 +3126,115 @@ traceroute -m 3 8.8.8.8 | jc --traceroute -p # or: jc -p traceroute -m
]
}
```
### ufw status
```bash
ufw status verbose | jc --ufw -p # or jc -p ufw status verbose
```
```json
{
"status": "active",
"logging": "on",
"logging_level": "low",
"default": "deny (incoming), allow (outgoing), disabled (routed)",
"new_profiles": "skip",
"rules": [
{
"action": "ALLOW",
"action_direction": "IN",
"index": null,
"network_protocol": "ipv4",
"to_interface": "any",
"to_transport": "any",
"to_service": null,
"to_ports": [
22
],
"to_ip": "0.0.0.0",
"to_ip_prefix": 0,
"comment": null,
"from_ip": "0.0.0.0",
"from_ip_prefix": 0,
"from_interface": "any",
"from_transport": "any",
"from_port_ranges": [
{
"start": 0,
"end": 65535
}
],
"from_service": null
},
{
"action": "ALLOW",
"action_direction": "IN",
"index": null,
"network_protocol": "ipv4",
"to_interface": "any",
"to_transport": "tcp",
"to_service": null,
"to_ports": [
80,
443
],
"to_ip": "0.0.0.0",
"to_ip_prefix": 0,
"comment": null,
"from_ip": "0.0.0.0",
"from_ip_prefix": 0,
"from_interface": "any",
"from_transport": "any",
"from_port_ranges": [
{
"start": 0,
"end": 65535
}
],
"from_service": null
}
]
}
```
### ufw app info [application]
```bash
ufw app info MSN | jc --ufw-appinfo -p # or: jc -p ufw app info MSN
```
```json
[
{
"profile": "MSN",
"title": "MSN Chat",
"description": "MSN chat protocol (with file transfer and voice)",
"tcp_list": [
1863,
6901
],
"udp_list": [
1863,
6901
],
"tcp_ranges": [
{
"start": 6891,
"end": 6900
}
],
"normalized_tcp_list": [
1863,
6901
],
"normalized_tcp_ranges": [
{
"start": 6891,
"end": 6900
}
],
"normalized_udp_list": [
1863,
6901
]
}
]
```
### uname -a
```bash
uname -a | jc --uname -p # or: jc -p uname -a

28
README.md
View File

@ -62,10 +62,6 @@ The `jc` parsers can also be used as python modules. In this case the output wil
```
Two representations of the data are possible. The default representation uses a strict schema per parser and converts known numbers to int/float JSON values. Certain known values of `None` are converted to JSON `null`, known boolean values are converted, and, in some cases, additional semantic context fields are added.
> Note: Some parsers have calculated epoch timestamp fields added to the output. Unless a timestamp field name has a `_utc` suffix it is considered naive. (i.e. based on the local timezone of the system the `jc` parser was run on).
>
> If a UTC timezone can be detected in the text of the command output, the timestamp will be timezone aware and have a `_utc` suffix on the key name. (e.g. `epoch_utc`) No other timezones are supported for aware timestamps.
To access the raw, pre-processed JSON, use the `-r` cli option or the `raw=True` function parameter in `parse()`.
Schemas for each parser can be found at the documentation link beside each parser below.
@ -102,7 +98,7 @@ pip3 install jc
| FreeBSD | `portsnap fetch update && cd /usr/ports/textproc/py-jc && make install clean` |
| Ansible filter plugin | `ansible-galaxy collection install community.general` |
> For more packages and binaries, see https://kellyjonbrazil.github.io/jc-packaging/.
> For more packages and binaries, see the [jc packaging](https://kellyjonbrazil.github.io/jc-packaging/) site.
## Usage
`jc` accepts piped input from `STDIN` and outputs a JSON representation of the previous command's output to `STDOUT`.
@ -115,8 +111,6 @@ jc [OPTIONS] COMMAND
```
The JSON output can be compact (default) or pretty formatted with the `-p` option.
> Note: For best results set the `LANG` locale environment variable to `C`. For example, either by setting directly on the command-line: `$ LANG=C date | jc --date`, or by exporting to the environment before running commands: `$ export LANG=C`.
### Parsers
- `--acpi` enables the `acpi` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/acpi))
@ -182,6 +176,8 @@ The JSON output can be compact (default) or pretty formatted with the `-p` optio
- `--timedatectl` enables the `timedatectl status` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/timedatectl))
- `--tracepath` enables the `tracepath` and `tracepath6` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/tracepath))
- `--traceroute` enables the `traceroute` and `traceroute6` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/traceroute))
- `--ufw` enables the `ufw status` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ufw))
- `--ufw-appinfo` enables the `ufw app info [application]` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ufw_appinfo))
- `--uname` enables the `uname -a` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/uname))
- `--upower` enables the `upower` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/upower))
- `--uptime` enables the `uptime` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/uptime))
@ -230,6 +226,24 @@ Local plugin filenames must be valid python module names, therefore must consist
> Note: The application data directory follows the [XDG Base Directory Specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html)
### Caveats
**Locale:**
For best results set the `LANG` locale environment variable to `C`. For example, either by setting directly on the command-line:
```
$ LANG=C date | jc --date
```
or by exporting to the environment before running commands:
```
$ export LANG=C
```
**Timezones:**
Some parsers have calculated epoch timestamp fields added to the output. Unless a timestamp field name has a `_utc` suffix it is considered naive. (i.e. based on the local timezone of the system the `jc` parser was run on).
If a UTC timezone can be detected in the text of the command output, the timestamp will be timezone aware and have a `_utc`P suffix on the key name. (e.g. `epoch_utc`) No other timezones are supported for aware timestamps.
## Compatibility
Some parsers like `ls`, `ps`, `dig`, etc. will work on any platform. Other parsers that are platform-specific will generate a warning message if they are used on an unsupported platform. To see all parser information, including compatibility, run `jc -ap`.

1
docs/parsers/ping.md
View File

@ -97,7 +97,6 @@ Examples:
]
}
$ ping -c 3 -p ff cnn.com | jc --ping -p -r
{
"destination_ip": "151.101.129.67",

225
docs/parsers/ufw.md Normal file
View File

@ -0,0 +1,225 @@
[Home](https://kellyjonbrazil.github.io/jc/)
# jc.parsers.ufw
jc - JSON CLI output utility `ufw status` command output parser
Usage (cli):
$ ufw status | jc --ufw
or
$ jc ufw status
Usage (module):
import jc.parsers.ufw
result = jc.parsers.ufw.parse(ufw_command_output)
Schema:
{
"status": string,
"logging": string,
"logging_level": string,
"default": string,
"new_profiles": string,
"rules": [
{
"action": string,
"action_direction": string, # null if blank
"index": integer, # null if blank
"network_protocol": string,
"to_ip": string,
"to_ip_prefix": integer,
"to_interface": string,
"to_transport": string,
"to_ports": [
integer
],
"to_port_ranges": [
{
"start": integer,
"end": integer
}
],
"to_service": string, # null if any to ports or port_ranges are set
"from_ip": string,
"from_ip_prefix": integer,
"from_interface": string,
"from_transport": string,
"from_ports": [
integer
],
"from_port_ranges": [
{
"start": integer,
"end": integer
}
],
"from_service": string, # null if any from ports or port_ranges are set
"comment": string # null if no comment
}
]
}
Examples:
$ ufw status verbose | jc --ufw -p
{
"status": "active",
"logging": "on",
"logging_level": "low",
"default": "deny (incoming), allow (outgoing), disabled (routed)",
"new_profiles": "skip",
"rules": [
{
"action": "ALLOW",
"action_direction": "IN",
"index": null,
"network_protocol": "ipv4",
"to_interface": "any",
"to_transport": "any",
"to_service": null,
"to_ports": [
22
],
"to_ip": "0.0.0.0",
"to_ip_prefix": 0,
"comment": null,
"from_ip": "0.0.0.0",
"from_ip_prefix": 0,
"from_interface": "any",
"from_transport": "any",
"from_port_ranges": [
{
"start": 0,
"end": 65535
}
],
"from_service": null
},
{
"action": "ALLOW",
"action_direction": "IN",
"index": null,
"network_protocol": "ipv4",
"to_interface": "any",
"to_transport": "tcp",
"to_service": null,
"to_ports": [
80,
443
],
"to_ip": "0.0.0.0",
"to_ip_prefix": 0,
"comment": null,
"from_ip": "0.0.0.0",
"from_ip_prefix": 0,
"from_interface": "any",
"from_transport": "any",
"from_port_ranges": [
{
"start": 0,
"end": 65535
}
],
"from_service": null
},
...
]
}
$ ufw status verbose | jc --ufw -p -r
{
"status": "active",
"logging": "on",
"logging_level": "low",
"default": "deny (incoming), allow (outgoing), disabled (routed)",
"new_profiles": "skip",
"rules": [
{
"action": "ALLOW",
"action_direction": "IN",
"index": null,
"network_protocol": "ipv4",
"to_interface": "any",
"to_transport": "any",
"to_service": null,
"to_ports": [
"22"
],
"to_ip": "0.0.0.0",
"to_ip_prefix": "0",
"comment": null,
"from_ip": "0.0.0.0",
"from_ip_prefix": "0",
"from_interface": "any",
"from_transport": "any",
"from_port_ranges": [
{
"start": "0",
"end": "65535"
}
],
"from_service": null
},
{
"action": "ALLOW",
"action_direction": "IN",
"index": null,
"network_protocol": "ipv4",
"to_interface": "any",
"to_transport": "tcp",
"to_service": null,
"to_ports": [
"80",
"443"
],
"to_ip": "0.0.0.0",
"to_ip_prefix": "0",
"comment": null,
"from_ip": "0.0.0.0",
"from_ip_prefix": "0",
"from_interface": "any",
"from_transport": "any",
"from_port_ranges": [
{
"start": "0",
"end": "65535"
}
],
"from_service": null
},
...
]
}
## info
```python
info()
```
Provides parser metadata (version, author, etc.)
## parse
```python
parse(data, raw=False, quiet=False)
```
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) output preprocessed JSON if True
quiet: (boolean) suppress warning messages if True
Returns:
Dictionary. Raw or processed structured data.
## Parser Information
Compatibility: linux
Version 1.0 by Kelly Brazil (kellyjonbrazil@gmail.com)

158
docs/parsers/ufw_appinfo.md Normal file
View File

@ -0,0 +1,158 @@
[Home](https://kellyjonbrazil.github.io/jc/)
# jc.parsers.ufw_appinfo
jc - JSON CLI output utility `ufw app info [application]` command output parser
Supports individual apps via `ufw app info [application]` and all apps list via `ufw app info all`.
Because `ufw` application definitions allow overlapping ports and port ranges, this parser preserves that behavior, but also provides `normalized` lists and ranges that remove duplicate ports and merge overlapping ranges.
Usage (cli):
$ ufw app info OpenSSH | jc --ufw-appinfo
or
$ jc ufw app info OpenSSH
Usage (module):
import jc.parsers.ufw_appinfo
result = jc.parsers.ufw_appinfo.parse(ufw_appinfo_command_output)
Schema:
[
{
"profile": string,
"title": string,
"description": string,
"tcp_list": [
integer
],
"tcp_ranges": [
{
"start": integer, # 'any' is converted to start/end: 0/65535
"end": integer
}
],
"udp_list": [
integer
],
"udp_ranges": [
{
"start": integer, # 'any' is converted to start/end: 0/65535
"end": integer
}
],
"normalized_tcp_list": [
integers # duplicates and overlapping are removed
],
"normalized_tcp_ranges": [
{
"start": integer, # 'any' is converted to start/end: 0/65535
"end": integers # overlapping are merged
}
],
"normalized_udp_list": [
integers # duplicates and overlapping are removed
],
"normalized_udp_ranges": [
{
"start": integer, # 'any' is converted to start/end: 0/65535
"end": integers # overlapping are merged
}
]
}
]
Examples:
$ ufw app info MSN | jc --ufw-appinfo -p
[
{
"profile": "MSN",
"title": "MSN Chat",
"description": "MSN chat protocol (with file transfer and voice)",
"tcp_list": [
1863,
6901
],
"udp_list": [
1863,
6901
],
"tcp_ranges": [
{
"start": 6891,
"end": 6900
}
],
"normalized_tcp_list": [
1863,
6901
],
"normalized_tcp_ranges": [
{
"start": 6891,
"end": 6900
}
],
"normalized_udp_list": [
1863,
6901
]
}
]
$ ufw app info MSN | jc --ufw-appinfo -p -r
[
{
"profile": "MSN",
"title": "MSN Chat",
"description": "MSN chat protocol (with file transfer and voice)",
"tcp_list": [
"1863",
"6901"
],
"udp_list": [
"1863",
"6901"
],
"tcp_ranges": [
{
"start": "6891",
"end": "6900"
}
]
}
]
## info
```python
info()
```
Provides parser metadata (version, author, etc.)
## parse
```python
parse(data, raw=False, quiet=False)
```
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) output preprocessed JSON if True
quiet: (boolean) suppress warning messages if True
Returns:
List of Dictionaries. Raw or processed structured data.
## Parser Information
Compatibility: linux
Version 1.0 by Kelly Brazil (kellyjonbrazil@gmail.com)

4
docs/utils.md
View File

@ -74,7 +74,7 @@ Returns:
convert_to_int(value)
```
Converts string input to integer by stripping all non-numeric characters
Converts string and float input to int. Strips all non-numeric characters from strings.
Parameters:
@ -90,7 +90,7 @@ Returns:
convert_to_float(value)
```
Converts string input to float by stripping all non-numeric characters
Converts string and int input to float. Strips all non-numeric characters from strings.
Parameters:

6
jc/__init__.py
View File

@ -62,12 +62,12 @@ Module Example:
... ;; Got answer:
... ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64612
... ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
...
...
... ;; OPT PSEUDOSECTION:
... ; EDNS: version: 0, flags:; udp: 4096
... ;; QUESTION SECTION:
... ;example.com. IN A
...
...
... ;; ANSWER SECTION:
... example.com. 29658 IN A 93.184.216.34
...
@ -86,4 +86,4 @@ Module Example:
"""
name = 'jc'
__version__ = '1.15.2'
__version__ = '1.15.3'

2
jc/cli.py
View File

@ -102,6 +102,8 @@ parsers = [
'timedatectl',
'tracepath',
'traceroute',
'ufw',
'ufw-appinfo',
'uname',
'upower',
'uptime',

BIN
jc/man/jc.1.gz
View File

Binary file not shown.

1
jc/parsers/ping.py
View File

@ -94,7 +94,6 @@ Examples:
]
}
$ ping -c 3 -p ff cnn.com | jc --ping -p -r
{
"destination_ip": "151.101.129.67",

466
jc/parsers/ufw.py Normal file
View File

@ -0,0 +1,466 @@
"""jc - JSON CLI output utility `ufw status` command output parser
Usage (cli):
$ ufw status | jc --ufw
or
$ jc ufw status
Usage (module):
import jc.parsers.ufw
result = jc.parsers.ufw.parse(ufw_command_output)
Schema:
{
"status": string,
"logging": string,
"logging_level": string,
"default": string,
"new_profiles": string,
"rules": [
{
"action": string,
"action_direction": string, # null if blank
"index": integer, # null if blank
"network_protocol": string,
"to_ip": string,
"to_ip_prefix": integer,
"to_interface": string,
"to_transport": string,
"to_ports": [
integer
],
"to_port_ranges": [
{
"start": integer,
"end": integer
}
],
"to_service": string, # null if any to ports or port_ranges are set
"from_ip": string,
"from_ip_prefix": integer,
"from_interface": string,
"from_transport": string,
"from_ports": [
integer
],
"from_port_ranges": [
{
"start": integer,
"end": integer
}
],
"from_service": string, # null if any from ports or port_ranges are set
"comment": string # null if no comment
}
]
}
Examples:
$ ufw status verbose | jc --ufw -p
{
"status": "active",
"logging": "on",
"logging_level": "low",
"default": "deny (incoming), allow (outgoing), disabled (routed)",
"new_profiles": "skip",
"rules": [
{
"action": "ALLOW",
"action_direction": "IN",
"index": null,
"network_protocol": "ipv4",
"to_interface": "any",
"to_transport": "any",
"to_service": null,
"to_ports": [
22
],
"to_ip": "0.0.0.0",
"to_ip_prefix": 0,
"comment": null,
"from_ip": "0.0.0.0",
"from_ip_prefix": 0,
"from_interface": "any",
"from_transport": "any",
"from_port_ranges": [
{
"start": 0,
"end": 65535
}
],
"from_service": null
},
{
"action": "ALLOW",
"action_direction": "IN",
"index": null,
"network_protocol": "ipv4",
"to_interface": "any",
"to_transport": "tcp",
"to_service": null,
"to_ports": [
80,
443
],
"to_ip": "0.0.0.0",
"to_ip_prefix": 0,
"comment": null,
"from_ip": "0.0.0.0",
"from_ip_prefix": 0,
"from_interface": "any",
"from_transport": "any",
"from_port_ranges": [
{
"start": 0,
"end": 65535
}
],
"from_service": null
},
...
]
}
$ ufw status verbose | jc --ufw -p -r
{
"status": "active",
"logging": "on",
"logging_level": "low",
"default": "deny (incoming), allow (outgoing), disabled (routed)",
"new_profiles": "skip",
"rules": [
{
"action": "ALLOW",
"action_direction": "IN",
"index": null,
"network_protocol": "ipv4",
"to_interface": "any",
"to_transport": "any",
"to_service": null,
"to_ports": [
"22"
],
"to_ip": "0.0.0.0",
"to_ip_prefix": "0",
"comment": null,
"from_ip": "0.0.0.0",
"from_ip_prefix": "0",
"from_interface": "any",
"from_transport": "any",
"from_port_ranges": [
{
"start": "0",
"end": "65535"
}
],
"from_service": null
},
{
"action": "ALLOW",
"action_direction": "IN",
"index": null,
"network_protocol": "ipv4",
"to_interface": "any",
"to_transport": "tcp",
"to_service": null,
"to_ports": [
"80",
"443"
],
"to_ip": "0.0.0.0",
"to_ip_prefix": "0",
"comment": null,
"from_ip": "0.0.0.0",
"from_ip_prefix": "0",
"from_interface": "any",
"from_transport": "any",
"from_port_ranges": [
{
"start": "0",
"end": "65535"
}
],
"from_service": null
},
...
]
}
"""
import jc.utils
import re
import ipaddress
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.0'
description = '`ufw status` command parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
compatible = ['linux']
magic_commands = ['ufw status']
__version__ = info.version
def _process(proc_data):
"""
Final processing to conform to the schema.
Parameters:
proc_data: (List of Dictionaries) raw structured data to process
Returns:
Dictionary. Structured to conform to the schema.
"""
int_list = ['index', 'to_ip_prefix', 'from_ip_prefix']
if 'rules' in proc_data:
for i, item in enumerate(proc_data['rules']):
for key in item:
if key in int_list:
proc_data['rules'][i][key] = jc.utils.convert_to_int(proc_data['rules'][i][key])
if key in ['to_ports', 'from_ports']:
for i2, item2 in enumerate(proc_data['rules'][i][key]):
proc_data['rules'][i][key][i2] = jc.utils.convert_to_int(item2)
if key in ['to_port_ranges', 'from_port_ranges']:
for i2, item2 in enumerate(proc_data['rules'][i][key]):
proc_data['rules'][i][key][i2]['start'] = jc.utils.convert_to_int(proc_data['rules'][i][key][i2]['start'])
proc_data['rules'][i][key][i2]['end'] = jc.utils.convert_to_int(proc_data['rules'][i][key][i2]['end'])
return proc_data
def _parse_to_from(linedata, direction, rule_obj=None):
if rule_obj is None:
rule_obj = {}
# pull out rule index, if they exist: [ 1]
if direction == 'to':
RE_LINE_NUM = re.compile(r'\[[ 0-9]+\]\s')
line_number_match = re.search(RE_LINE_NUM, linedata)
if line_number_match:
rule_obj['index'] = line_number_match.group(0).replace('[', '').replace(']', '').strip()
linedata = re.sub(RE_LINE_NUM, '', linedata)
else:
rule_obj['index'] = None
# pull out comments, if they exist
if direction == 'from':
RE_COMMENT = re.compile(r'#.+$')
comment_match = re.search(RE_COMMENT, linedata)
if comment_match:
rule_obj['comment'] = comment_match.group(0).lstrip('#').strip()
linedata = re.sub(RE_COMMENT, '', linedata)
else:
rule_obj['comment'] = None
# pull (v6)
RE_V6 = re.compile(r'\(v6\)')
v6_match = re.search(RE_V6, linedata)
if v6_match:
rule_obj['network_protocol'] = 'ipv6'
linedata = re.sub(RE_V6, '', linedata)
elif not rule_obj.get('network_protocol'):
rule_obj['network_protocol'] = 'ipv4'
# pull 'Anywhere' if exists. Assign to 0.0.0.0/0 or ::/0 depending on if (v6) is found
if 'Anywhere' in linedata:
if rule_obj.get('network_protocol') == 'ipv6':
rule_obj[direction + '_ip'] = '::'
rule_obj[direction + '_ip_prefix'] = '0'
elif rule_obj.get('network_protocol') == 'ipv4':
rule_obj[direction + '_ip'] = '0.0.0.0'
rule_obj[direction + '_ip_prefix'] = '0'
linedata = linedata.replace('Anywhere', '')
# pull out interface (after 'on')
linedata_list = linedata.split(' on ', maxsplit=1)
if len(linedata_list) > 1:
rule_obj[direction + '_interface'] = linedata_list[1].strip()
linedata = linedata_list[0]
else:
rule_obj[direction + '_interface'] = 'any'
# pull tcp/udp/etc. transport - strip on '/'
linedata_list = linedata.rsplit('/', maxsplit=1)
if len(linedata_list) > 1:
if linedata_list[1].strip() in ['tcp', 'udp', 'ah', 'esp', 'gre', 'ipv6', 'igmp']:
rule_obj[direction + '_transport'] = linedata_list[1].strip()
linedata = linedata_list[0]
else:
rule_obj[direction + '_transport'] = 'any'
else:
rule_obj[direction + '_transport'] = 'any'
# pull out ipv4 or ipv6 addresses
linedata_list = linedata.split()
new_linedata_list = []
valid_ip = None
for item in linedata_list:
try:
valid_ip = ipaddress.IPv4Interface(item)
except Exception:
try:
valid_ip = ipaddress.IPv6Interface(item)
except Exception:
new_linedata_list.append(item)
if valid_ip:
rule_obj[direction + '_ip'] = str(valid_ip.ip)
rule_obj[direction + '_ip_prefix'] = str(valid_ip.with_prefixlen.split('/')[1])
linedata = ' '.join(new_linedata_list)
# find the numeric port(s)
linedata_list = linedata.split(',')
port_list = []
port_ranges = []
for item in linedata_list:
if item.strip().isnumeric():
port_list.append(item.strip())
elif ':' in item:
p_range = item.strip().split(':', maxsplit=1)
port_ranges.append(
{
"start": p_range[0],
"end": p_range[1]
}
)
if port_list or port_ranges:
rule_obj[direction + '_service'] = None
linedata = ''
if port_list:
rule_obj[direction + '_ports'] = port_list
if port_ranges:
rule_obj[direction + '_port_ranges'] = port_ranges
# only thing left should be the service name.
if linedata.strip():
rule_obj[direction + '_service'] = linedata.strip()
rule_obj[direction + '_transport'] = None
# check if to/from IP addresses exist. If not, set to 0.0.0.0/0 or ::/0
if direction + '_ip' not in rule_obj:
if rule_obj.get('network_protocol') == 'ipv6':
rule_obj[direction + '_ip'] = '::'
rule_obj[direction + '_ip_prefix'] = '0'
elif rule_obj.get('network_protocol') == 'ipv4':
rule_obj[direction + '_ip'] = '0.0.0.0'
rule_obj[direction + '_ip_prefix'] = '0'
# finally set default ports if no ports exist and there should be some
if direction + '_transport' in rule_obj:
if rule_obj[direction + '_transport'] in ['tcp', 'udp', 'any']:
if not port_list and not port_ranges:
rule_obj[direction + '_port_ranges'] = [
{
'start': '0',
'end': '65535'
}
]
rule_obj[direction + '_service'] = None
return rule_obj
def parse(data, raw=False, quiet=False):
"""
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) output preprocessed JSON if True
quiet: (boolean) suppress warning messages if True
Returns:
Dictionary. Raw or processed structured data.
"""
if not quiet:
jc.utils.compatibility(__name__, info.compatible)
raw_output = {}
rules_list = []
if jc.utils.has_data(data):
rule_lines = False
for line in filter(None, data.splitlines()):
if line.startswith('Status: '):
raw_output['status'] = line.split(': ', maxsplit=1)[1]
continue
if line.startswith('Logging: '):
log_line = line.split(': ', maxsplit=1)
log_line = log_line[1]
log_line = log_line.split()
raw_output['logging'] = log_line[0]
if len(log_line) == 2:
raw_output['logging_level'] = log_line[1].replace('(', '').replace(')', '').strip()
continue
if line.startswith('Default: '):
raw_output['default'] = line.split(': ', maxsplit=1)[1]
continue
if line.startswith('New profiles: '):
raw_output['new_profiles'] = line.split(': ', maxsplit=1)[1]
continue
if 'To' in line and 'Action' in line and 'From' in line:
rule_lines = True
continue
if rule_lines:
if '------' in line:
continue
# Split on action. Left of Action is 'to', right of Action is 'from'
rule_obj = {}
splitline = re.split(r'(ALLOW IN|ALLOW OUT|ALLOW FWD|DENY IN|DENY OUT|DENY FWD|LIMIT IN|LIMIT OUT|LIMIT FWD|REJECT IN|REJECT OUT|REJECT FWD|ALLOW|DENY|LIMIT|REJECT)', line)
to_line = splitline[0]
action_line = splitline[1]
action_list = action_line.split()
from_line = splitline[2]
action_direction = None
if len(action_list) == 1:
action = action_list[0]
elif len(action_list) == 2:
action = action_list[0]
action_direction = action_list[1]
rule_obj['action'] = action
rule_obj['action_direction'] = action_direction
rule_obj.update(_parse_to_from(to_line, 'to'))
rule_obj.update(_parse_to_from(from_line, 'from', rule_obj))
rules_list.append(rule_obj)
raw_output['rules'] = rules_list
if raw:
return raw_output
else:
return _process(raw_output)

364
jc/parsers/ufw_appinfo.py Normal file
View File

@ -0,0 +1,364 @@
"""jc - JSON CLI output utility `ufw app info [application]` command output parser
Supports individual apps via `ufw app info [application]` and all apps list via `ufw app info all`.
Because `ufw` application definitions allow overlapping ports and port ranges, this parser preserves that behavior, but also provides `normalized` lists and ranges that remove duplicate ports and merge overlapping ranges.
Usage (cli):
$ ufw app info OpenSSH | jc --ufw-appinfo
or
$ jc ufw app info OpenSSH
Usage (module):
import jc.parsers.ufw_appinfo
result = jc.parsers.ufw_appinfo.parse(ufw_appinfo_command_output)
Schema:
[
{
"profile": string,
"title": string,
"description": string,
"tcp_list": [
integer
],
"tcp_ranges": [
{
"start": integer, # 'any' is converted to start/end: 0/65535
"end": integer
}
],
"udp_list": [
integer
],
"udp_ranges": [
{
"start": integer, # 'any' is converted to start/end: 0/65535
"end": integer
}
],
"normalized_tcp_list": [
integers # duplicates and overlapping are removed
],
"normalized_tcp_ranges": [
{
"start": integer, # 'any' is converted to start/end: 0/65535
"end": integers # overlapping are merged
}
],
"normalized_udp_list": [
integers # duplicates and overlapping are removed
],
"normalized_udp_ranges": [
{
"start": integer, # 'any' is converted to start/end: 0/65535
"end": integers # overlapping are merged
}
]
}
]
Examples:
$ ufw app info MSN | jc --ufw-appinfo -p
[
{
"profile": "MSN",
"title": "MSN Chat",
"description": "MSN chat protocol (with file transfer and voice)",
"tcp_list": [
1863,
6901
],
"udp_list": [
1863,
6901
],
"tcp_ranges": [
{
"start": 6891,
"end": 6900
}
],
"normalized_tcp_list": [
1863,
6901
],
"normalized_tcp_ranges": [
{
"start": 6891,
"end": 6900
}
],
"normalized_udp_list": [
1863,
6901
]
}
]
$ ufw app info MSN | jc --ufw-appinfo -p -r
[
{
"profile": "MSN",
"title": "MSN Chat",
"description": "MSN chat protocol (with file transfer and voice)",
"tcp_list": [
"1863",
"6901"
],
"udp_list": [
"1863",
"6901"
],
"tcp_ranges": [
{
"start": "6891",
"end": "6900"
}
]
}
]
"""
import jc.utils
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.0'
description = '`ufw app info [application]` command parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
compatible = ['linux']
magic_commands = ['ufw app']
__version__ = info.version
def _process(proc_data):
"""
Final processing to conform to the schema.
Parameters:
proc_data: (List of Dictionaries) raw structured data to process
Returns:
List of Dictionaries. Structured to conform to the schema.
"""
for profile in proc_data:
# convert to ints
int_list = ['start', 'end']
if 'tcp_list' in profile:
profile['tcp_list'] = [int(p) for p in profile['tcp_list']]
if 'udp_list' in profile:
profile['udp_list'] = [int(p) for p in profile['udp_list']]
for protocol in ['tcp', 'udp']:
if protocol + '_ranges' in profile:
for i, item in enumerate(profile[protocol + '_ranges']):
for key in item:
if key in int_list:
profile[protocol + '_ranges'][i][key] = int(profile[protocol + '_ranges'][i][key])
# create normalized port lists and port ranges (remove duplicates and merge ranges)
# dump ranges into a set of 0 - 65535
# if items in the port list are in the set, then remove them
# iterate through the set to find gaps and create new ranges based on them
for protocol in ['tcp', 'udp']:
port_set = set()
if protocol + '_ranges' in profile:
for item in profile[protocol + '_ranges']:
port_set.update(range(item['start'], item['end'] + 1))
if protocol + '_list' in profile:
new_port_list = sorted(set([p for p in profile[protocol + '_list'] if p not in port_set]))
if new_port_list:
profile['normalized_' + protocol + '_list'] = new_port_list
new_port_ranges = []
state = 'findstart' # 'findstart' or 'findend'
for port in range(0, 65535 + 2):
if state == 'findstart':
port_range_obj = {}
if port in port_set:
port_range_obj['start'] = port
state = 'findend'
continue
if state == 'findend':
if port not in port_set:
port_range_obj['end'] = port - 1
new_port_ranges.append(port_range_obj)
state = 'findstart'
if new_port_ranges:
profile['normalized_' + protocol + '_ranges'] = new_port_ranges
return proc_data
def _parse_port_list(data, port_list=None):
"""return a list of port strings"""
# 1,2,3,4,5,6,7,8,9,10,9,30,80:90,8080:8090
# overlapping and repeated port numbers are allowed
if port_list is None:
port_list = []
data = data.split(',')
data_list = [p.strip() for p in data if ':' not in p and 'any' not in p]
port_list.extend(data_list)
return port_list
def _parse_port_range(data, range_list=None):
"""return a list of dictionaries"""
# 1,2,3,4,5,6,7,8,9,10,9,30,80:90,8080:8090
# overlapping port ranges are allowed
if range_list is None:
range_list = []
data = data.strip().split(',')
ranges = [p.strip() for p in data if ':' in p]
range_obj = {}
if 'any' in data:
range_list.append(
{
'start': 0,
'end': 65535
}
)
for range_ in ranges:
range_obj = {
'start': range_.split(':')[0],
'end': range_.split(':')[1]
}
range_list.append(range_obj)
return range_list
def parse(data, raw=False, quiet=False):
"""
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) output preprocessed JSON if True
quiet: (boolean) suppress warning messages if True
Returns:
List of Dictionaries. Raw or processed structured data.
"""
if not quiet:
jc.utils.compatibility(__name__, info.compatible)
raw_output = []
item_obj = {}
if jc.utils.has_data(data):
ports = False
for line in filter(None, data.splitlines()):
if line.startswith('--'):
if item_obj:
raw_output.append(item_obj)
item_obj = {}
continue
if line.startswith('Profile:'):
item_obj['profile'] = line.split(': ')[1]
continue
if line.startswith('Title:'):
item_obj['title'] = line.split(': ')[1]
continue
if line.startswith('Description:'):
item_obj['description'] = line.split(': ')[1]
continue
if line.startswith('Port'):
ports = True
continue
if ports:
line_list = line.rsplit('/', maxsplit=1)
if len(line_list) == 2:
if line_list[1] == 'tcp':
tcp_prot_list = _parse_port_list(line_list[0])
if tcp_prot_list:
item_obj['tcp_list'] = tcp_prot_list
tcp_prot_range = _parse_port_range(line_list[0])
if tcp_prot_range:
item_obj['tcp_ranges'] = tcp_prot_range
elif line_list[1] == 'udp':
udp_prot_list = _parse_port_list(line_list[0])
if udp_prot_list:
item_obj['udp_list'] = udp_prot_list
udp_prot_range = _parse_port_range(line_list[0])
if udp_prot_range:
item_obj['udp_ranges'] = udp_prot_range
# 'any' case
else:
t_list = []
t_range = []
u_list = []
u_range = []
if 'tcp_list' in item_obj:
t_list = item_obj['tcp_list']
if 'tcp_ranges' in item_obj:
t_range = item_obj['tcp_ranges']
if 'udp_list' in item_obj:
u_list = item_obj['udp_list']
if 'udp_ranges' in item_obj:
u_range = item_obj['udp_ranges']
t_p_list = _parse_port_list(line, t_list)
if t_p_list:
item_obj['tcp_list'] = t_p_list
t_r_list = _parse_port_range(line, t_range)
if t_r_list:
item_obj['tcp_ranges'] = t_r_list
u_p_list = _parse_port_list(line, u_list)
if u_p_list:
item_obj['udp_list'] = u_p_list
u_r_list = _parse_port_range(line, u_range)
if u_r_list:
item_obj['udp_ranges'] = u_r_list
if item_obj:
raw_output.append(item_obj)
if raw:
return raw_output
else:
return _process(raw_output)

11
jc/utils.py
View File

@ -85,7 +85,7 @@ def has_data(data):
def convert_to_int(value):
"""
Converts string input to integer by stripping all non-numeric characters
Converts string and float input to int. Strips all non-numeric characters from strings.
Parameters:
@ -96,11 +96,12 @@ def convert_to_int(value):
integer/None Integer if successful conversion, otherwise None
"""
if isinstance(value, str):
str_val = re.sub(r'[^0-9\-\.]', '', value)
try:
return int(re.sub(r'[^0-9\-\.]', '', value))
except ValueError:
return int(str_val)
except (ValueError, TypeError):
try:
return int(convert_to_float(value))
return int(float(str_val))
except (ValueError, TypeError):
return None
@ -113,7 +114,7 @@ def convert_to_int(value):
def convert_to_float(value):
"""
Converts string input to float by stripping all non-numeric characters
Converts string and int input to float. Strips all non-numeric characters from strings.
Parameters:

BIN
man/jc.1.gz
View File

Binary file not shown.

1
runtests.sh
View File

@ -1,3 +1,4 @@
#!/bin/bash
# system should be in "America/Los_Angeles" timezone for all tests to pass
python3 -m unittest -v

2
setup.py
View File

@ -5,7 +5,7 @@ with open('README.md', 'r') as f:
setuptools.setup(
name='jc',
version='1.15.2',
version='1.15.3',
author='Kelly Brazil',
author_email='kellyjonbrazil@gmail.com',
description='Converts the output of popular command-line tools and file-types to JSON.',

13
templates/manpage_template
View File

@ -96,6 +96,19 @@ Local plugin filenames must be valid python module names, therefore must consist
Note: The application data directory follows the XDG Base Directory Specification
.SH CAVEATS
\fBLocale:\fP For best results set the \fBLANG\fP locale environment variable to \fBC\fP. For example, either by setting directly on the command-line:
\fB$ LANG=C date | jc --date\fP
or by exporting to the environment before running commands:
\fB$ export LANG=C\fP
\fBTimezones:\fP Some parsers have calculated epoch timestamp fields added to the output. Unless a timestamp field name has a \fB_utc\fP suffix it is considered naive. (i.e. based on the local timezone of the system the \fBjc\fP parser was run on).
If a UTC timezone can be detected in the text of the command output, the timestamp will be timezone aware and have a \fB_utc\fP suffix on the key name. (e.g. \fBepoch_utc\fP) No other timezones are supported for aware timestamps.
.SH EXAMPLES
Standard Syntax:
.RS

26
templates/readme_template
View File

@ -62,10 +62,6 @@ The `jc` parsers can also be used as python modules. In this case the output wil
```
Two representations of the data are possible. The default representation uses a strict schema per parser and converts known numbers to int/float JSON values. Certain known values of `None` are converted to JSON `null`, known boolean values are converted, and, in some cases, additional semantic context fields are added.
> Note: Some parsers have calculated epoch timestamp fields added to the output. Unless a timestamp field name has a `_utc` suffix it is considered naive. (i.e. based on the local timezone of the system the `jc` parser was run on).
>
> If a UTC timezone can be detected in the text of the command output, the timestamp will be timezone aware and have a `_utc` suffix on the key name. (e.g. `epoch_utc`) No other timezones are supported for aware timestamps.
To access the raw, pre-processed JSON, use the `-r` cli option or the `raw=True` function parameter in `parse()`.
Schemas for each parser can be found at the documentation link beside each parser below.
@ -102,7 +98,7 @@ pip3 install jc
| FreeBSD | `portsnap fetch update && cd /usr/ports/textproc/py-jc && make install clean` |
| Ansible filter plugin | `ansible-galaxy collection install community.general` |
> For more packages and binaries, see https://kellyjonbrazil.github.io/jc-packaging/.
> For more packages and binaries, see the [jc packaging](https://kellyjonbrazil.github.io/jc-packaging/) site.
## Usage
`jc` accepts piped input from `STDIN` and outputs a JSON representation of the previous command's output to `STDOUT`.
@ -115,8 +111,6 @@ jc [OPTIONS] COMMAND
```
The JSON output can be compact (default) or pretty formatted with the `-p` option.
> Note: For best results set the `LANG` locale environment variable to `C`. For example, either by setting directly on the command-line: `$ LANG=C date | jc --date`, or by exporting to the environment before running commands: `$ export LANG=C`.
### Parsers
{% for parser in jc.parsers %}
- `{{ parser.argument }}` enables the {{ parser.description }} ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/{{ parser.name }})){% endfor %}
@ -160,6 +154,24 @@ Local plugin filenames must be valid python module names, therefore must consist
> Note: The application data directory follows the [XDG Base Directory Specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html)
### Caveats
**Locale:**
For best results set the `LANG` locale environment variable to `C`. For example, either by setting directly on the command-line:
```
$ LANG=C date | jc --date
```
or by exporting to the environment before running commands:
```
$ export LANG=C
```
**Timezones:**
Some parsers have calculated epoch timestamp fields added to the output. Unless a timestamp field name has a `_utc` suffix it is considered naive. (i.e. based on the local timezone of the system the `jc` parser was run on).
If a UTC timezone can be detected in the text of the command output, the timestamp will be timezone aware and have a `_utc`P suffix on the key name. (e.g. `epoch_utc`) No other timezones are supported for aware timestamps.
## Compatibility
Some parsers like `ls`, `ps`, `dig`, etc. will work on any platform. Other parsers that are platform-specific will generate a warning message if they are used on an unsupported platform. To see all parser information, including compatibility, run `jc -ap`.

1
tests/fixtures/generic/ufw-appinfo-msn.json vendored Normal file
View File

@ -0,0 +1 @@
[{"profile":"MSN","title":"MSN Chat","description":"MSN chat protocol (with file transfer and voice)","tcp_list":[1863,6901],"udp_list":[1863,6901],"tcp_ranges":[{"start":6891,"end":6900}],"normalized_tcp_list":[1863,6901],"normalized_tcp_ranges":[{"start":6891,"end":6900}],"normalized_udp_list":[1863,6901]}]

9
tests/fixtures/generic/ufw-appinfo-msn.out vendored Normal file
View File

@ -0,0 +1,9 @@
Profile: MSN
Title: MSN Chat
Description: MSN chat protocol (with file transfer and voice)
Ports:
1863
6891:6900/tcp
6901

1
tests/fixtures/generic/ufw-appinfo-test.json vendored Normal file
View File

@ -0,0 +1 @@
[{"profile":"TEST","title":"My test app","description":"a longer description of the test app here.","tcp_list":[1,2,3,4,5,6,7,8,9,10,9,8,7,30,53],"tcp_ranges":[{"start":80,"end":90},{"start":8080,"end":8090}],"udp_ranges":[{"start":50,"end":51},{"start":40,"end":60}],"udp_list":[53],"normalized_tcp_list":[1,2,3,4,5,6,7,8,9,10,30,53],"normalized_tcp_ranges":[{"start":80,"end":90},{"start":8080,"end":8090}],"normalized_udp_ranges":[{"start":40,"end":60}]}]

9
tests/fixtures/generic/ufw-appinfo-test.out vendored Normal file
View File

@ -0,0 +1,9 @@
Profile: TEST
Title: My test app
Description: a longer description of the test app here.
Ports:
1,2,3,4,5,6,7,8,9,10,9,8,7,30,80:90,8080:8090/tcp
50:51,40:60/udp
53

1
tests/fixtures/generic/ufw-appinfo-test2.json vendored Normal file
View File

@ -0,0 +1 @@
[{"profile":"TEST2","title":"My test app2","description":"a longer description of the test app here.","tcp_ranges":[{"start":0,"end":65535}],"udp_ranges":[{"start":50,"end":51}],"tcp_list":[53],"udp_list":[53],"normalized_tcp_ranges":[{"start":0,"end":65535}],"normalized_udp_list":[53],"normalized_udp_ranges":[{"start":50,"end":51}]}]

9
tests/fixtures/generic/ufw-appinfo-test2.out vendored Normal file
View File

@ -0,0 +1,9 @@
Profile: TEST2
Title: My test app2
Description: a longer description of the test app here.
Ports:
any/tcp
50:51/udp
53

1
tests/fixtures/generic/ufw-appinfo-test3.json vendored Normal file
View File

@ -0,0 +1 @@
[{"profile":"TEST3","title":"My test app3","description":"test overlapping ports","tcp_list":[80,83,80,53],"tcp_ranges":[{"start":70,"end":90}],"udp_ranges":[{"start":50,"end":51}],"udp_list":[53],"normalized_tcp_list":[53],"normalized_tcp_ranges":[{"start":70,"end":90}],"normalized_udp_list":[53],"normalized_udp_ranges":[{"start":50,"end":51}]}]

9
tests/fixtures/generic/ufw-appinfo-test3.out vendored Normal file
View File

@ -0,0 +1,9 @@
Profile: TEST3
Title: My test app3
Description: test overlapping ports
Ports:
80,83,80,70:90/tcp
50:51/udp
53

1
tests/fixtures/generic/ufw-inactive.json vendored Normal file
View File

@ -0,0 +1 @@
{"status":"inactive","rules":[]}

1
tests/fixtures/generic/ufw-inactive.out vendored Normal file
View File

@ -0,0 +1 @@
Status: inactive

1
tests/fixtures/generic/ufw-numbered.json vendored Normal file
View File

File diff suppressed because one or more lines are too long

23
tests/fixtures/generic/ufw-numbered.out vendored Normal file
View File

@ -0,0 +1,23 @@
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), deny (routed)
New profiles: skip
To Action From
-- ------ ----
[ 1] 22/tcp ALLOW IN Anywhere
[ 2] 22/tcp (v6) ALLOW OUT Anywhere (v6)
[ 3] 443/tcp DENY 192.168.0.1
[ 4] 443/udp DENY OUT 192.168.0.7 8080:8081
[ 5] 22/tcp ALLOW 192.168.0.0/24
[ 6] 22/udp ALLOW 192.168.0.0/24 8080:8081 on en0
[ 7] 22/tcp (v6) ALLOW IN 2405:204:7449:49fc:f09a:6f4a:bc93:1955/64 on en1
[ 8] 80 ALLOW IN Anywhere
[ 9] 8080 (v6) ALLOW IN Anywhere (v6)
[10] Apache Full ALLOW IN Anywhere
[11] Apache Full (v6) ALLOW IN Anywhere (v6)
[12] OpenSSH (v6) DENY IN Anywhere (v6)
[13] 10.10.10.10 8080 on enp34s0 ALLOW 127.0.0.1 8000
[14] 50200:50300/tcp (v6) ALLOW Anywhere (v6)
[15] Anywhere (v6) ALLOW IN 2405:204:7449:49fc:f09a:6f4a:bc93:1955

1
tests/fixtures/generic/ufw-numbered2.json vendored Normal file
View File

File diff suppressed because one or more lines are too long

18
tests/fixtures/generic/ufw-numbered2.out vendored Normal file
View File

@ -0,0 +1,18 @@
Status: active
To Action From
-- ------ ----
[ 1] 224.0.0.251 mDNS ALLOW IN Anywhere
[ 2] Anywhere ALLOW IN 123.123.123.123
[ 3] 25 ALLOW IN Anywhere
[ 4] 80 ALLOW IN Anywhere
[ 5] 443 ALLOW IN Anywhere
[ 6] 465 ALLOW IN Anywhere
[ 7] 993 ALLOW IN Anywhere
[ 8] 995 ALLOW IN Anywhere
[ 9] ff02::fb mDNS ALLOW IN Anywhere (v6)
[10] 25 (v6) ALLOW IN Anywhere (v6)
[11] 80 (v6) ALLOW IN Anywhere (v6)
[12] 443 (v6) ALLOW IN Anywhere (v6)
[13] 465 (v6) ALLOW IN Anywhere (v6)
[14] 993 (v6) ALLOW IN Anywhere (v6)
[15] 995 (v6) ALLOW IN Anywhere (v6)

1
tests/fixtures/generic/ufw.json vendored Normal file
View File

File diff suppressed because one or more lines are too long

22
tests/fixtures/generic/ufw.out vendored Normal file
View File

@ -0,0 +1,22 @@
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), deny (routed)
New profiles: skip
To Action From
-- ------ ----
22/tcp ALLOW IN Anywhere
22/tcp (v6) ALLOW OUT Anywhere (v6)
443/tcp DENY 192.168.0.1 # nice comment
443/udp DENY OUT 192.168.0.7 8080:8081
22/tcp ALLOW 192.168.0.0/24
22/udp ALLOW 192.168.0.0/24 8080:8081 on en0
22/tcp (v6) ALLOW FWD 2405:204:7449:49fc:f09a:6f4a:bc93:1955/64 on en1 #commenting this rule
80 ALLOW IN Anywhere
8080 (v6) REJECT IN Anywhere (v6)
Apache Full ALLOW IN Anywhere # a comment
Apache Full (v6) ALLOW IN Anywhere (v6)
OpenSSH (v6) DENY IN Anywhere (v6)
10.10.10.10 8080 on enp34s0 ALLOW 127.0.0.1 8000
50200:50300/tcp (v6) DENY FWD Anywhere (v6)
Anywhere (v6) LIMIT 2405:204:7449:49fc:f09a:6f4a:bc93:1955 # this is a comment

1
tests/fixtures/ubuntu-18.04/ufw-appinfo-all.json vendored Normal file
View File

@ -0,0 +1 @@
[{"profile":"MSN","title":"MSN Chat","description":"MSN chat protocol (with file transfer and voice)","tcp_list":[1863,6901],"udp_list":[1863,6901],"tcp_ranges":[{"start":6891,"end":6900}],"normalized_tcp_list":[1863,6901],"normalized_tcp_ranges":[{"start":6891,"end":6900}],"normalized_udp_list":[1863,6901]},{"profile":"OpenSSH","title":"Secure shell server, an rshd replacement","description":"OpenSSH is a free implementation of the Secure Shell protocol.","tcp_list":[22],"normalized_tcp_list":[22]},{"profile":"TEST","title":"My test app","description":"a longer description of the test app here.","tcp_list":[1,2,3,4,5,6,7,8,9,10,30,53],"tcp_ranges":[{"start":80,"end":90},{"start":8080,"end":8090}],"udp_ranges":[{"start":50,"end":51}],"udp_list":[53],"normalized_tcp_list":[1,2,3,4,5,6,7,8,9,10,30,53],"normalized_tcp_ranges":[{"start":80,"end":90},{"start":8080,"end":8090}],"normalized_udp_list":[53],"normalized_udp_ranges":[{"start":50,"end":51}]},{"profile":"TEST2","title":"My test app2","description":"a longer description of the test app here.","tcp_ranges":[{"start":0,"end":65535}],"udp_ranges":[{"start":50,"end":51}],"tcp_list":[53],"udp_list":[53],"normalized_tcp_ranges":[{"start":0,"end":65535}],"normalized_udp_list":[53],"normalized_udp_ranges":[{"start":50,"end":51}]},{"profile":"TEST3","title":"My test app3","description":"test overlapping ports","tcp_list":[80,83,80,53],"tcp_ranges":[{"start":70,"end":90}],"udp_ranges":[{"start":50,"end":51}],"udp_list":[53],"normalized_tcp_list":[53],"normalized_tcp_ranges":[{"start":70,"end":90}],"normalized_udp_list":[53],"normalized_udp_ranges":[{"start":50,"end":51}]}]

51
tests/fixtures/ubuntu-18.04/ufw-appinfo-all.out vendored Normal file
View File

@ -0,0 +1,51 @@
Profile: MSN
Title: MSN Chat
Description: MSN chat protocol (with file transfer and voice)
Ports:
1863
6891:6900/tcp
6901
--
Profile: OpenSSH
Title: Secure shell server, an rshd replacement
Description: OpenSSH is a free implementation of the Secure Shell protocol.
Port:
22/tcp
--
Profile: TEST
Title: My test app
Description: a longer description of the test app here.
Ports:
1,2,3,4,5,6,7,8,9,10,30,80:90,8080:8090/tcp
50:51/udp
53
--
Profile: TEST2
Title: My test app2
Description: a longer description of the test app here.
Ports:
any/tcp
50:51/udp
53
--
Profile: TEST3
Title: My test app3
Description: test overlapping ports
Ports:
80,83,80,70:90/tcp
50:51/udp
53

1
tests/fixtures/ubuntu-18.04/ufw-numbered.json vendored Normal file
View File

File diff suppressed because one or more lines are too long

19
tests/fixtures/ubuntu-18.04/ufw-numbered.out vendored Normal file
View File

@ -0,0 +1,19 @@
Status: active
To Action From
-- ------ ----
[ 1] 22 ALLOW IN Anywhere
[ 2] 80,443/tcp ALLOW IN Anywhere
[ 3] 80 on eth0 ALLOW IN Anywhere # test
[ 4] 10.0.0.1/ipv6 ALLOW IN 10.4.0.0/16/ipv6
[ 5] 10.0.0.1/esp ALLOW IN Anywhere
[ 6] 10.0.0.1/esp ALLOW IN 10.4.0.0/16/esp
[ 7] 10.0.0.1/ah ALLOW IN Anywhere
[ 8] 10.0.0.1/ah ALLOW IN 10.4.0.0/16/ah
[ 9] 100:200,300:400/tcp ALLOW IN Anywhere
[10] 1,2,100:200,300:400/udp ALLOW IN Anywhere
[11] 22 (v6) ALLOW IN Anywhere (v6)
[12] 80,443/tcp (v6) ALLOW IN Anywhere (v6)
[13] 80 (v6) on eth0 ALLOW IN Anywhere (v6) # test
[14] 100:200,300:400/tcp (v6) ALLOW IN Anywhere (v6)
[15] 1,2,100:200,300:400/udp (v6) ALLOW IN Anywhere (v6)

1
tests/fixtures/ubuntu-18.04/ufw-verbose.json vendored Normal file
View File

File diff suppressed because one or more lines are too long

22
tests/fixtures/ubuntu-18.04/ufw-verbose.out vendored Normal file
View File

@ -0,0 +1,22 @@
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), disabled (routed)
New profiles: skip
To Action From
-- ------ ----
22 ALLOW IN Anywhere
80,443/tcp ALLOW IN Anywhere
80 on eth0 ALLOW IN Anywhere # test
10.0.0.1/ipv6 ALLOW IN 10.4.0.0/16/ipv6
10.0.0.1/esp ALLOW IN Anywhere
10.0.0.1/esp ALLOW IN 10.4.0.0/16/esp
10.0.0.1/ah ALLOW IN Anywhere
10.0.0.1/ah ALLOW IN 10.4.0.0/16/ah
100:200,300:400/tcp ALLOW IN Anywhere
1,2,100:200,300:400/udp ALLOW IN Anywhere
22 (v6) ALLOW IN Anywhere (v6)
80,443/tcp (v6) ALLOW IN Anywhere (v6)
80 (v6) on eth0 ALLOW IN Anywhere (v6) # test
100:200,300:400/tcp (v6) ALLOW IN Anywhere (v6)
1,2,100:200,300:400/udp (v6) ALLOW IN Anywhere (v6)

94
tests/test_ufw.py Normal file
View File

@ -0,0 +1,94 @@
import os
import json
import unittest
import jc.parsers.ufw
THIS_DIR = os.path.dirname(os.path.abspath(__file__))
class MyTests(unittest.TestCase):
def setUp(self):
# input
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/ubuntu-18.04/ufw-verbose.out'), 'r', encoding='utf-8') as f:
self.ubuntu_18_04_ufw_verbose = f.read()
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/ubuntu-18.04/ufw-numbered.out'), 'r', encoding='utf-8') as f:
self.ubuntu_18_04_ufw_numbered = f.read()
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/generic/ufw.out'), 'r', encoding='utf-8') as f:
self.generic_ufw = f.read()
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/generic/ufw-numbered.out'), 'r', encoding='utf-8') as f:
self.generic_ufw_numbered = f.read()
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/generic/ufw-numbered2.out'), 'r', encoding='utf-8') as f:
self.generic_ufw_numbered2 = f.read()
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/generic/ufw-inactive.out'), 'r', encoding='utf-8') as f:
self.generic_ufw_inactive = f.read()
# output
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/ubuntu-18.04/ufw-verbose.json'), 'r', encoding='utf-8') as f:
self.ubuntu_18_04_ufw_verbose_json = json.loads(f.read())
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/ubuntu-18.04/ufw-numbered.json'), 'r', encoding='utf-8') as f:
self.ubuntu_18_04_ufw_numbered_json = json.loads(f.read())
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/generic/ufw.json'), 'r', encoding='utf-8') as f:
self.generic_ufw_json = json.loads(f.read())
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/generic/ufw-numbered.json'), 'r', encoding='utf-8') as f:
self.generic_ufw_numbered_json = json.loads(f.read())
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/generic/ufw-numbered2.json'), 'r', encoding='utf-8') as f:
self.generic_ufw_numbered2_json = json.loads(f.read())
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/generic/ufw-inactive.json'), 'r', encoding='utf-8') as f:
self.generic_ufw_inactive_json = json.loads(f.read())
def test_ufw_nodata(self):
"""
Test 'ufw' with no data
"""
self.assertEqual(jc.parsers.ufw.parse('', quiet=True), {})
def test_ufw_ubuntu_18_04_verbose(self):
"""
Test 'ufw status verbose' on Ubuntu 18.04
"""
self.assertEqual(jc.parsers.ufw.parse(self.ubuntu_18_04_ufw_verbose, quiet=True), self.ubuntu_18_04_ufw_verbose_json)
def test_ufw_ubuntu_18_04_numbered(self):
"""
Test 'ufw status numbered' on Ubuntu 18.04
"""
self.assertEqual(jc.parsers.ufw.parse(self.ubuntu_18_04_ufw_numbered, quiet=True), self.ubuntu_18_04_ufw_numbered_json)
def test_ufw_generic_verbose(self):
"""
Test 'ufw status verbose' sample
"""
self.assertEqual(jc.parsers.ufw.parse(self.generic_ufw, quiet=True), self.generic_ufw_json)
def test_ufw_generic_verbose_numbered(self):
"""
Test 'ufw status verbose numbered' sample
"""
self.assertEqual(jc.parsers.ufw.parse(self.generic_ufw_numbered, quiet=True), self.generic_ufw_numbered_json)
def test_ufw_generic_verbose_numbered2(self):
"""
Test 'ufw status verbose numbered' sample
"""
self.assertEqual(jc.parsers.ufw.parse(self.generic_ufw_numbered2, quiet=True), self.generic_ufw_numbered2_json)
def test_ufw_generic_inactive(self):
"""
Test 'ufw status' when firewall is inactive
"""
self.assertEqual(jc.parsers.ufw.parse(self.generic_ufw_inactive, quiet=True), self.generic_ufw_inactive_json)
if __name__ == '__main__':
unittest.main()

82
tests/test_ufw_appinfo.py Normal file
View File

@ -0,0 +1,82 @@
import os
import json
import unittest
import jc.parsers.ufw_appinfo
THIS_DIR = os.path.dirname(os.path.abspath(__file__))
class MyTests(unittest.TestCase):
def setUp(self):
# input
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/ubuntu-18.04/ufw-appinfo-all.out'), 'r', encoding='utf-8') as f:
self.ubuntu_18_04_ufw_appinfo_all = f.read()
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/generic/ufw-appinfo-test.out'), 'r', encoding='utf-8') as f:
self.generic_ufw_appinfo_test = f.read()
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/generic/ufw-appinfo-test2.out'), 'r', encoding='utf-8') as f:
self.generic_ufw_appinfo_test2 = f.read()
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/generic/ufw-appinfo-test3.out'), 'r', encoding='utf-8') as f:
self.generic_ufw_appinfo_test3 = f.read()
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/generic/ufw-appinfo-msn.out'), 'r', encoding='utf-8') as f:
self.generic_ufw_appinfo_msn = f.read()
# output
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/ubuntu-18.04/ufw-appinfo-all.json'), 'r', encoding='utf-8') as f:
self.ubuntu_18_04_ufw_appinfo_all_json = json.loads(f.read())
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/generic/ufw-appinfo-test.json'), 'r', encoding='utf-8') as f:
self.generic_ufw_appinfo_test_json = json.loads(f.read())
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/generic/ufw-appinfo-test2.json'), 'r', encoding='utf-8') as f:
self.generic_ufw_appinfo_test2_json = json.loads(f.read())
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/generic/ufw-appinfo-test3.json'), 'r', encoding='utf-8') as f:
self.generic_ufw_appinfo_test3_json = json.loads(f.read())
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/generic/ufw-appinfo-msn.json'), 'r', encoding='utf-8') as f:
self.generic_ufw_appinfo_msn_json = json.loads(f.read())
def test_ufw_appinfo_nodata(self):
"""
Test 'ufw_appinfo' with no data
"""
self.assertEqual(jc.parsers.ufw_appinfo.parse('', quiet=True), [])
def test_ufw_appinfo_ubuntu_18_04_all(self):
"""
Test 'ufw app info all' on Ubuntu 18.04
"""
self.assertEqual(jc.parsers.ufw_appinfo.parse(self.ubuntu_18_04_ufw_appinfo_all, quiet=True), self.ubuntu_18_04_ufw_appinfo_all_json)
def test_ufw_appinfo_generic_test(self):
"""
Test 'ufw app info [application]' sample
"""
self.assertEqual(jc.parsers.ufw_appinfo.parse(self.generic_ufw_appinfo_test, quiet=True), self.generic_ufw_appinfo_test_json)
def test_ufw_appinfo_generic_test2(self):
"""
Test 'ufw app info [application]' sample
"""
self.assertEqual(jc.parsers.ufw_appinfo.parse(self.generic_ufw_appinfo_test2, quiet=True), self.generic_ufw_appinfo_test2_json)
def test_ufw_appinfo_generic_test3(self):
"""
Test 'ufw app info [application]' sample
"""
self.assertEqual(jc.parsers.ufw_appinfo.parse(self.generic_ufw_appinfo_test3, quiet=True), self.generic_ufw_appinfo_test3_json)
def test_ufw_appinfo_generic_msn(self):
"""
Test 'ufw app info MSN' sample
"""
self.assertEqual(jc.parsers.ufw_appinfo.parse(self.generic_ufw_appinfo_msn, quiet=True), self.generic_ufw_appinfo_msn_json)
if __name__ == '__main__':
unittest.main()

6
tests/test_utils.py
View File

@ -45,7 +45,7 @@ class MyTests(unittest.TestCase):
for input_string, expected_output in datetime_map.items():
self.assertEqual(jc.utils.timestamp(input_string).__dict__, expected_output)
def test_convert_to_int(self):
def test_utils_convert_to_int(self):
io_map = {
None: None,
True: 1,
@ -72,7 +72,7 @@ class MyTests(unittest.TestCase):
for input_string, expected_output in io_map.items():
self.assertEqual(jc.utils.convert_to_int(input_string), expected_output)
def test_convert_to_float(self):
def test_utils_convert_to_float(self):
io_map = {
None: None,
True: 1.0,
@ -99,7 +99,7 @@ class MyTests(unittest.TestCase):
for input_string, expected_output in io_map.items():
self.assertEqual(jc.utils.convert_to_float(input_string), expected_output)
def test_convert_to_bool(self):
def test_utils_convert_to_bool(self):
io_map = {
None: False,
True: True,