python/jc
1
0
Fork 0
mirror of https://github.com/kellyjonbrazil/jc.git synced 2025-06-19 00:17:51 +02:00
Code Issues Releases Activity

Merge pull request #127 from kellyjonbrazil/dev

Browse Source 
Dev v1.15.3
This commit is contained in:
Kelly Brazil
2021-04-26 12:12:02 -07:00
committed by GitHub
parent fcc7e52949 35464bbbfb
commit fd7861db11
45 changed files with 1784 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
CHANGELOG
View File

@ -1,5 +1,11 @@
jc changelog jc changelog
20210426 v1.15.3
- Add ufw status command parser tested on linux
- Add ufw-appinfo command parser tested on linux
- Fix deb package name to conform to standard
- Add Caveats section to readme and manpage
20210418 v1.15.2 20210418 v1.15.2
- Add systeminfo parser tested on Windows - Add systeminfo parser tested on Windows
- Update dig parser to fix an issue with IPv6 addresses in the server field - Update dig parser to fix an issue with IPv6 addresses in the server field

7
CONTRIBUTING.md
View File

@ -17,7 +17,7 @@ Pull requests are the best way to propose changes to the codebase (we use [Githu
2. Fork the repo and create your branch from `dev`, if available, otherwise `master`. 2. Fork the repo and create your branch from `dev`, if available, otherwise `master`.
3. If you've added code that should be tested, add tests. All new parsers should have several sample outputs and tests. 3. If you've added code that should be tested, add tests. All new parsers should have several sample outputs and tests.
4. Documentation is auto-generated from docstrings, so ensure they are clear and accurate. 4. Documentation is auto-generated from docstrings, so ensure they are clear and accurate.
5. Ensure the test suite passes. 5. Ensure the test suite passes. (Note: "**America/Los_Angeles**" timezone should be configured on the test system)
6. Make sure your code lints. 6. Make sure your code lints.
7. Issue that pull request! 7. Issue that pull request!
@ -61,6 +61,11 @@ Good:
] ]
``` ```
## Tests
It is essential to have good command output sample coverage and tests to keep the `jc` parser quality high.
Many parsers include calculated timestamp fields using the `jc.utils.timestamp` class. Naive timestamps created with this class should be generated on a system configured with the "**America/Los_Angeles**" timezone on linux/macOS/unix and "**Pacific Standard Time**" timezone on Windows for tests to pass on the Github Actions CI tests. This timezone should be configured on your local system before running the tests locally, as well.
## Any contributions you make will be under the MIT Software License ## Any contributions you make will be under the MIT Software License
In short, when you submit code changes, your submissions are understood to be under the same [MIT License](http://choosealicense.com/licenses/mit/) that covers the project. Feel free to contact the maintainers if that's a concern. In short, when you submit code changes, your submissions are understood to be under the same [MIT License](http://choosealicense.com/licenses/mit/) that covers the project. Feel free to contact the maintainers if that's a concern.

109
EXAMPLES.md
View File

@ -3126,6 +3126,115 @@ traceroute -m 3 8.8.8.8 | jc --traceroute -p # or: jc -p traceroute -m
] ]
} }
``` ```
### ufw status
```bash
ufw status verbose | jc --ufw -p # or jc -p ufw status verbose
```
```json
{
"status": "active",
"logging": "on",
"logging_level": "low",
"default": "deny (incoming), allow (outgoing), disabled (routed)",
"new_profiles": "skip",
"rules": [
{
"action": "ALLOW",
"action_direction": "IN",
"index": null,
"network_protocol": "ipv4",
"to_interface": "any",
"to_transport": "any",
"to_service": null,
"to_ports": [
22
],
"to_ip": "0.0.0.0",
"to_ip_prefix": 0,
"comment": null,
"from_ip": "0.0.0.0",
"from_ip_prefix": 0,
"from_interface": "any",
"from_transport": "any",
"from_port_ranges": [
{
"start": 0,
"end": 65535
}
],
"from_service": null
},
{
"action": "ALLOW",
"action_direction": "IN",
"index": null,
"network_protocol": "ipv4",
"to_interface": "any",
"to_transport": "tcp",
"to_service": null,
"to_ports": [
80,
443
],
"to_ip": "0.0.0.0",
"to_ip_prefix": 0,
"comment": null,
"from_ip": "0.0.0.0",
"from_ip_prefix": 0,
"from_interface": "any",
"from_transport": "any",
"from_port_ranges": [
{
"start": 0,
"end": 65535
}
],
"from_service": null
}
]
}
```
### ufw app info [application]
```bash
ufw app info MSN | jc --ufw-appinfo -p # or: jc -p ufw app info MSN
```
```json
[
{
"profile": "MSN",
"title": "MSN Chat",
"description": "MSN chat protocol (with file transfer and voice)",
"tcp_list": [
1863,
6901
],
"udp_list": [
1863,
6901
],
"tcp_ranges": [
{
"start": 6891,
"end": 6900
}
],
"normalized_tcp_list": [
1863,
6901
],
"normalized_tcp_ranges": [
{
"start": 6891,
"end": 6900
}
],
"normalized_udp_list": [
1863,
6901
]
}
]
```
### uname -a ### uname -a
```bash ```bash
uname -a | jc --uname -p # or: jc -p uname -a uname -a | jc --uname -p # or: jc -p uname -a

28
README.md
View File

@ -62,10 +62,6 @@ The `jc` parsers can also be used as python modules. In this case the output wil
``` ```
Two representations of the data are possible. The default representation uses a strict schema per parser and converts known numbers to int/float JSON values. Certain known values of `None` are converted to JSON `null`, known boolean values are converted, and, in some cases, additional semantic context fields are added. Two representations of the data are possible. The default representation uses a strict schema per parser and converts known numbers to int/float JSON values. Certain known values of `None` are converted to JSON `null`, known boolean values are converted, and, in some cases, additional semantic context fields are added.
> Note: Some parsers have calculated epoch timestamp fields added to the output. Unless a timestamp field name has a `_utc` suffix it is considered naive. (i.e. based on the local timezone of the system the `jc` parser was run on).
>
> If a UTC timezone can be detected in the text of the command output, the timestamp will be timezone aware and have a `_utc` suffix on the key name. (e.g. `epoch_utc`) No other timezones are supported for aware timestamps.
To access the raw, pre-processed JSON, use the `-r` cli option or the `raw=True` function parameter in `parse()`. To access the raw, pre-processed JSON, use the `-r` cli option or the `raw=True` function parameter in `parse()`.
Schemas for each parser can be found at the documentation link beside each parser below. Schemas for each parser can be found at the documentation link beside each parser below.
@ -102,7 +98,7 @@ pip3 install jc
| FreeBSD | `portsnap fetch update && cd /usr/ports/textproc/py-jc && make install clean` | | FreeBSD | `portsnap fetch update && cd /usr/ports/textproc/py-jc && make install clean` |
| Ansible filter plugin | `ansible-galaxy collection install community.general` | | Ansible filter plugin | `ansible-galaxy collection install community.general` |
> For more packages and binaries, see https://kellyjonbrazil.github.io/jc-packaging/. > For more packages and binaries, see the [jc packaging](https://kellyjonbrazil.github.io/jc-packaging/) site.
## Usage ## Usage
`jc` accepts piped input from `STDIN` and outputs a JSON representation of the previous command's output to `STDOUT`. `jc` accepts piped input from `STDIN` and outputs a JSON representation of the previous command's output to `STDOUT`.
@ -115,8 +111,6 @@ jc [OPTIONS] COMMAND
``` ```
The JSON output can be compact (default) or pretty formatted with the `-p` option. The JSON output can be compact (default) or pretty formatted with the `-p` option.
> Note: For best results set the `LANG` locale environment variable to `C`. For example, either by setting directly on the command-line: `$ LANG=C date | jc --date`, or by exporting to the environment before running commands: `$ export LANG=C`.
### Parsers ### Parsers
- `--acpi` enables the `acpi` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/acpi)) - `--acpi` enables the `acpi` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/acpi))
@ -182,6 +176,8 @@ The JSON output can be compact (default) or pretty formatted with the `-p` optio
- `--timedatectl` enables the `timedatectl status` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/timedatectl)) - `--timedatectl` enables the `timedatectl status` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/timedatectl))
- `--tracepath` enables the `tracepath` and `tracepath6` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/tracepath)) - `--tracepath` enables the `tracepath` and `tracepath6` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/tracepath))
- `--traceroute` enables the `traceroute` and `traceroute6` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/traceroute)) - `--traceroute` enables the `traceroute` and `traceroute6` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/traceroute))
- `--ufw` enables the `ufw status` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ufw))
- `--ufw-appinfo` enables the `ufw app info [application]` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/ufw_appinfo))
- `--uname` enables the `uname -a` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/uname)) - `--uname` enables the `uname -a` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/uname))
- `--upower` enables the `upower` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/upower)) - `--upower` enables the `upower` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/upower))
- `--uptime` enables the `uptime` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/uptime)) - `--uptime` enables the `uptime` command parser ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/uptime))
@ -230,6 +226,24 @@ Local plugin filenames must be valid python module names, therefore must consist
> Note: The application data directory follows the [XDG Base Directory Specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html) > Note: The application data directory follows the [XDG Base Directory Specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html)
### Caveats
**Locale:**
For best results set the `LANG` locale environment variable to `C`. For example, either by setting directly on the command-line:
```
$ LANG=C date | jc --date
```
or by exporting to the environment before running commands:
```
$ export LANG=C
```
**Timezones:**
Some parsers have calculated epoch timestamp fields added to the output. Unless a timestamp field name has a `_utc` suffix it is considered naive. (i.e. based on the local timezone of the system the `jc` parser was run on).
If a UTC timezone can be detected in the text of the command output, the timestamp will be timezone aware and have a `_utc`P suffix on the key name. (e.g. `epoch_utc`) No other timezones are supported for aware timestamps.
## Compatibility ## Compatibility
Some parsers like `ls`, `ps`, `dig`, etc. will work on any platform. Other parsers that are platform-specific will generate a warning message if they are used on an unsupported platform. To see all parser information, including compatibility, run `jc -ap`. Some parsers like `ls`, `ps`, `dig`, etc. will work on any platform. Other parsers that are platform-specific will generate a warning message if they are used on an unsupported platform. To see all parser information, including compatibility, run `jc -ap`.

1
docs/parsers/ping.md
View File

@ -97,7 +97,6 @@ Examples:
] ]
} }
$ ping -c 3 -p ff cnn.com | jc --ping -p -r $ ping -c 3 -p ff cnn.com | jc --ping -p -r
{ {
"destination_ip": "151.101.129.67", "destination_ip": "151.101.129.67",

225
docs/parsers/ufw.md Normal file
View File

@ -0,0 +1,225 @@
[Home](https://kellyjonbrazil.github.io/jc/)
# jc.parsers.ufw
jc - JSON CLI output utility `ufw status` command output parser
Usage (cli):
$ ufw status | jc --ufw
or
$ jc ufw status
Usage (module):
import jc.parsers.ufw
result = jc.parsers.ufw.parse(ufw_command_output)
Schema:
{
"status": string,
"logging": string,
"logging_level": string,
"default": string,
"new_profiles": string,
"rules": [
{
"action": string,
"action_direction": string, # null if blank
"index": integer, # null if blank
"network_protocol": string,
"to_ip": string,
"to_ip_prefix": integer,
"to_interface": string,
"to_transport": string,
"to_ports": [
integer
],
"to_port_ranges": [
{
"start": integer,
"end": integer
}
],
"to_service": string, # null if any to ports or port_ranges are set
"from_ip": string,
"from_ip_prefix": integer,
"from_interface": string,
"from_transport": string,
"from_ports": [
integer
],
"from_port_ranges": [
{
"start": integer,
"end": integer
}
],
"from_service": string, # null if any from ports or port_ranges are set
"comment": string # null if no comment
}
]
}
Examples:
$ ufw status verbose | jc --ufw -p
{
"status": "active",
"logging": "on",
"logging_level": "low",
"default": "deny (incoming), allow (outgoing), disabled (routed)",
"new_profiles": "skip",
"rules": [
{
"action": "ALLOW",
"action_direction": "IN",
"index": null,
"network_protocol": "ipv4",
"to_interface": "any",
"to_transport": "any",
"to_service": null,
"to_ports": [
22
],
"to_ip": "0.0.0.0",
"to_ip_prefix": 0,
"comment": null,
"from_ip": "0.0.0.0",
"from_ip_prefix": 0,
"from_interface": "any",
"from_transport": "any",
"from_port_ranges": [
{
"start": 0,
"end": 65535
}
],
"from_service": null
},
{
"action": "ALLOW",
"action_direction": "IN",
"index": null,
"network_protocol": "ipv4",
"to_interface": "any",
"to_transport": "tcp",
"to_service": null,
"to_ports": [
80,
443
],
"to_ip": "0.0.0.0",
"to_ip_prefix": 0,
"comment": null,
"from_ip": "0.0.0.0",
"from_ip_prefix": 0,
"from_interface": "any",
"from_transport": "any",
"from_port_ranges": [
{
"start": 0,
"end": 65535
}
],
"from_service": null
},
...
]
}
$ ufw status verbose | jc --ufw -p -r
{
"status": "active",
"logging": "on",
"logging_level": "low",
"default": "deny (incoming), allow (outgoing), disabled (routed)",
"new_profiles": "skip",
"rules": [
{
"action": "ALLOW",
"action_direction": "IN",
"index": null,
"network_protocol": "ipv4",
"to_interface": "any",
"to_transport": "any",
"to_service": null,
"to_ports": [
"22"
],
"to_ip": "0.0.0.0",
"to_ip_prefix": "0",
"comment": null,
"from_ip": "0.0.0.0",
"from_ip_prefix": "0",
"from_interface": "any",
"from_transport": "any",
"from_port_ranges": [
{
"start": "0",
"end": "65535"
}
],
"from_service": null
},
{
"action": "ALLOW",
"action_direction": "IN",
"index": null,
"network_protocol": "ipv4",
"to_interface": "any",
"to_transport": "tcp",
"to_service": null,
"to_ports": [
"80",
"443"
],
"to_ip": "0.0.0.0",
"to_ip_prefix": "0",
"comment": null,
"from_ip": "0.0.0.0",
"from_ip_prefix": "0",
"from_interface": "any",
"from_transport": "any",
"from_port_ranges": [
{
"start": "0",
"end": "65535"
}
],
"from_service": null
},
...
]
}
## info
```python
info()
```
Provides parser metadata (version, author, etc.)
## parse
```python
parse(data, raw=False, quiet=False)
```
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) output preprocessed JSON if True
quiet: (boolean) suppress warning messages if True
Returns:
Dictionary. Raw or processed structured data.
## Parser Information
Compatibility: linux
Version 1.0 by Kelly Brazil (kellyjonbrazil@gmail.com)

158
docs/parsers/ufw_appinfo.md Normal file
View File

@ -0,0 +1,158 @@
[Home](https://kellyjonbrazil.github.io/jc/)
# jc.parsers.ufw_appinfo
jc - JSON CLI output utility `ufw app info [application]` command output parser
Supports individual apps via `ufw app info [application]` and all apps list via `ufw app info all`.
Because `ufw` application definitions allow overlapping ports and port ranges, this parser preserves that behavior, but also provides `normalized` lists and ranges that remove duplicate ports and merge overlapping ranges.
Usage (cli):
$ ufw app info OpenSSH | jc --ufw-appinfo
or
$ jc ufw app info OpenSSH
Usage (module):
import jc.parsers.ufw_appinfo
result = jc.parsers.ufw_appinfo.parse(ufw_appinfo_command_output)
Schema:
[
{
"profile": string,
"title": string,
"description": string,
"tcp_list": [
integer
],
"tcp_ranges": [
{
"start": integer, # 'any' is converted to start/end: 0/65535
"end": integer
}
],
"udp_list": [
integer
],
"udp_ranges": [
{
"start": integer, # 'any' is converted to start/end: 0/65535
"end": integer
}
],
"normalized_tcp_list": [
integers # duplicates and overlapping are removed
],
"normalized_tcp_ranges": [
{
"start": integer, # 'any' is converted to start/end: 0/65535
"end": integers # overlapping are merged
}
],
"normalized_udp_list": [
integers # duplicates and overlapping are removed
],
"normalized_udp_ranges": [
{
"start": integer, # 'any' is converted to start/end: 0/65535
"end": integers # overlapping are merged
}
]
}
]
Examples:
$ ufw app info MSN | jc --ufw-appinfo -p
[
{
"profile": "MSN",
"title": "MSN Chat",
"description": "MSN chat protocol (with file transfer and voice)",
"tcp_list": [
1863,
6901
],
"udp_list": [
1863,
6901
],
"tcp_ranges": [
{
"start": 6891,
"end": 6900
}
],
"normalized_tcp_list": [
1863,
6901
],
"normalized_tcp_ranges": [
{
"start": 6891,
"end": 6900
}
],
"normalized_udp_list": [
1863,
6901
]
}
]
$ ufw app info MSN | jc --ufw-appinfo -p -r
[
{
"profile": "MSN",
"title": "MSN Chat",
"description": "MSN chat protocol (with file transfer and voice)",
"tcp_list": [
"1863",
"6901"
],
"udp_list": [
"1863",
"6901"
],
"tcp_ranges": [
{
"start": "6891",
"end": "6900"
}
]
}
]
## info
```python
info()
```
Provides parser metadata (version, author, etc.)
## parse
```python
parse(data, raw=False, quiet=False)
```
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) output preprocessed JSON if True
quiet: (boolean) suppress warning messages if True
Returns:
List of Dictionaries. Raw or processed structured data.
## Parser Information
Compatibility: linux
Version 1.0 by Kelly Brazil (kellyjonbrazil@gmail.com)

4
docs/utils.md
View File

@ -74,7 +74,7 @@ Returns:
convert_to_int(value) convert_to_int(value)
``` ```
Converts string input to integer by stripping all non-numeric characters Converts string and float input to int. Strips all non-numeric characters from strings.
Parameters: Parameters:
@ -90,7 +90,7 @@ Returns:
convert_to_float(value) convert_to_float(value)
``` ```
Converts string input to float by stripping all non-numeric characters Converts string and int input to float. Strips all non-numeric characters from strings.
Parameters: Parameters:

2
jc/__init__.py
View File

@ -86,4 +86,4 @@ Module Example:
""" """
name = 'jc' name = 'jc'
__version__ = '1.15.2' __version__ = '1.15.3'

2
jc/cli.py
View File

@ -102,6 +102,8 @@ parsers = [
'timedatectl', 'timedatectl',
'tracepath', 'tracepath',
'traceroute', 'traceroute',
'ufw',
'ufw-appinfo',
'uname', 'uname',
'upower', 'upower',
'uptime', 'uptime',

BIN
jc/man/jc.1.gz
View File

Binary file not shown.

1
jc/parsers/ping.py
View File

@ -94,7 +94,6 @@ Examples:
] ]
} }
$ ping -c 3 -p ff cnn.com | jc --ping -p -r $ ping -c 3 -p ff cnn.com | jc --ping -p -r
{ {
"destination_ip": "151.101.129.67", "destination_ip": "151.101.129.67",

466
jc/parsers/ufw.py Normal file
View File

@ -0,0 +1,466 @@
"""jc - JSON CLI output utility `ufw status` command output parser
Usage (cli):
$ ufw status | jc --ufw
or
$ jc ufw status
Usage (module):
import jc.parsers.ufw
result = jc.parsers.ufw.parse(ufw_command_output)
Schema:
{
"status": string,
"logging": string,
"logging_level": string,
"default": string,
"new_profiles": string,
"rules": [
{
"action": string,
"action_direction": string, # null if blank
"index": integer, # null if blank
"network_protocol": string,
"to_ip": string,
"to_ip_prefix": integer,
"to_interface": string,
"to_transport": string,
"to_ports": [
integer
],
"to_port_ranges": [
{
"start": integer,
"end": integer
}
],
"to_service": string, # null if any to ports or port_ranges are set
"from_ip": string,
"from_ip_prefix": integer,
"from_interface": string,
"from_transport": string,
"from_ports": [
integer
],
"from_port_ranges": [
{
"start": integer,
"end": integer
}
],
"from_service": string, # null if any from ports or port_ranges are set
"comment": string # null if no comment
}
]
}
Examples:
$ ufw status verbose | jc --ufw -p
{
"status": "active",
"logging": "on",
"logging_level": "low",
"default": "deny (incoming), allow (outgoing), disabled (routed)",
"new_profiles": "skip",
"rules": [
{
"action": "ALLOW",
"action_direction": "IN",
"index": null,
"network_protocol": "ipv4",
"to_interface": "any",
"to_transport": "any",
"to_service": null,
"to_ports": [
22
],
"to_ip": "0.0.0.0",
"to_ip_prefix": 0,
"comment": null,
"from_ip": "0.0.0.0",
"from_ip_prefix": 0,
"from_interface": "any",
"from_transport": "any",
"from_port_ranges": [
{
"start": 0,
"end": 65535
}
],
"from_service": null
},
{
"action": "ALLOW",
"action_direction": "IN",
"index": null,
"network_protocol": "ipv4",
"to_interface": "any",
"to_transport": "tcp",
"to_service": null,
"to_ports": [
80,
443
],
"to_ip": "0.0.0.0",
"to_ip_prefix": 0,
"comment": null,
"from_ip": "0.0.0.0",
"from_ip_prefix": 0,
"from_interface": "any",
"from_transport": "any",
"from_port_ranges": [
{
"start": 0,
"end": 65535
}
],
"from_service": null
},
...
]
}
$ ufw status verbose | jc --ufw -p -r
{
"status": "active",
"logging": "on",
"logging_level": "low",
"default": "deny (incoming), allow (outgoing), disabled (routed)",
"new_profiles": "skip",
"rules": [
{
"action": "ALLOW",
"action_direction": "IN",
"index": null,
"network_protocol": "ipv4",
"to_interface": "any",
"to_transport": "any",
"to_service": null,
"to_ports": [
"22"
],
"to_ip": "0.0.0.0",
"to_ip_prefix": "0",
"comment": null,
"from_ip": "0.0.0.0",
"from_ip_prefix": "0",
"from_interface": "any",
"from_transport": "any",
"from_port_ranges": [
{
"start": "0",
"end": "65535"
}
],
"from_service": null
},
{
"action": "ALLOW",
"action_direction": "IN",
"index": null,
"network_protocol": "ipv4",
"to_interface": "any",
"to_transport": "tcp",
"to_service": null,
"to_ports": [
"80",
"443"
],
"to_ip": "0.0.0.0",
"to_ip_prefix": "0",
"comment": null,
"from_ip": "0.0.0.0",
"from_ip_prefix": "0",
"from_interface": "any",
"from_transport": "any",
"from_port_ranges": [
{
"start": "0",
"end": "65535"
}
],
"from_service": null
},
...
]
}
"""
import jc.utils
import re
import ipaddress
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.0'
description = '`ufw status` command parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
compatible = ['linux']
magic_commands = ['ufw status']
__version__ = info.version
def _process(proc_data):
"""
Final processing to conform to the schema.
Parameters:
proc_data: (List of Dictionaries) raw structured data to process
Returns:
Dictionary. Structured to conform to the schema.
"""
int_list = ['index', 'to_ip_prefix', 'from_ip_prefix']
if 'rules' in proc_data:
for i, item in enumerate(proc_data['rules']):
for key in item:
if key in int_list:
proc_data['rules'][i][key] = jc.utils.convert_to_int(proc_data['rules'][i][key])
if key in ['to_ports', 'from_ports']:
for i2, item2 in enumerate(proc_data['rules'][i][key]):
proc_data['rules'][i][key][i2] = jc.utils.convert_to_int(item2)
if key in ['to_port_ranges', 'from_port_ranges']:
for i2, item2 in enumerate(proc_data['rules'][i][key]):
proc_data['rules'][i][key][i2]['start'] = jc.utils.convert_to_int(proc_data['rules'][i][key][i2]['start'])
proc_data['rules'][i][key][i2]['end'] = jc.utils.convert_to_int(proc_data['rules'][i][key][i2]['end'])
return proc_data
def _parse_to_from(linedata, direction, rule_obj=None):
if rule_obj is None:
rule_obj = {}
# pull out rule index, if they exist: [ 1]
if direction == 'to':
RE_LINE_NUM = re.compile(r'\[[ 0-9]+\]\s')
line_number_match = re.search(RE_LINE_NUM, linedata)
if line_number_match:
rule_obj['index'] = line_number_match.group(0).replace('[', '').replace(']', '').strip()
linedata = re.sub(RE_LINE_NUM, '', linedata)
else:
rule_obj['index'] = None
# pull out comments, if they exist
if direction == 'from':
RE_COMMENT = re.compile(r'#.+$')
comment_match = re.search(RE_COMMENT, linedata)
if comment_match:
rule_obj['comment'] = comment_match.group(0).lstrip('#').strip()
linedata = re.sub(RE_COMMENT, '', linedata)
else:
rule_obj['comment'] = None
# pull (v6)
RE_V6 = re.compile(r'\(v6\)')
v6_match = re.search(RE_V6, linedata)
if v6_match:
rule_obj['network_protocol'] = 'ipv6'
linedata = re.sub(RE_V6, '', linedata)
elif not rule_obj.get('network_protocol'):
rule_obj['network_protocol'] = 'ipv4'
# pull 'Anywhere' if exists. Assign to 0.0.0.0/0 or ::/0 depending on if (v6) is found
if 'Anywhere' in linedata:
if rule_obj.get('network_protocol') == 'ipv6':
rule_obj[direction + '_ip'] = '::'
rule_obj[direction + '_ip_prefix'] = '0'
elif rule_obj.get('network_protocol') == 'ipv4':
rule_obj[direction + '_ip'] = '0.0.0.0'
rule_obj[direction + '_ip_prefix'] = '0'
linedata = linedata.replace('Anywhere', '')
# pull out interface (after 'on')
linedata_list = linedata.split(' on ', maxsplit=1)
if len(linedata_list) > 1:
rule_obj[direction + '_interface'] = linedata_list[1].strip()
linedata = linedata_list[0]
else:
rule_obj[direction + '_interface'] = 'any'
# pull tcp/udp/etc. transport - strip on '/'
linedata_list = linedata.rsplit('/', maxsplit=1)
if len(linedata_list) > 1:
if linedata_list[1].strip() in ['tcp', 'udp', 'ah', 'esp', 'gre', 'ipv6', 'igmp']:
rule_obj[direction + '_transport'] = linedata_list[1].strip()
linedata = linedata_list[0]
else:
rule_obj[direction + '_transport'] = 'any'
else:
rule_obj[direction + '_transport'] = 'any'
# pull out ipv4 or ipv6 addresses
linedata_list = linedata.split()
new_linedata_list = []
valid_ip = None
for item in linedata_list:
try:
valid_ip = ipaddress.IPv4Interface(item)
except Exception:
try:
valid_ip = ipaddress.IPv6Interface(item)
except Exception:
new_linedata_list.append(item)
if valid_ip:
rule_obj[direction + '_ip'] = str(valid_ip.ip)
rule_obj[direction + '_ip_prefix'] = str(valid_ip.with_prefixlen.split('/')[1])
linedata = ' '.join(new_linedata_list)
# find the numeric port(s)
linedata_list = linedata.split(',')
port_list = []
port_ranges = []
for item in linedata_list:
if item.strip().isnumeric():
port_list.append(item.strip())
elif ':' in item:
p_range = item.strip().split(':', maxsplit=1)
port_ranges.append(
{
"start": p_range[0],
"end": p_range[1]
}
)
if port_list or port_ranges:
rule_obj[direction + '_service'] = None
linedata = ''
if port_list:
rule_obj[direction + '_ports'] = port_list
if port_ranges:
rule_obj[direction + '_port_ranges'] = port_ranges
# only thing left should be the service name.
if linedata.strip():
rule_obj[direction + '_service'] = linedata.strip()
rule_obj[direction + '_transport'] = None
# check if to/from IP addresses exist. If not, set to 0.0.0.0/0 or ::/0
if direction + '_ip' not in rule_obj:
if rule_obj.get('network_protocol') == 'ipv6':
rule_obj[direction + '_ip'] = '::'
rule_obj[direction + '_ip_prefix'] = '0'
elif rule_obj.get('network_protocol') == 'ipv4':
rule_obj[direction + '_ip'] = '0.0.0.0'
rule_obj[direction + '_ip_prefix'] = '0'
# finally set default ports if no ports exist and there should be some
if direction + '_transport' in rule_obj:
if rule_obj[direction + '_transport'] in ['tcp', 'udp', 'any']:
if not port_list and not port_ranges:
rule_obj[direction + '_port_ranges'] = [
{
'start': '0',
'end': '65535'
}
]
rule_obj[direction + '_service'] = None
return rule_obj
def parse(data, raw=False, quiet=False):
"""
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) output preprocessed JSON if True
quiet: (boolean) suppress warning messages if True
Returns:
Dictionary. Raw or processed structured data.
"""
if not quiet:
jc.utils.compatibility(__name__, info.compatible)
raw_output = {}
rules_list = []
if jc.utils.has_data(data):
rule_lines = False
for line in filter(None, data.splitlines()):
if line.startswith('Status: '):
raw_output['status'] = line.split(': ', maxsplit=1)[1]
continue
if line.startswith('Logging: '):
log_line = line.split(': ', maxsplit=1)
log_line = log_line[1]
log_line = log_line.split()
raw_output['logging'] = log_line[0]
if len(log_line) == 2:
raw_output['logging_level'] = log_line[1].replace('(', '').replace(')', '').strip()
continue
if line.startswith('Default: '):
raw_output['default'] = line.split(': ', maxsplit=1)[1]
continue
if line.startswith('New profiles: '):
raw_output['new_profiles'] = line.split(': ', maxsplit=1)[1]
continue
if 'To' in line and 'Action' in line and 'From' in line:
rule_lines = True
continue
if rule_lines:
if '------' in line:
continue
# Split on action. Left of Action is 'to', right of Action is 'from'
rule_obj = {}
splitline = re.split(r'(ALLOW IN|ALLOW OUT|ALLOW FWD|DENY IN|DENY OUT|DENY FWD|LIMIT IN|LIMIT OUT|LIMIT FWD|REJECT IN|REJECT OUT|REJECT FWD|ALLOW|DENY|LIMIT|REJECT)', line)
to_line = splitline[0]
action_line = splitline[1]
action_list = action_line.split()
from_line = splitline[2]
action_direction = None
if len(action_list) == 1:
action = action_list[0]
elif len(action_list) == 2:
action = action_list[0]
action_direction = action_list[1]
rule_obj['action'] = action
rule_obj['action_direction'] = action_direction
rule_obj.update(_parse_to_from(to_line, 'to'))
rule_obj.update(_parse_to_from(from_line, 'from', rule_obj))
rules_list.append(rule_obj)
raw_output['rules'] = rules_list
if raw:
return raw_output
else:
return _process(raw_output)

364
jc/parsers/ufw_appinfo.py Normal file
View File

@ -0,0 +1,364 @@
"""jc - JSON CLI output utility `ufw app info [application]` command output parser
Supports individual apps via `ufw app info [application]` and all apps list via `ufw app info all`.
Because `ufw` application definitions allow overlapping ports and port ranges, this parser preserves that behavior, but also provides `normalized` lists and ranges that remove duplicate ports and merge overlapping ranges.
Usage (cli):
$ ufw app info OpenSSH | jc --ufw-appinfo
or
$ jc ufw app info OpenSSH
Usage (module):
import jc.parsers.ufw_appinfo
result = jc.parsers.ufw_appinfo.parse(ufw_appinfo_command_output)
Schema:
[
{
"profile": string,
"title": string,
"description": string,
"tcp_list": [
integer
],
"tcp_ranges": [
{
"start": integer, # 'any' is converted to start/end: 0/65535
"end": integer
}
],
"udp_list": [
integer
],
"udp_ranges": [
{
"start": integer, # 'any' is converted to start/end: 0/65535
"end": integer
}
],
"normalized_tcp_list": [
integers # duplicates and overlapping are removed
],
"normalized_tcp_ranges": [
{
"start": integer, # 'any' is converted to start/end: 0/65535
"end": integers # overlapping are merged
}
],
"normalized_udp_list": [
integers # duplicates and overlapping are removed
],
"normalized_udp_ranges": [
{
"start": integer, # 'any' is converted to start/end: 0/65535
"end": integers # overlapping are merged
}
]
}
]
Examples:
$ ufw app info MSN | jc --ufw-appinfo -p
[
{
"profile": "MSN",
"title": "MSN Chat",
"description": "MSN chat protocol (with file transfer and voice)",
"tcp_list": [
1863,
6901
],
"udp_list": [
1863,
6901
],
"tcp_ranges": [
{
"start": 6891,
"end": 6900
}
],
"normalized_tcp_list": [
1863,
6901
],
"normalized_tcp_ranges": [
{
"start": 6891,
"end": 6900
}
],
"normalized_udp_list": [
1863,
6901
]
}
]
$ ufw app info MSN | jc --ufw-appinfo -p -r
[
{
"profile": "MSN",
"title": "MSN Chat",
"description": "MSN chat protocol (with file transfer and voice)",
"tcp_list": [
"1863",
"6901"
],
"udp_list": [
"1863",
"6901"
],
"tcp_ranges": [
{
"start": "6891",
"end": "6900"
}
]
}
]
"""
import jc.utils
class info():
"""Provides parser metadata (version, author, etc.)"""
version = '1.0'
description = '`ufw app info [application]` command parser'
author = 'Kelly Brazil'
author_email = 'kellyjonbrazil@gmail.com'
compatible = ['linux']
magic_commands = ['ufw app']
__version__ = info.version
def _process(proc_data):
"""
Final processing to conform to the schema.
Parameters:
proc_data: (List of Dictionaries) raw structured data to process
Returns:
List of Dictionaries. Structured to conform to the schema.
"""
for profile in proc_data:
# convert to ints
int_list = ['start', 'end']
if 'tcp_list' in profile:
profile['tcp_list'] = [int(p) for p in profile['tcp_list']]
if 'udp_list' in profile:
profile['udp_list'] = [int(p) for p in profile['udp_list']]
for protocol in ['tcp', 'udp']:
if protocol + '_ranges' in profile:
for i, item in enumerate(profile[protocol + '_ranges']):
for key in item:
if key in int_list:
profile[protocol + '_ranges'][i][key] = int(profile[protocol + '_ranges'][i][key])
# create normalized port lists and port ranges (remove duplicates and merge ranges)
# dump ranges into a set of 0 - 65535
# if items in the port list are in the set, then remove them
# iterate through the set to find gaps and create new ranges based on them
for protocol in ['tcp', 'udp']:
port_set = set()
if protocol + '_ranges' in profile:
for item in profile[protocol + '_ranges']:
port_set.update(range(item['start'], item['end'] + 1))
if protocol + '_list' in profile:
new_port_list = sorted(set([p for p in profile[protocol + '_list'] if p not in port_set]))
if new_port_list:
profile['normalized_' + protocol + '_list'] = new_port_list
new_port_ranges = []
state = 'findstart' # 'findstart' or 'findend'
for port in range(0, 65535 + 2):
if state == 'findstart':
port_range_obj = {}
if port in port_set:
port_range_obj['start'] = port
state = 'findend'
continue
if state == 'findend':
if port not in port_set:
port_range_obj['end'] = port - 1
new_port_ranges.append(port_range_obj)
state = 'findstart'
if new_port_ranges:
profile['normalized_' + protocol + '_ranges'] = new_port_ranges
return proc_data
def _parse_port_list(data, port_list=None):
"""return a list of port strings"""
# 1,2,3,4,5,6,7,8,9,10,9,30,80:90,8080:8090
# overlapping and repeated port numbers are allowed
if port_list is None:
port_list = []
data = data.split(',')
data_list = [p.strip() for p in data if ':' not in p and 'any' not in p]
port_list.extend(data_list)
return port_list
def _parse_port_range(data, range_list=None):
"""return a list of dictionaries"""
# 1,2,3,4,5,6,7,8,9,10,9,30,80:90,8080:8090
# overlapping port ranges are allowed
if range_list is None:
range_list = []
data = data.strip().split(',')
ranges = [p.strip() for p in data if ':' in p]
range_obj = {}
if 'any' in data:
range_list.append(
{
'start': 0,
'end': 65535
}
)
for range_ in ranges:
range_obj = {
'start': range_.split(':')[0],
'end': range_.split(':')[1]
}
range_list.append(range_obj)
return range_list
def parse(data, raw=False, quiet=False):
"""
Main text parsing function
Parameters:
data: (string) text data to parse
raw: (boolean) output preprocessed JSON if True
quiet: (boolean) suppress warning messages if True
Returns:
List of Dictionaries. Raw or processed structured data.
"""
if not quiet:
jc.utils.compatibility(__name__, info.compatible)
raw_output = []
item_obj = {}
if jc.utils.has_data(data):
ports = False
for line in filter(None, data.splitlines()):
if line.startswith('--'):
if item_obj:
raw_output.append(item_obj)
item_obj = {}
continue
if line.startswith('Profile:'):
item_obj['profile'] = line.split(': ')[1]
continue
if line.startswith('Title:'):
item_obj['title'] = line.split(': ')[1]
continue
if line.startswith('Description:'):
item_obj['description'] = line.split(': ')[1]
continue
if line.startswith('Port'):
ports = True
continue
if ports:
line_list = line.rsplit('/', maxsplit=1)
if len(line_list) == 2:
if line_list[1] == 'tcp':
tcp_prot_list = _parse_port_list(line_list[0])
if tcp_prot_list:
item_obj['tcp_list'] = tcp_prot_list
tcp_prot_range = _parse_port_range(line_list[0])
if tcp_prot_range:
item_obj['tcp_ranges'] = tcp_prot_range
elif line_list[1] == 'udp':
udp_prot_list = _parse_port_list(line_list[0])
if udp_prot_list:
item_obj['udp_list'] = udp_prot_list
udp_prot_range = _parse_port_range(line_list[0])
if udp_prot_range:
item_obj['udp_ranges'] = udp_prot_range
# 'any' case
else:
t_list = []
t_range = []
u_list = []
u_range = []
if 'tcp_list' in item_obj:
t_list = item_obj['tcp_list']
if 'tcp_ranges' in item_obj:
t_range = item_obj['tcp_ranges']
if 'udp_list' in item_obj:
u_list = item_obj['udp_list']
if 'udp_ranges' in item_obj:
u_range = item_obj['udp_ranges']
t_p_list = _parse_port_list(line, t_list)
if t_p_list:
item_obj['tcp_list'] = t_p_list
t_r_list = _parse_port_range(line, t_range)
if t_r_list:
item_obj['tcp_ranges'] = t_r_list
u_p_list = _parse_port_list(line, u_list)
if u_p_list:
item_obj['udp_list'] = u_p_list
u_r_list = _parse_port_range(line, u_range)
if u_r_list:
item_obj['udp_ranges'] = u_r_list
if item_obj:
raw_output.append(item_obj)
if raw:
return raw_output
else:
return _process(raw_output)

11
jc/utils.py
View File

@ -85,7 +85,7 @@ def has_data(data):
def convert_to_int(value): def convert_to_int(value):
""" """
Converts string input to integer by stripping all non-numeric characters Converts string and float input to int. Strips all non-numeric characters from strings.
Parameters: Parameters:
@ -96,11 +96,12 @@ def convert_to_int(value):
integer/None Integer if successful conversion, otherwise None integer/None Integer if successful conversion, otherwise None
""" """
if isinstance(value, str): if isinstance(value, str):
str_val = re.sub(r'[^0-9\-\.]', '', value)
try: try:
return int(re.sub(r'[^0-9\-\.]', '', value)) return int(str_val)
except ValueError: except (ValueError, TypeError):
try: try:
return int(convert_to_float(value)) return int(float(str_val))
except (ValueError, TypeError): except (ValueError, TypeError):
return None return None
@ -113,7 +114,7 @@ def convert_to_int(value):
def convert_to_float(value): def convert_to_float(value):
""" """
Converts string input to float by stripping all non-numeric characters Converts string and int input to float. Strips all non-numeric characters from strings.
Parameters: Parameters:

BIN
man/jc.1.gz
View File

Binary file not shown.

1
runtests.sh
View File

@ -1,3 +1,4 @@
#!/bin/bash #!/bin/bash
# system should be in "America/Los_Angeles" timezone for all tests to pass
python3 -m unittest -v python3 -m unittest -v

2
setup.py
View File

@ -5,7 +5,7 @@ with open('README.md', 'r') as f:
setuptools.setup( setuptools.setup(
name='jc', name='jc',
version='1.15.2', version='1.15.3',
author='Kelly Brazil', author='Kelly Brazil',
author_email='kellyjonbrazil@gmail.com', author_email='kellyjonbrazil@gmail.com',
description='Converts the output of popular command-line tools and file-types to JSON.', description='Converts the output of popular command-line tools and file-types to JSON.',

13
templates/manpage_template
View File

@ -96,6 +96,19 @@ Local plugin filenames must be valid python module names, therefore must consist
Note: The application data directory follows the XDG Base Directory Specification Note: The application data directory follows the XDG Base Directory Specification
.SH CAVEATS
\fBLocale:\fP For best results set the \fBLANG\fP locale environment variable to \fBC\fP. For example, either by setting directly on the command-line:
\fB$ LANG=C date | jc --date\fP
or by exporting to the environment before running commands:
\fB$ export LANG=C\fP
\fBTimezones:\fP Some parsers have calculated epoch timestamp fields added to the output. Unless a timestamp field name has a \fB_utc\fP suffix it is considered naive. (i.e. based on the local timezone of the system the \fBjc\fP parser was run on).
If a UTC timezone can be detected in the text of the command output, the timestamp will be timezone aware and have a \fB_utc\fP suffix on the key name. (e.g. \fBepoch_utc\fP) No other timezones are supported for aware timestamps.
.SH EXAMPLES .SH EXAMPLES
Standard Syntax: Standard Syntax:
.RS .RS

26
templates/readme_template
View File

@ -62,10 +62,6 @@ The `jc` parsers can also be used as python modules. In this case the output wil
``` ```
Two representations of the data are possible. The default representation uses a strict schema per parser and converts known numbers to int/float JSON values. Certain known values of `None` are converted to JSON `null`, known boolean values are converted, and, in some cases, additional semantic context fields are added. Two representations of the data are possible. The default representation uses a strict schema per parser and converts known numbers to int/float JSON values. Certain known values of `None` are converted to JSON `null`, known boolean values are converted, and, in some cases, additional semantic context fields are added.
> Note: Some parsers have calculated epoch timestamp fields added to the output. Unless a timestamp field name has a `_utc` suffix it is considered naive. (i.e. based on the local timezone of the system the `jc` parser was run on).
>
> If a UTC timezone can be detected in the text of the command output, the timestamp will be timezone aware and have a `_utc` suffix on the key name. (e.g. `epoch_utc`) No other timezones are supported for aware timestamps.
To access the raw, pre-processed JSON, use the `-r` cli option or the `raw=True` function parameter in `parse()`. To access the raw, pre-processed JSON, use the `-r` cli option or the `raw=True` function parameter in `parse()`.
Schemas for each parser can be found at the documentation link beside each parser below. Schemas for each parser can be found at the documentation link beside each parser below.
@ -102,7 +98,7 @@ pip3 install jc
| FreeBSD | `portsnap fetch update && cd /usr/ports/textproc/py-jc && make install clean` | | FreeBSD | `portsnap fetch update && cd /usr/ports/textproc/py-jc && make install clean` |
| Ansible filter plugin | `ansible-galaxy collection install community.general` | | Ansible filter plugin | `ansible-galaxy collection install community.general` |
> For more packages and binaries, see https://kellyjonbrazil.github.io/jc-packaging/. > For more packages and binaries, see the [jc packaging](https://kellyjonbrazil.github.io/jc-packaging/) site.
## Usage ## Usage
`jc` accepts piped input from `STDIN` and outputs a JSON representation of the previous command's output to `STDOUT`. `jc` accepts piped input from `STDIN` and outputs a JSON representation of the previous command's output to `STDOUT`.
@ -115,8 +111,6 @@ jc [OPTIONS] COMMAND
``` ```
The JSON output can be compact (default) or pretty formatted with the `-p` option. The JSON output can be compact (default) or pretty formatted with the `-p` option.
> Note: For best results set the `LANG` locale environment variable to `C`. For example, either by setting directly on the command-line: `$ LANG=C date | jc --date`, or by exporting to the environment before running commands: `$ export LANG=C`.
### Parsers ### Parsers
{% for parser in jc.parsers %} {% for parser in jc.parsers %}
- `{{ parser.argument }}` enables the {{ parser.description }} ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/{{ parser.name }})){% endfor %} - `{{ parser.argument }}` enables the {{ parser.description }} ([documentation](https://kellyjonbrazil.github.io/jc/docs/parsers/{{ parser.name }})){% endfor %}
@ -160,6 +154,24 @@ Local plugin filenames must be valid python module names, therefore must consist
> Note: The application data directory follows the [XDG Base Directory Specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html) > Note: The application data directory follows the [XDG Base Directory Specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html)
### Caveats
**Locale:**
For best results set the `LANG` locale environment variable to `C`. For example, either by setting directly on the command-line:
```
$ LANG=C date | jc --date
```
or by exporting to the environment before running commands:
```
$ export LANG=C
```
**Timezones:**
Some parsers have calculated epoch timestamp fields added to the output. Unless a timestamp field name has a `_utc` suffix it is considered naive. (i.e. based on the local timezone of the system the `jc` parser was run on).
If a UTC timezone can be detected in the text of the command output, the timestamp will be timezone aware and have a `_utc`P suffix on the key name. (e.g. `epoch_utc`) No other timezones are supported for aware timestamps.
## Compatibility ## Compatibility
Some parsers like `ls`, `ps`, `dig`, etc. will work on any platform. Other parsers that are platform-specific will generate a warning message if they are used on an unsupported platform. To see all parser information, including compatibility, run `jc -ap`. Some parsers like `ls`, `ps`, `dig`, etc. will work on any platform. Other parsers that are platform-specific will generate a warning message if they are used on an unsupported platform. To see all parser information, including compatibility, run `jc -ap`.

1
tests/fixtures/generic/ufw-appinfo-msn.json vendored Normal file
View File

@ -0,0 +1 @@
[{"profile":"MSN","title":"MSN Chat","description":"MSN chat protocol (with file transfer and voice)","tcp_list":[1863,6901],"udp_list":[1863,6901],"tcp_ranges":[{"start":6891,"end":6900}],"normalized_tcp_list":[1863,6901],"normalized_tcp_ranges":[{"start":6891,"end":6900}],"normalized_udp_list":[1863,6901]}]

9
tests/fixtures/generic/ufw-appinfo-msn.out vendored Normal file
View File

@ -0,0 +1,9 @@
Profile: MSN
Title: MSN Chat
Description: MSN chat protocol (with file transfer and voice)
Ports:
1863
6891:6900/tcp
6901

1
tests/fixtures/generic/ufw-appinfo-test.json vendored Normal file
View File

@ -0,0 +1 @@
[{"profile":"TEST","title":"My test app","description":"a longer description of the test app here.","tcp_list":[1,2,3,4,5,6,7,8,9,10,9,8,7,30,53],"tcp_ranges":[{"start":80,"end":90},{"start":8080,"end":8090}],"udp_ranges":[{"start":50,"end":51},{"start":40,"end":60}],"udp_list":[53],"normalized_tcp_list":[1,2,3,4,5,6,7,8,9,10,30,53],"normalized_tcp_ranges":[{"start":80,"end":90},{"start":8080,"end":8090}],"normalized_udp_ranges":[{"start":40,"end":60}]}]

9
tests/fixtures/generic/ufw-appinfo-test.out vendored Normal file
View File

@ -0,0 +1,9 @@
Profile: TEST
Title: My test app
Description: a longer description of the test app here.
Ports:
1,2,3,4,5,6,7,8,9,10,9,8,7,30,80:90,8080:8090/tcp
50:51,40:60/udp
53

1
tests/fixtures/generic/ufw-appinfo-test2.json vendored Normal file
View File

@ -0,0 +1 @@
[{"profile":"TEST2","title":"My test app2","description":"a longer description of the test app here.","tcp_ranges":[{"start":0,"end":65535}],"udp_ranges":[{"start":50,"end":51}],"tcp_list":[53],"udp_list":[53],"normalized_tcp_ranges":[{"start":0,"end":65535}],"normalized_udp_list":[53],"normalized_udp_ranges":[{"start":50,"end":51}]}]

9
tests/fixtures/generic/ufw-appinfo-test2.out vendored Normal file
View File

@ -0,0 +1,9 @@
Profile: TEST2
Title: My test app2
Description: a longer description of the test app here.
Ports:
any/tcp
50:51/udp
53

1
tests/fixtures/generic/ufw-appinfo-test3.json vendored Normal file
View File

@ -0,0 +1 @@
[{"profile":"TEST3","title":"My test app3","description":"test overlapping ports","tcp_list":[80,83,80,53],"tcp_ranges":[{"start":70,"end":90}],"udp_ranges":[{"start":50,"end":51}],"udp_list":[53],"normalized_tcp_list":[53],"normalized_tcp_ranges":[{"start":70,"end":90}],"normalized_udp_list":[53],"normalized_udp_ranges":[{"start":50,"end":51}]}]

9
tests/fixtures/generic/ufw-appinfo-test3.out vendored Normal file
View File

@ -0,0 +1,9 @@
Profile: TEST3
Title: My test app3
Description: test overlapping ports
Ports:
80,83,80,70:90/tcp
50:51/udp
53

1
tests/fixtures/generic/ufw-inactive.json vendored Normal file
View File

@ -0,0 +1 @@
{"status":"inactive","rules":[]}

1
tests/fixtures/generic/ufw-inactive.out vendored Normal file
View File

@ -0,0 +1 @@
Status: inactive

1
tests/fixtures/generic/ufw-numbered.json vendored Normal file
View File

File diff suppressed because one or more lines are too long

23
tests/fixtures/generic/ufw-numbered.out vendored Normal file
View File

@ -0,0 +1,23 @@
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), deny (routed)
New profiles: skip
To Action From
-- ------ ----
[ 1] 22/tcp ALLOW IN Anywhere
[ 2] 22/tcp (v6) ALLOW OUT Anywhere (v6)
[ 3] 443/tcp DENY 192.168.0.1
[ 4] 443/udp DENY OUT 192.168.0.7 8080:8081
[ 5] 22/tcp ALLOW 192.168.0.0/24
[ 6] 22/udp ALLOW 192.168.0.0/24 8080:8081 on en0
[ 7] 22/tcp (v6) ALLOW IN 2405:204:7449:49fc:f09a:6f4a:bc93:1955/64 on en1
[ 8] 80 ALLOW IN Anywhere
[ 9] 8080 (v6) ALLOW IN Anywhere (v6)
[10] Apache Full ALLOW IN Anywhere
[11] Apache Full (v6) ALLOW IN Anywhere (v6)
[12] OpenSSH (v6) DENY IN Anywhere (v6)
[13] 10.10.10.10 8080 on enp34s0 ALLOW 127.0.0.1 8000
[14] 50200:50300/tcp (v6) ALLOW Anywhere (v6)
[15] Anywhere (v6) ALLOW IN 2405:204:7449:49fc:f09a:6f4a:bc93:1955

1
tests/fixtures/generic/ufw-numbered2.json vendored Normal file
View File

File diff suppressed because one or more lines are too long

18
tests/fixtures/generic/ufw-numbered2.out vendored Normal file
View File

@ -0,0 +1,18 @@
Status: active
To Action From
-- ------ ----
[ 1] 224.0.0.251 mDNS ALLOW IN Anywhere
[ 2] Anywhere ALLOW IN 123.123.123.123
[ 3] 25 ALLOW IN Anywhere
[ 4] 80 ALLOW IN Anywhere
[ 5] 443 ALLOW IN Anywhere
[ 6] 465 ALLOW IN Anywhere
[ 7] 993 ALLOW IN Anywhere
[ 8] 995 ALLOW IN Anywhere
[ 9] ff02::fb mDNS ALLOW IN Anywhere (v6)
[10] 25 (v6) ALLOW IN Anywhere (v6)
[11] 80 (v6) ALLOW IN Anywhere (v6)
[12] 443 (v6) ALLOW IN Anywhere (v6)
[13] 465 (v6) ALLOW IN Anywhere (v6)
[14] 993 (v6) ALLOW IN Anywhere (v6)
[15] 995 (v6) ALLOW IN Anywhere (v6)

1
tests/fixtures/generic/ufw.json vendored Normal file
View File

File diff suppressed because one or more lines are too long

22
tests/fixtures/generic/ufw.out vendored Normal file
View File

@ -0,0 +1,22 @@
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), deny (routed)
New profiles: skip
To Action From
-- ------ ----
22/tcp ALLOW IN Anywhere
22/tcp (v6) ALLOW OUT Anywhere (v6)
443/tcp DENY 192.168.0.1 # nice comment
443/udp DENY OUT 192.168.0.7 8080:8081
22/tcp ALLOW 192.168.0.0/24
22/udp ALLOW 192.168.0.0/24 8080:8081 on en0
22/tcp (v6) ALLOW FWD 2405:204:7449:49fc:f09a:6f4a:bc93:1955/64 on en1 #commenting this rule
80 ALLOW IN Anywhere
8080 (v6) REJECT IN Anywhere (v6)
Apache Full ALLOW IN Anywhere # a comment
Apache Full (v6) ALLOW IN Anywhere (v6)
OpenSSH (v6) DENY IN Anywhere (v6)
10.10.10.10 8080 on enp34s0 ALLOW 127.0.0.1 8000
50200:50300/tcp (v6) DENY FWD Anywhere (v6)
Anywhere (v6) LIMIT 2405:204:7449:49fc:f09a:6f4a:bc93:1955 # this is a comment

1
tests/fixtures/ubuntu-18.04/ufw-appinfo-all.json vendored Normal file
View File

@ -0,0 +1 @@
[{"profile":"MSN","title":"MSN Chat","description":"MSN chat protocol (with file transfer and voice)","tcp_list":[1863,6901],"udp_list":[1863,6901],"tcp_ranges":[{"start":6891,"end":6900}],"normalized_tcp_list":[1863,6901],"normalized_tcp_ranges":[{"start":6891,"end":6900}],"normalized_udp_list":[1863,6901]},{"profile":"OpenSSH","title":"Secure shell server, an rshd replacement","description":"OpenSSH is a free implementation of the Secure Shell protocol.","tcp_list":[22],"normalized_tcp_list":[22]},{"profile":"TEST","title":"My test app","description":"a longer description of the test app here.","tcp_list":[1,2,3,4,5,6,7,8,9,10,30,53],"tcp_ranges":[{"start":80,"end":90},{"start":8080,"end":8090}],"udp_ranges":[{"start":50,"end":51}],"udp_list":[53],"normalized_tcp_list":[1,2,3,4,5,6,7,8,9,10,30,53],"normalized_tcp_ranges":[{"start":80,"end":90},{"start":8080,"end":8090}],"normalized_udp_list":[53],"normalized_udp_ranges":[{"start":50,"end":51}]},{"profile":"TEST2","title":"My test app2","description":"a longer description of the test app here.","tcp_ranges":[{"start":0,"end":65535}],"udp_ranges":[{"start":50,"end":51}],"tcp_list":[53],"udp_list":[53],"normalized_tcp_ranges":[{"start":0,"end":65535}],"normalized_udp_list":[53],"normalized_udp_ranges":[{"start":50,"end":51}]},{"profile":"TEST3","title":"My test app3","description":"test overlapping ports","tcp_list":[80,83,80,53],"tcp_ranges":[{"start":70,"end":90}],"udp_ranges":[{"start":50,"end":51}],"udp_list":[53],"normalized_tcp_list":[53],"normalized_tcp_ranges":[{"start":70,"end":90}],"normalized_udp_list":[53],"normalized_udp_ranges":[{"start":50,"end":51}]}]

51
tests/fixtures/ubuntu-18.04/ufw-appinfo-all.out vendored Normal file
View File

@ -0,0 +1,51 @@
Profile: MSN
Title: MSN Chat
Description: MSN chat protocol (with file transfer and voice)
Ports:
1863
6891:6900/tcp
6901
--
Profile: OpenSSH
Title: Secure shell server, an rshd replacement
Description: OpenSSH is a free implementation of the Secure Shell protocol.
Port:
22/tcp
--
Profile: TEST
Title: My test app
Description: a longer description of the test app here.
Ports:
1,2,3,4,5,6,7,8,9,10,30,80:90,8080:8090/tcp
50:51/udp
53
--
Profile: TEST2
Title: My test app2
Description: a longer description of the test app here.
Ports:
any/tcp
50:51/udp
53
--
Profile: TEST3
Title: My test app3
Description: test overlapping ports
Ports:
80,83,80,70:90/tcp
50:51/udp
53

1
tests/fixtures/ubuntu-18.04/ufw-numbered.json vendored Normal file
View File

File diff suppressed because one or more lines are too long

19
tests/fixtures/ubuntu-18.04/ufw-numbered.out vendored Normal file
View File

@ -0,0 +1,19 @@
Status: active
To Action From
-- ------ ----
[ 1] 22 ALLOW IN Anywhere
[ 2] 80,443/tcp ALLOW IN Anywhere
[ 3] 80 on eth0 ALLOW IN Anywhere # test
[ 4] 10.0.0.1/ipv6 ALLOW IN 10.4.0.0/16/ipv6
[ 5] 10.0.0.1/esp ALLOW IN Anywhere
[ 6] 10.0.0.1/esp ALLOW IN 10.4.0.0/16/esp
[ 7] 10.0.0.1/ah ALLOW IN Anywhere
[ 8] 10.0.0.1/ah ALLOW IN 10.4.0.0/16/ah
[ 9] 100:200,300:400/tcp ALLOW IN Anywhere
[10] 1,2,100:200,300:400/udp ALLOW IN Anywhere
[11] 22 (v6) ALLOW IN Anywhere (v6)
[12] 80,443/tcp (v6) ALLOW IN Anywhere (v6)
[13] 80 (v6) on eth0 ALLOW IN Anywhere (v6) # test
[14] 100:200,300:400/tcp (v6) ALLOW IN Anywhere (v6)
[15] 1,2,100:200,300:400/udp (v6) ALLOW IN Anywhere (v6)

1
tests/fixtures/ubuntu-18.04/ufw-verbose.json vendored Normal file
View File

File diff suppressed because one or more lines are too long

22
tests/fixtures/ubuntu-18.04/ufw-verbose.out vendored Normal file
View File

@ -0,0 +1,22 @@
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), disabled (routed)
New profiles: skip
To Action From
-- ------ ----
22 ALLOW IN Anywhere
80,443/tcp ALLOW IN Anywhere
80 on eth0 ALLOW IN Anywhere # test
10.0.0.1/ipv6 ALLOW IN 10.4.0.0/16/ipv6
10.0.0.1/esp ALLOW IN Anywhere
10.0.0.1/esp ALLOW IN 10.4.0.0/16/esp
10.0.0.1/ah ALLOW IN Anywhere
10.0.0.1/ah ALLOW IN 10.4.0.0/16/ah
100:200,300:400/tcp ALLOW IN Anywhere
1,2,100:200,300:400/udp ALLOW IN Anywhere
22 (v6) ALLOW IN Anywhere (v6)
80,443/tcp (v6) ALLOW IN Anywhere (v6)
80 (v6) on eth0 ALLOW IN Anywhere (v6) # test
100:200,300:400/tcp (v6) ALLOW IN Anywhere (v6)
1,2,100:200,300:400/udp (v6) ALLOW IN Anywhere (v6)

94
tests/test_ufw.py Normal file
View File

@ -0,0 +1,94 @@
import os
import json
import unittest
import jc.parsers.ufw
THIS_DIR = os.path.dirname(os.path.abspath(__file__))
class MyTests(unittest.TestCase):
def setUp(self):
# input
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/ubuntu-18.04/ufw-verbose.out'), 'r', encoding='utf-8') as f:
self.ubuntu_18_04_ufw_verbose = f.read()
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/ubuntu-18.04/ufw-numbered.out'), 'r', encoding='utf-8') as f:
self.ubuntu_18_04_ufw_numbered = f.read()
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/generic/ufw.out'), 'r', encoding='utf-8') as f:
self.generic_ufw = f.read()
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/generic/ufw-numbered.out'), 'r', encoding='utf-8') as f:
self.generic_ufw_numbered = f.read()
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/generic/ufw-numbered2.out'), 'r', encoding='utf-8') as f:
self.generic_ufw_numbered2 = f.read()
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/generic/ufw-inactive.out'), 'r', encoding='utf-8') as f:
self.generic_ufw_inactive = f.read()
# output
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/ubuntu-18.04/ufw-verbose.json'), 'r', encoding='utf-8') as f:
self.ubuntu_18_04_ufw_verbose_json = json.loads(f.read())
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/ubuntu-18.04/ufw-numbered.json'), 'r', encoding='utf-8') as f:
self.ubuntu_18_04_ufw_numbered_json = json.loads(f.read())
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/generic/ufw.json'), 'r', encoding='utf-8') as f:
self.generic_ufw_json = json.loads(f.read())
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/generic/ufw-numbered.json'), 'r', encoding='utf-8') as f:
self.generic_ufw_numbered_json = json.loads(f.read())
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/generic/ufw-numbered2.json'), 'r', encoding='utf-8') as f:
self.generic_ufw_numbered2_json = json.loads(f.read())
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/generic/ufw-inactive.json'), 'r', encoding='utf-8') as f:
self.generic_ufw_inactive_json = json.loads(f.read())
def test_ufw_nodata(self):
"""
Test 'ufw' with no data
"""
self.assertEqual(jc.parsers.ufw.parse('', quiet=True), {})
def test_ufw_ubuntu_18_04_verbose(self):
"""
Test 'ufw status verbose' on Ubuntu 18.04
"""
self.assertEqual(jc.parsers.ufw.parse(self.ubuntu_18_04_ufw_verbose, quiet=True), self.ubuntu_18_04_ufw_verbose_json)
def test_ufw_ubuntu_18_04_numbered(self):
"""
Test 'ufw status numbered' on Ubuntu 18.04
"""
self.assertEqual(jc.parsers.ufw.parse(self.ubuntu_18_04_ufw_numbered, quiet=True), self.ubuntu_18_04_ufw_numbered_json)
def test_ufw_generic_verbose(self):
"""
Test 'ufw status verbose' sample
"""
self.assertEqual(jc.parsers.ufw.parse(self.generic_ufw, quiet=True), self.generic_ufw_json)
def test_ufw_generic_verbose_numbered(self):
"""
Test 'ufw status verbose numbered' sample
"""
self.assertEqual(jc.parsers.ufw.parse(self.generic_ufw_numbered, quiet=True), self.generic_ufw_numbered_json)
def test_ufw_generic_verbose_numbered2(self):
"""
Test 'ufw status verbose numbered' sample
"""
self.assertEqual(jc.parsers.ufw.parse(self.generic_ufw_numbered2, quiet=True), self.generic_ufw_numbered2_json)
def test_ufw_generic_inactive(self):
"""
Test 'ufw status' when firewall is inactive
"""
self.assertEqual(jc.parsers.ufw.parse(self.generic_ufw_inactive, quiet=True), self.generic_ufw_inactive_json)
if __name__ == '__main__':
unittest.main()

82
tests/test_ufw_appinfo.py Normal file
View File

@ -0,0 +1,82 @@
import os
import json
import unittest
import jc.parsers.ufw_appinfo
THIS_DIR = os.path.dirname(os.path.abspath(__file__))
class MyTests(unittest.TestCase):
def setUp(self):
# input
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/ubuntu-18.04/ufw-appinfo-all.out'), 'r', encoding='utf-8') as f:
self.ubuntu_18_04_ufw_appinfo_all = f.read()
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/generic/ufw-appinfo-test.out'), 'r', encoding='utf-8') as f:
self.generic_ufw_appinfo_test = f.read()
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/generic/ufw-appinfo-test2.out'), 'r', encoding='utf-8') as f:
self.generic_ufw_appinfo_test2 = f.read()
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/generic/ufw-appinfo-test3.out'), 'r', encoding='utf-8') as f:
self.generic_ufw_appinfo_test3 = f.read()
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/generic/ufw-appinfo-msn.out'), 'r', encoding='utf-8') as f:
self.generic_ufw_appinfo_msn = f.read()
# output
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/ubuntu-18.04/ufw-appinfo-all.json'), 'r', encoding='utf-8') as f:
self.ubuntu_18_04_ufw_appinfo_all_json = json.loads(f.read())
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/generic/ufw-appinfo-test.json'), 'r', encoding='utf-8') as f:
self.generic_ufw_appinfo_test_json = json.loads(f.read())
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/generic/ufw-appinfo-test2.json'), 'r', encoding='utf-8') as f:
self.generic_ufw_appinfo_test2_json = json.loads(f.read())
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/generic/ufw-appinfo-test3.json'), 'r', encoding='utf-8') as f:
self.generic_ufw_appinfo_test3_json = json.loads(f.read())
with open(os.path.join(THIS_DIR, os.pardir, 'tests/fixtures/generic/ufw-appinfo-msn.json'), 'r', encoding='utf-8') as f:
self.generic_ufw_appinfo_msn_json = json.loads(f.read())
def test_ufw_appinfo_nodata(self):
"""
Test 'ufw_appinfo' with no data
"""
self.assertEqual(jc.parsers.ufw_appinfo.parse('', quiet=True), [])
def test_ufw_appinfo_ubuntu_18_04_all(self):
"""
Test 'ufw app info all' on Ubuntu 18.04
"""
self.assertEqual(jc.parsers.ufw_appinfo.parse(self.ubuntu_18_04_ufw_appinfo_all, quiet=True), self.ubuntu_18_04_ufw_appinfo_all_json)
def test_ufw_appinfo_generic_test(self):
"""
Test 'ufw app info [application]' sample
"""
self.assertEqual(jc.parsers.ufw_appinfo.parse(self.generic_ufw_appinfo_test, quiet=True), self.generic_ufw_appinfo_test_json)
def test_ufw_appinfo_generic_test2(self):
"""
Test 'ufw app info [application]' sample
"""
self.assertEqual(jc.parsers.ufw_appinfo.parse(self.generic_ufw_appinfo_test2, quiet=True), self.generic_ufw_appinfo_test2_json)
def test_ufw_appinfo_generic_test3(self):
"""
Test 'ufw app info [application]' sample
"""
self.assertEqual(jc.parsers.ufw_appinfo.parse(self.generic_ufw_appinfo_test3, quiet=True), self.generic_ufw_appinfo_test3_json)
def test_ufw_appinfo_generic_msn(self):
"""
Test 'ufw app info MSN' sample
"""
self.assertEqual(jc.parsers.ufw_appinfo.parse(self.generic_ufw_appinfo_msn, quiet=True), self.generic_ufw_appinfo_msn_json)
if __name__ == '__main__':
unittest.main()

6
tests/test_utils.py
View File

@ -45,7 +45,7 @@ class MyTests(unittest.TestCase):
for input_string, expected_output in datetime_map.items(): for input_string, expected_output in datetime_map.items():
self.assertEqual(jc.utils.timestamp(input_string).__dict__, expected_output) self.assertEqual(jc.utils.timestamp(input_string).__dict__, expected_output)
def test_convert_to_int(self): def test_utils_convert_to_int(self):
io_map = { io_map = {
None: None, None: None,
True: 1, True: 1,
@ -72,7 +72,7 @@ class MyTests(unittest.TestCase):
for input_string, expected_output in io_map.items(): for input_string, expected_output in io_map.items():
self.assertEqual(jc.utils.convert_to_int(input_string), expected_output) self.assertEqual(jc.utils.convert_to_int(input_string), expected_output)
def test_convert_to_float(self): def test_utils_convert_to_float(self):
io_map = { io_map = {
None: None, None: None,
True: 1.0, True: 1.0,
@ -99,7 +99,7 @@ class MyTests(unittest.TestCase):
for input_string, expected_output in io_map.items(): for input_string, expected_output in io_map.items():
self.assertEqual(jc.utils.convert_to_float(input_string), expected_output) self.assertEqual(jc.utils.convert_to_float(input_string), expected_output)
def test_convert_to_bool(self): def test_utils_convert_to_bool(self):
io_map = { io_map = {
None: False, None: False,
True: True, True: True,