The network measurements execution framework.
Netrics streamlines the development, debugging, distribution and periodic execution of tasks – specifically supporting the execution of computer network measurements.
This software is both a framework and a bundle: Netrics is distributed with built-in, research-backed measurements, and the installed software may be trivially configured and extended.
The project aims to support network researchers, engineers, providers and users.
Features:
-
Configuration for Humans: in either YAML or TOML formats
-
Measurements built in: Provide your own measurements or hit the ground running with built-in, pre-configured measurements from Ookla, Measurement Lab, iPerf, and more
-
Open, operating system-level interfaces: measurements are implemented as arbitrary executables reading structured standard input parameters, writing structured standard output results, etc.
-
Diverse installation targets: native support for *nix devices featuring Python version 3.8 or later, via pip or via all-in-one pre-built bundles – and, Docker images, too!
-
Debugging tools: command-line utilities to test and debug measurement executables – on your laptop or on your Raspberry Pi
-
Python SDK: simple tools to help in the implementation of measurements in Python (and serving as prototypes for their implementation in other languages)
|
💡
|
Not interested in network measurements? Netrics is itself a distribution of Fate, a general-purpose, feature-rich scheduler for periodic tasks. |
Not quite ready to dive in? Check out our guide in the netrics-demo repository.
Netrics publishes a number of Debian-based Docker images, for the ease of users familiar with Docker, to support otherwise unsupported systems, and otherwise to provision Netrics in systems utilizing containers.
For example, to get started with the default Netrics Docker image:
docker run ghcr.io/internet-equity/netricsThe "one-line" installation script is the recommended method for most humans to get started with Netrics.
The script will download and unpack the pre-built bundle appropriate to your system, as well as perform the software’s initialization.
The one-liner below will install Netrics for the current user:
curl -sL https://github.com/internet-equity/netrics/raw/0.0.1/install.py | python3To install Netrics system-wide you might use sudo:
curl -sL https://github.com/internet-equity/netrics/raw/0.0.1/install.py | sudo python3|
💡
|
You may pass arguments to the installer – even in the above form – separated from the Python executable with a single dash: curl -sL https://github.com/internet-equity/netrics/raw/0.0.1/install.py | python3 - --helpAnd, for example, if the installer cannot find a suitable location for the Netrics executables – one which is both writeable by the current user and on the |
Alternatively, download the installer to interact with it:
curl -LO https://github.com/internet-equity/netrics/raw/0.0.1/install.py
python3 install.py --helpNetrics software is implemented in Python and requires Python version 3.8 or above.
Casual inspection of your system’s /usr/bin/ directory may indicate which versions of Python you have installed:
ls -1 /usr/bin/python3.*Python installed? Great. That’s all the pre-built bundle needs.
Find the appropriate build for your system on the Netrics release announcement page.
For example, the below package includes the Netrics software bundle version 0.0.1, and all its CLI commands, pre-built for the ARM CPU architecture (aarch64) and Python version 3.8 (py38):
netrics-all-0.0.1-py38-aarch64.zip
Having downloaded the above ZIP file, installation is as simple as the following:
sudo unzip netrics-all-0.0.1-py38-aarch64.zip -d /usr/local/binAlternatively, install Netrics to any directory from which you’d like to execute it, or to any writeable directory on your PATH, e.g.:
unzip netrics-all-0.0.1-py38-aarch64.zip -d ~/.local/bin|
💡
|
If you’re into that kind of thing, a (multi)-line no-brainer – making use of the TAR file alternative – might look like the following: ARCH=$(arch)
PYTHON=py$(python3 -c "import sys; print(*sys.version_info[:2], sep='')")
NETRICS=$(curl -s https://api.github.com/repos/internet-equity/netrics/releases/latest | jq -r .tag_name)
curl -sL https://github.com/internet-equity/netrics/releases/download/$NETRICS/netrics-all-$NETRICS-$PYTHON-$ARCH.tar | sudo tar -xf - -C /usr/local/bin/…But, why not use the one-line installer, instead? |
With Python installed, the Package Installer for Python – pip – should be installed, as well. If not, it may be installed with the following:
python3 -m ensurepip --upgradeNetrics may then be installed via pip install. Refer to the sections that follow for specific information on downloading and installing the Netrics package.
|
💡
|
Commands of the form As appropriate to the target system, you might instead install Netrics under your user path only: pip install --user [URI]Or, to ensure successful installation, consider a virtual environment, with which to isolate the library’s dependencies from others on the system. Better yet, consider the additional utility pipx: pipx install [URI]With Finally, consider one of the preceding installation methods, such as the one-line installer, which will attempt to install Netrics as a pre-built, pre-packaged bundle, without risking the above concerns. |
Netrics may be installed from its source repository via pip.
To make use of an SSH configuration, e.g.:
pip install git+ssh://git@github.com/chicago-cdac/netrics.gitNote that the above URI may also include a Git reference specification, such as a tag or a branch:
pip install git+ssh://git@github.com/chicago-cdac/netrics.git@mainAlternatively, you may supply HTTPS URIs to the above.
With HTTPS, it is also possible to request a code archive of a particular tag or branch, (which may be faster than the above):
pip install https://github.com/chicago-cdac/netrics/archive/REF.zipOutside of the Docker container, and installation by a package manager or by the one-line installer, initialization is suggested to set up your Netrics installation.
Pre-built bundle, PyPI and source distributions feature the netrics sub-command init:
netrics initThe above, (executed from a standard shell), will walk you through the process of initializing your system for Netrics, (executing all tasks which follow below).
To initialize configuration in particular, init features the sub-command conf:
netrics init confThe above will copy the built-in default configuration distributed with Netrics to the appropriate path on your system (or to a specified path). From there, this configuration may be customized.
|
ℹ️
|
Netrics is really a distribution of Fate, and as such shares its configuration and execution scheme. |
Netrics expects two configuration files: measurements and defaults.
Should either file not be found on the sytem, Netrics will fall back to its built-in configuration. As necessary for your installation, to initialize these files for customization, see Initialization.
|
💡
|
Netrics supports both TOML and YAML configuration formats. |
|
ℹ️
|
The commands In lieu of these, files |
The measurements file configures and schedules programs to be executed by Netrics. These configured programs are alternately called "measurements," "tasks" and "modules."
Only one setting is strictly required of a measurement: its schedule. (Without this setting, a measurement may be executed ad-hoc via the debug command; however, it cannot be scheduled.)
Additionally, measurement configuration must indicate what is to be executed. This may be indicated either via the setting exec or command, or it will be inferred.
The example below demonstrates configuration options further.
| measurements.toml | measurements.yaml |
|---|---|
[ping]
schedule = "0 */6 * *"
[ping-slim]
command = "ping"
schedule = "*/30 * * *"
param = {target = ["google.com"]}
[cowsay]
exec = "cowsay"
schedule = "@hourly"
param = "yo dawg"
path = {result = "/root/cows/"}
[cowsay-custom]
exec = ["cowsay", "-e", "^^"]
schedule = "@daily"
param = "i heard you like cows"
# no file extension for result files; do not attempt to detect
format = {result = ""}
path = {result = "/root/cows/"}
[dump-db]
exec = ["sh", "/home/ubuntu/dump-db"]
schedule = "@daily"
format = {result = "csv"} |
ping:
schedule: "0 */6 * *"
ping-slim:
command: ping
schedule: "*/30 * * *"
param: {target: [google.com]}
cowsay:
exec: cowsay
schedule: "@hourly"
param: yo dawg
path: {result: /root/cows/}
cowsay-custom:
exec: [cowsay, -e, ^^]
schedule: "@daily"
param: i heard you like cows
# no file extension for result files; do not attempt to detect
format: {result: null}
path: {result: /root/cows/}
dump-db:
exec: [sh, /home/ubuntu/dump-db]
schedule: "@daily"
format: {result: csv} |
In the above example, the "measurements" cowsay, cowsay-custom and dump-db each specify the exec setting. With this setting, a measurement may execute any system command.
Note, however, that Netrics will not, by default, launch a shell to interpret the value of your measurement’s exec setting. This setting must be either a string or an array indicating an executable command available through the process environment’s PATH. Command arguments are only accepted via array notation.
Netrics further features a plug-in system whereby programs abiding by its contract are granted greater functionality. Any program may abide by this contract, (including those specified via exec). Programs available through the process environment’s PATH under a name bearing the netrics- prefix – e.g., netrics-ping – enjoy the small privilege of becoming Netrics "commands."
In the above example, the measurement ping-slim specifies the command ping. This simply instructs Netrics to execute a program under the name netrics-ping.
The example measurement ping neglects to specify a command at all. The ping command will be inferred for it as well – this is: the program netrics-ping.
Under the framework contract, programs may be given configured parameters via their process’s standard input.
The example measurement ping-slim is configured to input to the ping command the parameters:
{
"target": ["google.com"]
}The cowsay measurement, on the other hand, is configured with the scalar string input: "yo dawg".
Structured (non-scalar) parameters are serialized to JSON by default. (This default may be overidden either per-measurement or globally. See: format.)
The format setting, when specified, must be a mapping.
The defaults of settings nested under format may be overidden per-measurement or globally.
param
The nested setting param indicates the serialization format of structured parameters (given by top-level measurement setting param). JSON (json), TOML (toml) and YAML (yaml) serialization formats are supported. The default format is JSON.
result
The nested setting result indicates in what format results will be produced by the measurement’s
standard output.
The default for this setting is "auto" – Netrics will attempt to characterize the
measurement result format, so as to assign an appropriate extension to its generated file name.
JSON (json), TOML (toml) and YAML (yaml) serializations support "auto" characterization.
Alternatively, the result format may be specified explicitly: in addition to the values json,
toml and yaml, this setting supports csv.
Finally, result characterization may be disabled by any "false-y" value, such as null (in YAML), or the empty string (generally).
The path setting, when specified, must be a mapping.
The defaults of settings nested under path may be overidden per-measurement or globally.
result
The nested setting result indicates the directory path to which measurement result files are written. The default path is installation-dependent (e.g., /var/log/netrics/result/ when Netrics is installed system-wide).
Settings format and path may be overidden globally via the defaults file, as in the example below.
| defaults.toml | defaults.yaml |
|---|---|
[format]
param = "json"
result = "auto"
[path]
result = "/var/log/netrics/result/" |
format:
param: json
result: auto
path:
result: /var/log/netrics/result/ |
Netrics includes a set of built-in measurement commands, such as netrics-ping.
Any task configuration may specify the command setting with the value ping to make use of this built-in; (or, a task with the label ping may omit this setting to default to this command).
| command | executable | parameters (defaults) | description |
|---|---|---|---|
|
|
… |
… |
|
|
… |
… |
|
|
… |
… |
|
|
… |
Run a network diagnostic test using Measurement Lab’s ndt7-client. |
|
|
… |
Run a network diagnostic test using Ookla’s Speedtest. |
|
|
{
"count": 10,
"interval": 0.25,
"targets": [
"facebook.com",
"google.com",
"nytimes.com"
],
"timeout": 5,
"verbose": false
} |
Execute the |
|
|
… |
… |
The Netrics framework invokes executables available to the operating system. As such, built-in measurements enjoy next-to-nil privilege relative to any other installed executable; and, measurements abiding by the framework’s expectations may be added with a minimum of effort.
The framework communicates with the programs it executes through the operating system, principally via processes' standard input, standard output, standard error and exit code.
Minima
An executed task must at minimum:
-
write its result to standard output (though this is ignored if reporting failure)
-
report its success or failure via exit code (only exit code
0indicates successful execution)
|
💡
|
The examples below represent shell scripts; and, Netrics’s built-in measurements are implemented in Python. Tasks may execute any program. And "commands" named with the netrics- prefix may themselves be implemented in any language.
|
This may be accomplished as simply as the following example executable, which reports network status as indicated by sending an ICMP Echo request (ping) to host example.com:
#!/bin/sh
# For this simple example we're not interested in detailed ping data
# (and we don't want it echo'd as a "result") -- discard it.
ping -c 1 -w 1 example.com > /dev/null (1)
# Rather, determine our result according to ping's own exit codes.
case $? in
0)
echo '{"example.com": "FAST ENOUGH"}' (2)
exit 0 (3)
;;
1) (4)
echo '{"example.com": "TOO SLOW"}'
exit 0
;;
*)
exit 1 (5)
;;
esac-
As noted in the preceding comment, care must be taken with shell scripts which pass through sub-processes' standard output and error. Any standard output is treated as part of a measurement’s "result." And any standard error will be logged.
-
Results are reported via an executable’s standard output. Results may be in any plain text format (or none at all). (JSON is merely a handy one, and enjoys automatic detection.)
-
The default exit code of a program is of course
0. It doesn’t hurt to make this explicit: any non-zero exit code indicates to the framework a failed execution. Failures are logged as such. Any content written to standard output by a failed task is not recorded as a measurement result. -
The underlying
pingutility (from Linux package iputils) communicates state with its own exit codes: exit code1indicates packets were not received. This is an error state for iputils; but, for our measurement, this is a valid result. We detect this state, report it, and exit with the success code0. -
Any other case indicates an error with our measurement. We exit with a non-zero exit code to notify the framework of this failure. As this is a shell script, any standard error written by the
pingutility has been passed through and captured; (and, we could write our own).
Parameterization
Tasks' input may be configured in the measurements file and is supplied to executables via their standard input. Structured input is serialized in JSON format by default. (See: param.)
We might extend our example to read and process JSON-encoded standard input via the jq utility:
#!/bin/bash
# collect targets from standard input parameters
#
# we expect input of the form:
#
# {
# "targets": ["host0", "host1", ..., "hostN"]
# }
#
PARAM="$(jq -r '.targets | join(" ")' 2> /dev/null)" (1)
# default to just Wikipedia
if [ -z "$PARAM" ]; then
PARAM="wikipedia.org"
fi (2)
# run all measurements concurrently
# (and collect their PIDs for inspection)
PIDS=()
for dest in $PARAM; do
ping -c 1 -w 1 $dest > /dev/null &
pids+=($!)
done (3)
# collect measurements' exit codes
CODES=()
for pid in ${PIDS[*]}; do
wait $pid
CODES+=($?)
done
# convert exit code to a status
STATUS=()
for code in ${CODES[*]}; do
case "$code" in
0)
STATUS+=("FAST ENOUGH")
;;
1)
STATUS+=("TOO SLOW")
;;
*)
echo 'FAILURE!!!' > &2
exit 1 (4)
;;
esac
done
# generate report
jq '
[ .targets, .statuses | split(" ") ]
| transpose
| map( {(.[0]): .[1]})
' <<DOC
{
"targets": "$PARAM",
"statuses": "${STATUS[@]}"
}
DOC (5)-
It’s perfectly reasonable to log issues with parameterization to standard error. But there might be no input at all. Rather than differentiate these cases in our shell script, we just silence any complaints from
jq. -
The user may elect not to configure any parameters, and so we fall back to a default.
-
Our underlying measurement is much the same as before; only now, we test each configured target in parallel.
-
Any of our measurements could still fail in a way we don’t know how to handle. In this case, this task elects to report the entire run as a failure. Additionally, a profoundly interesting message is logged via standard error.
-
Yikes!!! We elected to write our executable in Bash to show how simple it can be. But there’s nothing simple about that. Admittedly, we might have serialized our result in any format – CSV is supported, for one; and, even space- or tab-separated values would suffice, here. But, now we’ve demonstrated the limits of this implementation, as well. For your executable, you might select another language….
For more robust examples, consult Netrics’s built-in measurement commands (implemented in Python).
Measurement executables may nominally associate themselves with the Netrics framework and become "commands" by simply being available on the process environment PATH under a name beginning with the prefix netrics-.
In this manner, built-in measurements such as netrics-ping are distributed alongside the netrics framework command, and may be referred to in configuration as ping.
Any other discovered executable, such as netrics-cowsay[1], will be treated the same way.
Any executable may be invoked (with optional arguments) by the Netrics execute command:
netrics debug execute [options] command [arguments]The above generates an execution report for use in development and debugging.
Options such as -i|--stdin may be useful to supply measurment parameters to the executable according to the framework’s contract.
Once added to Netrics configuration, executables become tasks, (also known as measurements or modules). These may be invoked ad-hoc by the run command:
netrics debug run [options] taskThe options and output of the run command are similar to those of execute.
Unlike with scheduled tasks, the results of tasks performed by run are not, by default, persisted to file. Either specify option --record to capture these as configured, or option --stdout to capture these at an arbitrary path.
Having tested your novel measurement, it might be added to the Netrics framework for availability across all installations of this software via pull request.
At this time, all Netrics builtins are implemented in Python, as simple submodules of the Netrics sub-package netrics.measurement. As such, built-in measurement module files need not be marked with the "execute" bit nor need they include a "shebang" line (e.g. #!/usr/bin/env python3).
-
Name your module succinctly and appropriately for its functionality. Do not include any
netrics-prefix. E.g.:MOD.py. -
Place your module under the path: src/netrics/measurement/.
-
The functionality of your measurement should be invoked entirely by a module-level function:
main(). This function will be invoked without arguments. -
Optional: Enable invocation of your module through the package –
python -m netrics.measurement.MOD– with the final module-level block:if __name__ == '__main__': main()
-
Configure the Netrics distribution to install your command executable by adding a line to the pyproject.toml file section
tool.poetry.scripts, e.g.:[tool.poetry.scripts] netrics-MOD = "netrics.measurement.MOD:main"
-
Add your command to this document’s table of built-in measurements.
The Netrics framework is implemented in Python and the framework’s distribution is managed via Poetry.
Python v3.8 may be supplied by an operating system package manager, by python.org, or by a utility such as pyenv; pyenv is recommended for development but not required.
With Python installed, Poetry may be installed according to its instructions.
|
💡
|
If you are managing your own virtual environment, e.g. via pyenv-virtualenv, then this step may be as simple as pip install poetry. However, this tooling is not required, and Poetry offers its own automated set-up, as well as management of virtual environments.
|
Finally, from the root directory of a repository clone, the framework may be installed in development mode:
poetry install|
ℹ️
|
Poetry will use any existing, activated virtual environment, or create its own into which dependencies will be installed. |
The netrics command is now available for use in your development environment.
|
❗
|
For simplicity, it is presumed that A virtual environment under management by Poetry may be activated via sub-shell with: poetry shellAlternatively, any command installed into Poetry’s virtual environment may be executed ad-hoc via the poetry run netrics ... |
netrics-cowsay … yet!