test-runner - vpsAdminOS test suite evaluator
test-runner [global options] command [command options] [arguments...]
test-runner is a command-line tool to evaluate vpsAdminOS test suite, running
selected tests and reporting results.
ls [path-pattern] -l, --label label=value | label!=value
Filter tests by selected label, which is either tested for
equality or inequality.
-t, --tag tag|^tag
Filter tests that have tag set. If the tag begins with ^, then
filter tests that do not have tag set.
--filter expr
Filter tests by metadata expression. Expressions can test tags and labels,
combine conditions with && and ||, and group them with parentheses.
Use tag=value to require a tag and tag!=value to reject a tag.
Other keys are treated as labels, e.g. runtime!=long.
--system system
Nix system to evaluate tests for. Defaults to x86_64-linux.
--test-config path
Path to a Nix file returning additional test framework configuration.
Requires the tested flake to export
lib.testFramework.mkTests and lib.testFramework.mkTestsMeta.
test [path-pattern] -l, --label label=value | label!=value
Filter tests by selected label, which is either tested for
equality or inequality.
-t, --tag tag|^tag
Filter tests that have tag set. If the tag begins with ^, then
filter tests that do not have tag set.
--filter expr
Filter tests by metadata expression. Expressions can test tags and labels,
combine conditions with && and ||, and group them with parentheses.
Use tag=value to require a tag and tag!=value to reject a tag.
Other keys are treated as labels, e.g. runtime!=long.
-j, --jobs n|auto
Maximum number of tests to run in parallel. When set to auto, the
runner starts enough scheduler workers for all selected tests and lets
resource reservations decide how many VMs can actually run.
--max-memory-mib n
Maximum memory available to running test VMs, in MiB. Detected memory
capacity is refreshed during the run; this value is an upper bound and is
used as a fallback when detection is unavailable.
--max-shm-mib n
Maximum /dev/shm space available to running test VMs, in MiB. Detected
capacity is refreshed during the run; this value is an upper bound and is
used as a fallback when detection is unavailable.
--max-cpus n
Maximum CPUs available to running test VMs. Detected CPU capacity is
refreshed during the run; this value is an upper bound and is used as a
fallback when detection is unavailable.
--memory-overcommit factor
Multiply detected memory capacity by factor before applying the
configured reserve and maximum. Defaults to 1.0.
--shm-overcommit factor
Multiply detected /dev/shm capacity by factor before applying the
configured reserve and maximum. Defaults to 1.0.
--cpu-overcommit factor
Multiply detected CPU capacity by factor before applying the configured
reserve and maximum. Defaults to 1.5.
--resource-refresh-interval seconds
How often to refresh detected resource capacity while scheduling tests.
Defaults to 15.
--memory-reserve-mib n
Memory to keep reserved from detected or configured capacity, in MiB.
--shm-reserve-mib n
/dev/shm space to keep reserved from detected or configured capacity,
in MiB.
--cpu-reserve n
CPUs to keep reserved from detected or configured capacity.
-f, --fresh
Recreate disk files for virtual machines if they already exist.
--system system
Nix system to evaluate tests for. Defaults to x86_64-linux.
--test-config path
Path to a Nix file returning additional test framework configuration.
Requires the tested flake to export
lib.testFramework.mkTests and lib.testFramework.mkTestsMeta.
--timeout n
Default timeout for machine commands that wait until execution becomes
possible, or until a command fails or succeeds. This option changes
the default value, which is used when tests do not set the timeout
themselves. In seconds, defaults to 900.
--stop-on-failure
Stop further execution when a test fails.
--destructive
Determines whether machine disk files are kept
--state-dir dir
Set custom path to directory where generated configs, logs, and test
state are stored.
Defaults to $TMPDIR or /tmp.
--system system
Nix system to evaluate tests for. Defaults to x86_64-linux.
--test-config path
Path to a Nix file returning additional test framework configuration.
--filter accepts boolean expressions over test script metadata:
tag=ci matches scripts tagged ci.tag!=manual matches scripts without tag manual.runtime=long matches scripts with label runtime set to long.runtime!=long matches scripts whose runtime label is not long.Use && for AND, || for OR and parentheses for grouping. && binds more
strongly than ||. Multiple --filter, --tag and --label options are
combined with AND.
Examples:
test-runner ls --filter 'tag=ci && tag!=manual'
test-runner test --filter 'tag=ci && (tag=vps || tag=storage)'
test-runner test -t ci --filter 'runtime!=long && (tag=vps || tag=storage)'
./test-runner.sh test -t ci --filter 'tag=vps || tag=storage'
./test-runner.sh ls --filter 'tag=ci && runtime!=long'
Shell quoting is required for expressions containing &&, || or parentheses.
debug test --state-dir dir
Set custom path to directory where generated configs, logs, and test
state are stored.
Defaults to $TMPDIR or /tmp.
Report bugs to https://github.com/vpsfreecz/vpsadminos/issues.
test-runner is a part of vpsAdminOS.