htf — Command line utility¶
The HILSTER Testing Framework comes with a command line utility called htf
.
htf
can be used to run tests, open docs and view the included
open-source licenses.
Running tests¶
To run all tests within the current folder simply run
htf
By default htf
creates an HTML-testreport called testreport.html
in the current folder.
Specifying tests¶
htf
may be followed by run
and <tests>
. You can also omit run
and only supply test-specifiers, eg.
htf run <tests>
respectively
htf <tests>
<tests>
is a list of test-specifiers. They are separated by whitespaces.
A test-specifier may be a python module, a folder or an import string. Test-specifiers can be mixed.
If a test-specifier is a python module the file is imported and all testable objects are collected into a test suite that is used for the test run.
htf test_1.py
If a test-specifier is a folder the folder is searched recursively and all python modules found within (see File pattern) are imported and all testable objects are collected into a test suite. Subsequently the test suite is executed.
Consider the folder tests
, which contains python modules with unit tests.
To execute all test cases in the folder tests
, simply run
htf tests
In case tests
is an importable package or module, located in the python search path, the names might collide.
In this case, simply use
htf tests/
If a test-specifier is an import string, this may be a package, a module, a class (inheriting from htf.TestCase
)
or a method.
If a test-specifier is a package or a module, it is imported and containing tests are executed.
htf tests # Example for a python package containing tests
htf tests.test_example # Example for a python module containing tests
If a class is used it should be a testcase inheriting from htf.TestCase
.
This testcase is used in a test suite and all tests are run.
htf tests.test_example.ExampleTestCase # Example for a class containing tests
If a method is used it should be a method within a test case.
htf tests.test_example.ExampleTestCase.test_one # Example for a test method
Running multiple tests¶
To run multiple tests within a test run simply add more test-specifiers at the end.
htf test1 test2 test3
Different test-specifiers can also be mixed
htf tests/ test1.Test1TestCase test2.py
Test-title¶
To specify the test’s title add the -T
or --title
option followed by a string
that will be used as the title. If the title contains whitespaces, make sure to put quotes around it.
htf --title="This is my first test" tests/
HTML-report¶
By default htf
generates an HTML-report called testreport.html
.
To specify an HTML-report add -H
or --html-report
.
htf --html-report="tests.html" tests/
This option may be repeated to generate multiple HTML-reports.
htf --html-report="tests1.html" --html-report="tests2.html" tests/
JUnit-XML-report¶
htf
is also able to generate JUnit-XML-compatible XML-reports. The reports can be evaluated by different tools, eg.
Jenkins.
To specify an XML-report add -x
or --junit-xml-report
.
htf --junit-xml-report=results.xml tests/
This option may also be repeated.
JSON-report¶
htf
is also able to generate JSON-reports.
To specify a JSON-report add -j
or --json-report
.
htf --json-report=results.json tests/
This option may also be repeated.
Filename-templates¶
Filenames for reports can include prepopulated templates.
Available templates:
{{title}}
- the title as a filename{{date}}
- the current date"%Y-%m-%d_%H-%M-%S"
{{hostname}}
,{{host}}
or{{node}}
- the hostname{{htf_version}}
or{{version}}
- the htf version, eg.htf_1.0.0
{{python_version}}
- the python version, eg.Python_3.5.1
To generate an HTML-report containing the current node and the current date use
htf -H test_{{host}}_{{date}}.html tests/
Verbose output¶
To enable verbose output add -v
or --verbose
.
Catch CTRL-C¶
Long-running test suites can be terminated using a keyboard interrupt (CTRL-C
). Normally,
the execution is terminated immediately and no report is generated.
If -c
or --catch
is used and CTRL-C
is pressed the
current test is run to completion and the report is generated.
No further tests are run afterwards.
Fail fast¶
If -f
or --failfast
is used the test run ends after the first
failing test.
File pattern¶
If a test-specifier is a folder the default pattern used for file discovery is test*.py
.
The pattern can be changed using the -p
or --pattern
options.
htf -p \*.py tests/
Wildcards (*) are supported. Depending on your shell, escaping must be applied.
Bash-completion¶
htf
is also shipped with complete-htf
which provides
automatic command completion for the bash
shell.
complete-htf
completes the first positional commands (run
,
docs
and licenses
) but also expands test-specifiers
(eg. file names, packages, modules, classes and methods).
Using the Bash-completion makes it easy to specify tests (without typing-errors).
Let’s assume you a have an importable python package called tests
.
Two test-modules called test_1.py
and test_2.py
are located within
tests
.
Both contain a test-case called TestCase1
and TestCase2
, respectively.
To easily run tests.test_1.TestCase1
, type
htf te<TAB>
tests tests.
htf tests.<TAB>
htf tests.test_<TAB>
tests.test_1 tests.test_1. tests.test_2 tests.test_2.
htf tests.test_1.<TAB>
htf tests.test_1.TestCa<TAB>
htf tests.test_1.TestCase<TAB>
tests.test_1.TestCase tests.test_1.TestCase.
htf tests.test_1.TestCase<ENTER>
# ...
Enable Bash-completion¶
To enable Bash-completion type
complete -f -C 'complete-htf' htf
To persistently enable bash completion, add the command to your ~.bashrc
.
alternatively put it into a new file called /etc/bash_completion.d/htf
, eg.
echo "complete -f -C 'complete-htf' htf" >> ~/.bashrc
. ~/.bashrc
or
echo "complete -f -C 'complete-htf' htf" > /etc/bash_completion.d/htf
. /etc/bash_completion.d/htf
To check if Bash-completion is enabled type
complete -p | grep htf