Fluster is a testing framework written in Python for decoders conformance. It is composed of a CLI application that runs a number of test suites with the supported decoders. Its purpose is to check different decoder implementations against known test suites with known and tested results. It has been originally designed to check the conformance of H.265/HEVC decoders, but it also supports H.264/AVC and can be easily extended to add more decoders and test suites.
Fluster requires Python 3.6+ to work. It has zero dependencies apart from that. The requirements.txt file includes Python's modules used only for development.
The framework works with test suites. Each test suite is associated with one codec and contains a number of test vectors. Each test vector consists of an input file and and the expected result. The input file will be fed into each decoder that supports the codec of the test suite. The file format is a JSON file. You can find the ones included in the test_suites directory.
The decoders are the ones in charge of doing the decoding given an input file.
They implement two methods: decode which is mandatory and check which is
optional. Check out the decoder class for reference. The
@register_decoder decorator is used to ensure the framework takes them into
account. Fluster is agnostic as to how the decoding itself is done. So far, all
decoders are external processes that need to run with a number of parameters,
but they could actually be decoders written in Python as far as Fluster is
concerned. The decoders directory contains all supported
decoders.
In order to run the tests for the different test suites and decoders, a
resources directory containing all the input files for each test suite needs
to be collected first.
-
Download the resources for the test suites. This will create the
resourcesdirectory containing the input files for each test suite../fluster.py downloadwill spawn a number of processes to download and extract the files. You can change the number of parallel processes used with-j. It defaults to 2x number of logical cores. -
(Optional) Build the reference decoders for H.264/AVC and H.265/HEVC running
make decoders. It assumes you have CMake and a native compiler such as gcc or clang installed so that they can be built. The resulting binaries will be moved to a newdecodersdirectory in the root. -
List the test suites, the decoders and which of them can run using
./fluster.py list -c
List of available test suites:
JVT-AVC_V1
Codec: Codec.H264
Description: JVT AVC version 1
Test vectors: 135
JCT-VC-HEVC_V1
Codec: Codec.H265
Description: JCT-VC HEVC version 1
Test vectors: 147
dummy
Codec: Codec.Dummy
Description: Dummy test suite
Test vectors: 1
List of available decoders:
Dummy
Dummy: This is a dummy implementation for the dummy codec ✔️
H264
Fluendo-H.264-SW-Gst0.10: Fluendo H.264 SW decoder for GStreamer 0.10 ❌
Fluendo-H.264-SW-Gst1.0: Fluendo H.264 SW decoder for GStreamer 1.0 ❌
GStreamer-H.264-VA-API-Gst1.0: GStreamer H.264 VA-API decoder for GStreamer 1.0 ✔️
JCT-VT-H264: JCT-VT H.264/AVC reference decoder ✔️
FFmpeg-H264: FFmpeg H.264 decoder ✔️
H265
Fluendo-H.265-HW-Gst1.0: Fluendo H.265 HW decoder for GStreamer 1.0 ✔️
Fluendo-H.265-SW-Gst0.10: Fluendo H.265 SW decoder for GStreamer 0.10 ❌
Fluendo-H.265-SW-Gst1.0: Fluendo H.265 SW decoder for GStreamer 1.0 ❌
GStreamer-H.265-VA-API-Gst1.0: GStreamer H.265 VA-API decoder for GStreamer 1.0 ✔️
JCT-VT-H265: JCT-VT H.265/HEVC reference decoder ✔️
FFmpeg-H265: FFmpeg H.265 decoder ✔️
-
Run the test suite (or a number of them) for all decoders (or a number of them). By default, decoder test are run in parallel. By default it uses the same amount of parallel jobs as number of cores, but it can be configured using the
-joption. You can pass-dto filter only the decoders that you want to run,-tsfor the test suites and-tvfor the test vectors. Examples:./fluster.py runruns all test suites for all decoders available that match each test suite's codec./fluster.py run -ts JCT-VC-HEVC_V1runs the JCT-VC-HEVC_V1 test suite for all decoders that support H.265/HEVC./fluster.py run -ts JCT-VC-HEVC_V1 -tv AMP_A_Samsung_7runs only the test vector AMP_A_Samsung_7 of the JCT-VC-HEVC_V1 test suite./fluster.py run -d FFmpeg-H265runs the FFmpeg-H265 decoder on all test suites for H.265/HEVC./fluster.py run -d FFmpeg-H265 -j1runs the FFmpeg-H265 decoder on all test suites for H.265/HEVC using one job
- H.264/AVC
- H.265/HEVC
- Dummy test suite for testing purposes
- JCT-VT H.265/HEVC as reference decoder for H.265/HEVC
- JCT-VT H.264/AVC as reference decoder for H.264/AVC
- GStreamer's vaapih265dec for H.265/HEVC
- GStreamer's vaapih264dec for H.264/AVC
- FFmpeg's H.265/HEVC
- FFmpeg's H.264/AVC
- Fluendo's propietary decoders for H.264/AVC and H.265/HEVC that are included in Fluendo Codec Pack
- Dummy decoder for testing purposes
usage: fluster.py [-h] [-v] {list,l,run,r,download,d,reference} ...
optional arguments:
-h, --help show this help message and exit
-v, --verbose increase output verbosity
subcommands:
{list,l,run,r,download,d,reference}
list (l) show list of available test suites and decoders
run (r) run test suites for decoders
download (d) downloads test suites resources
reference (r) use a specific decoder to set its results for the test suites given
./fluster.py --help
usage: fluster.py [-h] [-r RESOURCES] [-o OUTPUT] {list,l,run,r,download,d,reference} ...
optional arguments:
-h, --help show this help message and exit
-r RESOURCES, --resources RESOURCES
set the directory where resources are taken from
-o OUTPUT, --output OUTPUT
set the directory where test results will be stored
subcommands:
{list,l,run,r,download,d,reference}
list (l) show list of available test suites and decoders
run (r) run test suites for decoders
download (d) downloads test suites resources
reference (r) use a specific decoder to set its results for the test suites given
./fluster.py run --help
usage: fluster.py run [-h] [-j JOBS] [-t TIMEOUT] [-ff] [-q] [-ts TESTSUITES [TESTSUITES ...]] [-tv TESTVECTORS [TESTVECTORS ...]] [-d DECODERS [DECODERS ...]] [-s] [-k] [-th THRESHOLD] [-tth TIME_THRESHOLD]
optional arguments:
-h, --help show this help message and exit
-j JOBS, --jobs JOBS number of parallel jobs to use. 1x logical cores by default. 0 means all logical cores
-t TIMEOUT, --timeout TIMEOUT
timeout in secs for each decoding. Defaults to 30 secs
-ff, --failfast stop after first fail
-q, --quiet don't show every test run
-ts TESTSUITES [TESTSUITES ...], --testsuites TESTSUITES [TESTSUITES ...]
run only the specific test suites
-tv TESTVECTORS [TESTVECTORS ...], --testvectors TESTVECTORS [TESTVECTORS ...]
run only the specific test vectors
-d DECODERS [DECODERS ...], --decoders DECODERS [DECODERS ...]
run only the specific decoders
-s, --summary generate a summary in Markdown format for each test suite
-k, --keep keep output files generated during the test
-th THRESHOLD, --threshold THRESHOLD
set exit code to 2 if threshold tests are not success. exit code is 0 otherwise
-tth TIME_THRESHOLD, --time-threshold TIME_THRESHOLD
set exit code to 3 if test suite takes longer than treshold seconds. exit code is 0 otherwise
./fluster.py download --help
usage: fluster.py download [-h] [-j JOBS] [-k] [testsuites [testsuites ...]]
positional arguments:
testsuites list of testsuites to download
optional arguments:
-h, --help show this help message and exit
-j JOBS, --jobs JOBS number of parallel jobs to use. 2x logical cores by default.0 means all logical cores
-k, --keep keep downloaded file after extracting
./fluster.py reference --help
usage: fluster.py reference [-h] [-j JOBS] [-t TIMEOUT] [-q] decoder testsuites [testsuites ...]
positional arguments:
decoder decoder to run
testsuites list of testsuites to run the decoder with
optional arguments:
-h, --help show this help message and exit
-j JOBS, --jobs JOBS number of parallel jobs to use. 1x logical cores by default.0 means all logical cores
-t TIMEOUT, --timeout TIMEOUT
timeout in secs for each decoding. Defaults to 5 secs
-q, --quiet don't show every test run
Fluster in English means to (cause to) become nervous or confused. It looks a very appropriate name for testing decoders.
- Create a new decoder in fluster/decoders directory
- Implement the
decodemethod - Use the
register_decoderdecorator - Ensure to set
hw_acceleration = Trueif it requires hardware - Optionally, implement the
checkto know whether the decoder is available to be run
Check out the JSON format they follow in the test_suites directory. Add a new json file within and Fluster will automatically pick it up.
There is also a generator script for the conformance test suites that you can use as a base to generate automatically new ones.
We can easily use Fluster in a CI to test that our test suites for particular decoders are still working as good as they were before. There are two arguments to be used with the run command that can help us achieve that. Both commands work only when running a single test suite:
-
-th/--thresholdsets the minimum number of tests that need to be success in order to consider the command not a failure. In case of failure, the exit code is 2. Please notice that even of some tests fail, the exit code will still be 0 as long as the threshold is met. -
-tth/--time-thresholdsets the maximum amount of time for a test suite to run and be considered a success. The exit code 3 in case it takes longer and 0 otherwise. Please notice that even of some tests fail, the exit code will still be 0 as long as the time it takes is less than the threshold.
- Fork the repo
- Install the required Python modules for development using
pip3 install -r requirements.txt - Install the git hook that will run for every commit to ensure it works before pushing. About git hooks
- Modify the code . Make sure the git hook is properly checking that the
basic functionality still works. You can also execute
make checkmanually - Create a new PR with your changes
- Make sure the GitHub Actions is running and its result is a pass
In case you find any problem or want to report something, don't hesitate to search for similar issues. Only when the issue can't be found, a new one should be created. Please try to provide as many details and context as possible to help us diagnose it.
LGPLv3
Fluster - testing framework for decoders conformance
Copyright (C) 2020, Fluendo, S.A.
Author: Pablo Marcos Oltra <pmarcos@fluendo.com>, Fluendo, S.A.
Author: Andoni Morales Alastruey <amorales@fluendo.com>, Fluendo, S.A.
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Library General Public
License as published by the Free Software Foundation; either
version 2 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Library General Public License for more details.
You should have received a copy of the GNU Library General Public
License along with this library; if not, write to the
Free Software Foundation, Inc., 59 Temple Place - Suite 330,
Boston, MA 02111-1307, USA.