pocket-cnn is a framework to map small Convolutional Neural Networks (CNN) fully on a FPGA. There is no communication outside the FPGA needed, except of providing the image and reading the result.
Note that weights and activations are fixed to 8 bit, due to ONNX limitations. For binary neural networks in hardware, you can check out pocket-bnn.
Before using the framework, the PYTHONPATH has to be extended by path/to/pocket-cnn/code/python_tools. A complete end-to-end example can be found at the example folder.
For running the tests, ghdl, vunit, onnx and fpbinary are required. For debugging, gtkwave and netron are useful. To run the full testsuite, simply execute:
cd code/vhdl/sim/vunit
python3 run_all.pyThis section describes the different workflows. In the end-to-end example, an ONNX model gets trained with pytorch, quantized and processed until it's ready for synthesis by pocket-cnn. There are some other workflows. At first, the quantized ONNX model can be directly trained in the CNN framework. This requires that the framework supports the quantization of pocket-cnn. The rest of the workflow is the same as described above. Another branch in the workflow is to create ONNX models manually. This doesn't have any relevance for real models, but is really useful for testing. I. e. small ONNX models can be created fastly without any training. This allows to test (i. e. simulate) multiple configurations extensively. The test models are defined in the model zoo.
| Layer | Properties | Limitations |
|---|---|---|
| Convolution |
|
Quantization of the activations and weights: Scale has to be power-of-two, zero point has to be zero. |
| Maximum Pooling |
|
- |
| Global Average Pooling | - | The averaging factor is quantized to the 16 bit fixed point value of 1 / height * width. |
| Zero Padding | - | The padding has to be the same at each edge. |
| (Leaky) ReLU | - | Leaky ReLU has a fixed alpha of 0.125 |
Before using the framework, you should be aware of several limitations:
- It is not complete. There might be several bugs and things missing. Please open an issue.
- There will be a different accuracy and loss due to the 8 bit quantization.
- Only small CNN can be synthesized, because the weights get mapped to LUT.
- Only a subset of layers is supported.
Most of the toplevel generics are describing the CNN architecture. They get derived from the ONNX model and don't need to be specified manually by the user. A table, containing the most important toplevel generics and signals, can be found here. The communication protocol is similar in all submodules of this design.
pocket-cnn accepts ONNX models as input. Each convolution layer of the CNN gets converted to a processing element (PE). This is the central element of the hardware design. The convolution in a PE can be preceeded by zero padding and followed by ReLU and/or maximum pooling. See also pe.vhd. Below are a few common configurations.
The python framework takes care of converting the ONNX model into the VHDL representation. The VHDL toplevel can be found at top.vhd. The toplevel structure is illustrated in the following image.
More details about the specific modules can be found here.
Can be found at the documentation folder and in the issues.