Karrot is Flask Webapp that acts as a lag reporting tool able to parse Burrow (a monitoring companion for Apache Kafka) http notifications. It offers AWS CloudWatch lag reporting as well as a Prometheus metrics exporter.
If you want to read more about Karrot please go check this Medium article
If cloudwatch reporter is used (which is enabled by default), you must run Karrot with credentials that have the following IAM permissions:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"cloudwatch:PutMetricData"
],
"Resource": "*"
}
]
}- Docker
docker run -it --rm \
--name karrot \
-v ~/.aws:/root/.aws \
lowess/karrot- Kubernetes
# Add the Repository to Helm:
$ helm repo add lowess-helm https://lowess.github.io/helm-charts
# Install karrrot helm chart:
$ helm install lowess-helm/karrot- Local Development with built-in Flask server (:warning: Do not use in production!)
export FLASK_APP=karrot
export FLASK_ENV=development
flask run- Run in production with
gunicorn(:white_check_mark: Valid for production usage):
export FLASK_APP=karrot.wsgi
export FLASK_ENV=production
flask "karrot:create_app()" --bind 127.0.0.1:5000 -w 4| env | value | description |
|---|---|---|
KARROT_LOG |
INFO, DEBUG, ERROR |
The log level to use for the Karrot app |
KARROT_REPORTERS |
prometheus,cloudwatch |
A CSV list of reporters to use in Karrot |
- Cloudwatch
| env | value | description |
|---|---|---|
KARROT_CLOUDWATCH_NAMESPACE |
GumGum/Kafka/Burrow/ConsumerLag |
The Cloudwatch namespace prefix to use for lag reporting |
KARROT_CLOUDWATCH_INTERVAL |
30 |
The Cloudwatch flush interval to execute the put_metric_data api call |
# Install development requirements
make dev
# Setup pre-commit hook locally
pre-commit install
# Run full tests
make tests
# Build Sphinx documentation locally
make docs- VSCode
settings.json
{
"python.linting.pylintEnabled": false,
"python.linting.flake8Enabled": true,
"python.linting.enabled": true,
"python.formatting.provider": "black",
"python.pythonPath": "~/.pyenv/versions/konnector/bin/python",
"python.venvPath": "~/.pyenv/shims",
"editor.formatOnPaste": false,
"editor.formatOnSave": true
}The project uses Flake8 linter and Black autoformatter
Tests are automated with Tox and run with Pytest suite. Codecoverage is also reported during test runs.
The documentation can be genered with Sphinx
# Start the stack locally using docker-compose
cd docker/
docker-compose up
# Send messages to a test topic
docker exec -it \
docker_kafka_1 \
kafka-producer-perf-test.sh \
--producer-props bootstrap.servers=localhost:9092 \
--throughput 10 \
--num-record 100000000 \
--record-size 100 \
--topic topicA
# Run a consumer polling from the test topic
docker exec -it \
docker_kafka_1 \
kafka-console-consumer.sh \
--bootstrap-server kafka:9092 \
--topic topicA \
--from-beginning \
--group topicA-consumerMade with ♥ by GumGum engineers