szczypiorek allows one:
- to define ones settings / credentials as a part of incredibly readable and flexible yaml format
- using as an extra template variables and imports
- and then define with minimal amount of code environment parsers which can be used to parse such yaml files
- and allow one to use them directly in their python code
- all of that on top of the mechanism allowing the secured storage and usage of most likely containing sensitive information yaml files.
In order to start one needs to perform below steps.
It should contain some sort of settings, let's create one under deploy/development.yml
folder:
aws:
secret: this is my secret
is_sensitive: true
age_days: 121
Somewhere in your code create a parser (e.g. env.py
):
import szczypiorek as ev
class MyEnvParser(ev.EnvParser):
aws_secret = ev.CharField()
aws_is_sensitive = ev.BooleanField()
aws_age_days = ev.IntegerField()
env = MyEnvParser().parse()
One must make sure that the yaml file will be protected therefore run in the terminal:
szczypiorek encrypt ./deploy
Make sure to add to .gitignore
.szczypiorek_encryption_key
deploy/development.yml
In certain situations, like when one created dynamical resources during the deployment phase, one would like to be able to replace certain parts of existing .szczyp
file with some dynamical values.
In order to achieve that one would have to use the replace
command:
szczypiorek replace -r a.b:12 -r a.g.f.secret:password
By default it would use the default encryption key and default szczyp file but one can overwrite it by passing extra arguments. Please run:
szczypiorek replace --help
To learn more.
- Encryption key is searched in various places in this order:
SZCZYPIOREK_ENCRYPTION_KEY
environment variable where it's expected that the content of the encryption key will be stored.SZCZYPIOREK_ENCRYPTION_KEY_FILE
environment variable where the path to chosen encryption key file is stored- if above is not found szczypiorek tries default
.szczypiorek_encryption_key
file.
- Environment gpg is searched in various places in this order:
SZCZYPIOREK_ENVIRONMENT_GPG
environment variable where it's expected that the content of the environment gpg will be stored.SZCZYPIOREK_ENVIRONMENT_GPG_FILE
environment variable where the path to chosen environment gpg file is stored- if above is not found szczypiorek tries default
env.gpg
file.
When deployed for the parser to work correctly one needs two things:
- A single
env.gpg
file available in the root of the project - A
.szczypiorek_encryption_key
storing the secret generated during theencrypt
phase. WARNING! Make sure that this file is not kept in the repository itself but rather created dynamically during the deployment phase. Till that time it could be stored in for examplegithub secrets
or similar system such asAWS secrets
,Azure Key Vault
etc.
For the szczypiorek
to work locally one needs 3 things:
- One
gpg
file for exampledevelopment.gpg
- A environment variable
SZCZYPIOREK_ENVIRONMENT_GPG_FILE=development.gpg
pointing to that file's location - A
.szczypiorek_encryption_key
file where a secret for decryption is stored.
TODO: add it
One contributor is changing the yaml files but the other cannot see those reflected in the gpg file?
TODO: add it
FIXME: add it!!!
szczypiorek print-env cosphere_api.conf.env.api_env
TODO: add it
USE BELOW LEGACY AS INSPIRATION:
Use the below example as a inspiration regarding type of fields one can define.
import szczypiorek as env
class MyEnvParser(env.EnvParser):
secret_key = env.CharField()
is_important = env.BooleanField()
aws_url = env.URLField()
number_of_workers = env.IntegerField()
unit_price = env.FloatField()
Each field supports the following arguments:
required
(boolean) - if environment variable is required to be presentdefault
(target type) - default to be used in case environment variable was not foundallow_null
(boolean) - if environment variable can be nullabledescription
(str) - where one can describe the purpose of a given field when the name itself is not enough to capture it.
Besides those some fields are supporting extra fields:
-
CharField
:min_length
- validates if minimum amount of characters was providedmax_length
- validates if maximum amount of characters was provided
TODO: add it - decribe the usage of template variables
There's still a lot of work to do, even though the basic functionality of szczypiorek is production ready:
-
[HIGH PRIO] add integration with various keyvaults
-
[HIGH PRIO] add integration precommit hooks for checking if you're not commiting the secrets!
-
[HIGH PRIO] enable versioning of secrets and a command which is able to clean up all unused secrets
-
[HIGH PRIO] use https://github.com/squidfunk/mkdocs-material for docs
-
[HIGH PRIO] host the docs on github pages https://www.mkdocs.org/user-guide/deploying-your-docs/
-
[HIGH PRIO] replace the parsing mechanism with Pydantic models to make it more modern and more like python static typing.
-
[HIGH PRIO] add an option to sync the secret of the admin
-
[LOW PRIO] Enable imports in the yml files. It should take the following form:
imports:
- shared.yml
- development.yml
And then when running encrypt
one stores at the level of gpg files all its dependencies therefore making it self contained. Before creating the gpg file it could create structure like:
- filename: shared.yml
content: ...
- filename: development.yml
content: ...
# MAIN FILE
- filename: integration.yml
content: ...
When running decrypt
it "unpacks" all files with warnings that existing ones would be overwritten. There will be an option to force it without messages. When decrypting
in the parser the imports are taken into account and therefore applied right away so the resulting env
is just a result of applying all dependencies.
- [LOW PRIO] support DB_POSTGRES_CONNECT validator:
- it expects certain fields to be present
- it allows fields mapping
- [LOW PRIO] support SENTRY_PING validator:
- it uses https://develop.sentry.dev/sdk/store/
- it just pings the API
- [LOW PRIO] support OAUTH2_PING validator:
- it just creates a OAUTH2 session and tries not to fail while doing this
- [LOW PRIO] support API_PING validator:
- it just pings the API
- it allows fields mapping
- [LOW PRIO] support AWS_S3_CLOUDFRONT_PING validator:
- it just pings the endpoint
- [LOW PRIO] support AWS_S3_BUCKET_LIST validator:
- it just lists the bucket
- [LOW PRIO] support AWS_POLLY_RENDER_SAMPLE validator:
- it renders simple text
- [LOW PRIO] handle ROTATION of the szczypiorek secret itself. One can with one command run:
decrypt
- render new key
encrypt
- [LOW PRIO] support PRODUCTION gpg file where all validators etc are removed only the essential parts are left