Grab-bag of utilities and other stuff that I use with AWS. Easier to keep track of as a project rather than lots of gists.
Each is documented in its header, and exposes functions that may be useful for other programs.
To run you must have boto3 installed.
| Program | Description |
|---|---|
| assume-role.py | Spawns a subshell with authentication credentials for a specified role. |
| cf-env.py | Populates environment variables from the parameters and outputs of a CloudFormation stack. |
| cf-runner.py | Creates/updates CloudFormation scripts, using an external store of common parameters |
| kinesis_reader.py | Reads from a Kinesis stream, writing output as JSON. |
| logs_reader.py | Reads from a CloudWatch Logs log group/stream, writing output as JSON. |
| sm-env.py | Populates environment variables from a Secrets Manager secret. |
Isolated pieces of code or configuration, intended to be pasted elsewhere.
Python modules that can be incorporated into larger programs. Each has a README that shows example usage.
| Module | Description |
|---|---|
| aws_util.create_client() | Creates a service client, optionally assuming a role to do so. |
| kinesis.KinesisWriter | Writes messages to a Kinesis stream, properly handling retries. |
Complete Lambda implementations and code intended to be used with Lambdas. Mostly in Python.
| Directory | Contents |
|---|---|
| cloudwatch-log-cleanup | Deletes CloudWatch log streams that are empty because of the log group's retention period. |
| cloudwatch-log-transform | Transforms CloudWatch Logs events from a Kinesis stream. |
| elb-to-es | Imports Elastic Load Balancer logfiles into Elasticsearch. |
| es-index-cleanup | Deletes up old indexes from an Elasticsearch cluster. See this blog post for more info. |
| json-logging | A module that will configure the Python logging framework for JSON output with Lambda-specific metadata. |
| Directory | Contents |
|---|---|
| modules/lambda | Module for creating Lambda functions. |
| modules/sqs | Module for creating SQS queues. |
| modules/s3_deployment | Module for maintaining Lambda deployment bundles on S3. |
| examples/provision-via-bastion | Example of provisioning an instance via a bastion host. |
| templates | Templates for common config files. |