Nicecream FM History

If you are a nicecream.fm addict, I got you covered. You can now have all those songs playing each and every minute, even when you are not online 😁.

The web app is made of an API and a SPA.

Live: SPA - API

Features:

Channels history
Social sign in with Google
Bookmarks
...

References:

Deployment

Local (Docker)

Requirements:

Docker
Docker Compose
GPC account
Google OAuth2 client with http://api.lvh.me:8080/user/google/callback added to Authorized redirect URIs

Build the api image:
docker-compose build api

Generate a Fernet secret key that will be used to encrypt the session cookie:
docker run --rm nicecream_history_api python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"

Copy .env.sample to .env and fill in API_SESSION_COOKIE_SECRET_KEY, API_GOOGLE_CLIENT_ID, API_GOOGLE_CLIENT_SECRET:
cp .env.sample .env

Run and open http://api.lvh.me:8080 in browser:
docker-compose up -d

References:

ngrok, lvh.me and nip.io: A Trilogy for Local Development and Testing

AWS

Describe and provision infrastructure with CloudFormation.

Requirements:

Internet domain (i.e example.com) and access to DNS management 😎
Docker
Docker Compose
GPC account
Google OAuth2 client with https://api.[example.com]/user/google/callback added to Authorized redirect URIs
AWS account
AWS CLI
AWS certificate for *.example.com generated into the region where deployment will be done.

References:

cloudonaut.io templates

CloudFormation

The bucket will hold all CloudFormation templates.

aws cloudformation create-stack \
--stack-name [CLOUDFORMATION_STACK_NAME] \
--template-body file://cloudformation/templates/s3.yaml \
--parameters \
    ParameterKey=Access,ParameterValue=PublicRead \
    ParameterKey=EnvironmentName,ParameterValue=[ENVIRONMENT_NAME] \
--capabilities CAPABILITY_IAM

Upload:
aws s3 sync ./cloudformation/templates s3://[CLOUDFORMATION_BUCKET]

Alert

Receive alerts from resources. Supported transports are: Slack, Email, HTTP endpoint, HTTPS endpoint

aws cloudformation create-stack \
--stack-name [ALERT_STACK_NAME] \
--template-url https://s3.amazonaws.com/[CLOUDFORMATION_BUCKET]/alert.yaml \
--parameters \
    ParameterKey=FallbackEmail,ParameterValue=[FALLBACK_EMAIL] \
    ParameterKey=Email,ParameterValue=[EMAIL] \
--capabilities CAPABILITY_IAM

./slack_alert_lambda.sh [SLACK_LAMBDA_FUNCTION_NAME]

VPC

VPC with 2 public & 2 private subnets in two availability zones (Zone A & B).

aws cloudformation create-stack \
--stack-name [VPC_STACK_NAME] \
--template-url https://s3.amazonaws.com/[CLOUDFORMATION_BUCKET]/vpc-2azs.yaml \
--parameters \
    ParameterKey=EnvironmentName,ParameterValue=[ENVIRONMENT_NAME]

Client Security Group

Security group used for gaining access to resources which are not integrated into the VPC (i.e ElastiCache).

aws cloudformation create-stack \
--stack-name [CLIENT_SG_STACK_NAME] \
--template-url https://s3.amazonaws.com/[CLOUDFORMATION_BUCKET]/client-sg.yaml \
--parameters \
    ParameterKey=ParentVPCStack,ParameterValue=[VPC_STACK_NAME] \
    ParameterKey=EnvironmentName,ParameterValue=[ENVIRONMENT_NAME]

VPC EC2 NAT/SSH Bastion

EC2 instance used for NAT and as a SSH Bastion (optional).
Best practices recommends to have one independent stack for each availability zone.

NAT (per availability zone):

Must deploy for each zone A & B.

aws cloudformation create-stack \
--stack-name [VPC_NAT_STACK_NAME] \
--template-url https://s3.amazonaws.com/[CLOUDFORMATION_BUCKET]/vpc-nat-instance.yaml \
--parameters \
    ParameterKey=ParentVPCStack,ParameterValue=[VPC_STACK_NAME] \
    ParameterKey=ParentAlertStack,ParameterValue=[ALERT_STACK_NAME] \
    ParameterKey=SubnetZone,ParameterValue=[SUBNET_ZONE] \
    ParameterKey=EnvironmentName,ParameterValue=[ENVIRONMENT_NAME] \
--capabilities CAPABILITY_IAM

NAT (per VPC):

aws cloudformation create-stack \
--stack-name [VPC_NAT_STACK_NAME] \
--template-url https://s3.amazonaws.com/[CLOUDFORMATION_BUCKET]/vpc-nat-instance.yaml \
--parameters \
    ParameterKey=ParentVPCStack,ParameterValue=[VPC_STACK_NAME] \
    ParameterKey=ParentAlertStack,ParameterValue=[ALERT_STACK_NAME] \
    ParameterKey=EnvironmentName,ParameterValue=[ENVIRONMENT_NAME] \
--capabilities CAPABILITY_IAM

NAT and SSH (per VPC):

aws cloudformation create-stack \
--stack-name [VPC_NAT_STACK_NAME] \
--template-url https://s3.amazonaws.com/[CLOUDFORMATION_BUCKET]/vpc-nat-instance.yaml \
--parameters \
    ParameterKey=ParentVPCStack,ParameterValue=[VPC_STACK_NAME] \
    ParameterKey=ParentAlertStack,ParameterValue=[ALERT_STACK_NAME] \
    ParameterKey=ParentClientStack,ParameterValue=[CLIENT_SG_STACK_NAME] \
    ParameterKey=KeyName,ParameterValue=[EC2_KEY_NAME] \
    ParameterKey=SSHAccessCidrIp,ParameterValue='[IPv4_SSH_ALLOWED]/32' \
    ParameterKey=EnvironmentName,ParameterValue=[ENVIRONMENT_NAME] \
--capabilities CAPABILITY_IAM

RDS PostgresSQL

Multi-AZ/Replica disabled by default.

aws cloudformation create-stack \
--stack-name [RDS_STACK_NAME] \
--template-url https://s3.amazonaws.com/[CLOUDFORMATION_BUCKET]/rds-postgres.yaml \
--parameters \
    ParameterKey=ParentVPCStack,ParameterValue=[VPC_STACK_NAME] \
    ParameterKey=ParentClientStack,ParameterValue=[CLIENT_SG_STACK_NAME] \
    ParameterKey=ParentAlertStack,ParameterValue=[ALERT_STACK_NAME] \
    ParameterKey=EnvironmentName,ParameterValue=[ENVIRONMENT_NAME] \
    ParameterKey=DBMasterUsername,ParameterValue=[DATABASE_USERNAME] \
    ParameterKey=DBMasterUserPassword,ParameterValue=[DATABASE_PASSWORD] \
    ParameterKey=DBName,ParameterValue=[DATABASE_NAME]

ElastiCache Redis

Multi-AZ/Replica disabled by default.

aws cloudformation create-stack \
--stack-name [ELASTICACHE_STACK_NAME] \
--template-url https://s3.amazonaws.com/[CLOUDFORMATION_BUCKET]/elasticache-redis.yaml \
--parameters \
    ParameterKey=ParentVPCStack,ParameterValue=[VPC_STACK_NAME] \
    ParameterKey=ParentClientStack,ParameterValue=[CLIENT_SG_STACK_NAME] \
    ParameterKey=ParentAlertStack,ParameterValue=[ALERT_STACK_NAME] \
    ParameterKey=EnvironmentName,ParameterValue=[ENVIRONMENT_NAME]

EC2 ALB access logs

aws cloudformation create-stack \
--stack-name [S3_ACCESS_LOG_STACK_NAME] \
--template-url https://s3.amazonaws.com/[CLOUDFORMATION_BUCKET]/s3.yaml \
--parameters \
    ParameterKey=Access,ParameterValue=ElbAccessLogWrite \
    ParameterKey=EnvironmentName,ParameterValue=[ENVIRONMENT_NAME] \
--capabilities CAPABILITY_IAM

ECS cluster

ECS Cluster with one public ALB.

Cluster autoscaling based on A better solution to ECS AutoScaling. Things to consider:

maximum memory (default=256MiB) and maximum CPU (default=256Units)
container shortage (default=0) and excess (default=4) thresholds
instance type (default=t2.micro)

aws cloudformation create-stack \
--stack-name [ECS_STACK_NAME] \
--template-url https://s3.amazonaws.com/[CLOUDFORMATION_BUCKET]/ecs-cluster.yaml \
--parameters \
    ParameterKey=ParentVPCStack,ParameterValue=[VPC_STACK_NAME] \
    ParameterKey=ParentAlertStack,ParameterValue=[ALERT_STACK_NAME] \
    ParameterKey=ParentClientStackOne,ParameterValue=[CLIENT_SG_STACK_NAME] \
    ParameterKey=ParentS3StackAccessLog,ParameterValue=[S3_ACCESS_LOG_STACK_NAME] \
    ParameterKey=SubnetsReach,ParameterValue=Private \
    ParameterKey=SSMParametersPrefix,ParameterValue=\"[AWS_SSM_PREFIX]\" \
    ParameterKey=LoadBalancerCertificateArn,ParameterValue=[SSL_CERTIFICATE_ARN] \
    ParameterKey=EnvironmentName,ParameterValue=[ENVIRONMENT_NAME] \
--capabilities CAPABILITY_IAM

Add a CNAME record for api.example.com, pointing to ECS cluster ALB:
api.example.com. 3600 IN CNAME XXX.YYY.elb.amazonaws.com.

ECS services

KMS parameters encryption key

Encryption key used on SSM parameters encryption.

aws cloudformation create-stack \
--stack-name [KMS_STACK_NAME] \
--template-url https://s3.amazonaws.com/[CLOUDFORMATION_BUCKET]/kms.yaml \
--parameters \
    ParameterKey=ParentVPCStack,ParameterValue=[VPC_STACK_NAME] \
    ParameterKey=ParentClusterStack,ParameterValue=[ECS_STACK_NAME] \
    ParameterKey=Administrators,ParameterValue=\"$(aws iam get-group --group-name [IAM_GROUP] --output text | awk 'BEGIN {FS="\t";ORS=","} {print $2}' | sed 's/,$//')\" \
    ParameterKey=EnvironmentName,ParameterValue=[ENVIRONMENT_NAME]

Parameters

Check .env.sample and api/settings.py for needed parameters.
Required parameters for production:

PGHOST (String)
PGPORT (String)
PGUSER (String)
PGPASSWORD (SecureString)
PGDATABASE (String)
REDIS_HOST (String)
REDIS_PORT (String)
SPA_URL (String)
API_SESSION_COOKIE_SECRET_KEY (SecureString)
API_SESSION_COOKIE_DOMAIN (String)
API_GOOGLE_CLIENT_ID (String)
API_GOOGLE_CLIENT_SECRET (SecureString)
API_GOOGLE_REDIRECT_URL (String)
API_CSRF_COOKIE_DOMAIN (String)
API_CORS_ALLOWED (String)
API_CORS_ORIGIN (String)

aws ssm put-parameter \
--name "/[AWS_SSM_PREFIX]/[PARAMETER_NAME]" \
--type "String" \
--value "[PARAMETER_VALUE]"

aws ssm put-parameter \
--name "/[AWS_SSM_PREFIX]/[PARAMETER_NAME]" \
--type "SecureString" \
--value "[PARAMETER_VALUE]" \
--key-id [KMS_KEY_ID]

ECR

Create an ECR repository and upload the API image before starting ECS services.

$(aws ecr get-login --no-include-email --region [AWS_REGION])

docker build \
--tag [REPOSITORY_NAME]:latest \
--tag [REPOSITORY_NAME]:$(echo $(git rev-parse --short HEAD)) .

docker push [REPOSITORY_NAME]:latest && \
docker push [REPOSITORY_NAME]:$(echo $(git rev-parse --short HEAD))

Database migrations

Migrations task definition. Uses the same API Docker image.

aws cloudformation create-stack \
--stack-name [MIGRATIONS_TASK_STACK_NAME] \
--template-url https://s3.amazonaws.com/[CLOUDFORMATION_BUCKET]/ecs-task.yaml \
--parameters \
    ParameterKey=ParentClusterStack,ParameterValue=[ECS_STACK_NAME] \
    ParameterKey=Image,ParameterValue=[REPOSITORY_NAME]:latest \
    ParameterKey=SSMParametersPrefix,ParameterValue=\"[AWS_SSM_PREFIX]\" \
    ParameterKey=Command,ParameterValue=\"alembic,upgrade,head\"

aws ecs run-task \
--cluster [ECS_CLUSTER_NAME] \
--task-definition [MIGRATIONS_TASK_STACK_NAME]

Crawler service

Uses the same API Docker image.
Default maximum memory = 256MiB and default maximum CPU = 256Units.
Review ECS Cluster autoscaling on maximum memory & CPU changes.

aws cloudformation create-stack \
--stack-name [CRAWLER_SERVICE_STACK_NAME] \
--template-url https://s3.amazonaws.com/[CLOUDFORMATION_BUCKET]/ecs-service-worker.yaml \
--parameters \
    ParameterKey=ParentClusterStack,ParameterValue=[ECS_STACK_NAME] \
    ParameterKey=ParentAlertStack,ParameterValue=[ALERT_STACK_NAME] \
    ParameterKey=DesiredCount,ParameterValue=1 \
    ParameterKey=MinCapacity,ParameterValue=1 \
    ParameterKey=MaxCapacity,ParameterValue=1 \
    ParameterKey=Image,ParameterValue=[REPOSITORY_NAME]:latest \
    ParameterKey=SSMParametersPrefix,ParameterValue=\"[AWS_SSM_PREFIX]\" \
    ParameterKey=Command,ParameterValue=\"python,-m,crawler\" \
--capabilities CAPABILITY_IAM

API service

Default maximum memory = 256MiB and default maximum CPU = 256Units.
Review ECS Cluster autoscaling on maximum memory & CPU changes.

aws cloudformation create-stack \
--stack-name [API_SERVICE_STACK_NAME] \
--template-url https://s3.amazonaws.com/[CLOUDFORMATION_BUCKET]/ecs-service.yaml \
--parameters \
    ParameterKey=ParentVPCStack,ParameterValue=[VPC_STACK_NAME] \
    ParameterKey=ParentClusterStack,ParameterValue=[ECS_STACK_NAME] \
    ParameterKey=ParentAlertStack,ParameterValue=[ALERT_STACK_NAME] \
    ParameterKey=Image,ParameterValue=[REPOSITORY_NAME]:latest \
    ParameterKey=SSMParametersPrefix,ParameterValue=\"[AWS_SSM_PREFIX]\" \
    ParameterKey=HealthCheckPath,ParameterValue=channels \
    ParameterKey=LoadBalancerHttps,ParameterValue=true \
--capabilities CAPABILITY_IAM

Update:

aws ecs update-service \
--cluster [ECS_CLUSTER_NAME] \
--force-new-deployment \
--service [ECS_SERVICE_NAME]

Name		Name	Last commit message	Last commit date
Latest commit History 31 Commits
alembic		alembic
api		api
cloudformation		cloudformation
crawler		crawler
tests/integration		tests/integration
.coveragerc		.coveragerc
.dockerignore		.dockerignore
.env.sample		.env.sample
.gitignore		.gitignore
.travis.yml		.travis.yml
Dockerfile		Dockerfile
Pipfile		Pipfile
Pipfile.lock		Pipfile.lock
README.md		README.md
docker-compose.yml		docker-compose.yml
docker-entrypoint.sh		docker-entrypoint.sh
pytest.ini		pytest.ini
wait.sh		wait.sh

altcatalin/nicecream-history

Folders and files

Latest commit

History

Repository files navigation

Nicecream FM History

Deployment

Local (Docker)

AWS

CloudFormation

Alert

VPC

Client Security Group

VPC EC2 NAT/SSH Bastion

RDS PostgresSQL

ElastiCache Redis

EC2 ALB access logs

ECS cluster

ECS services

KMS parameters encryption key

Parameters

ECR

Database migrations

Crawler service

API service

About

Topics

Resources

Stars

Watchers

Forks

Languages