Build and CI platform for adsabs.

• Builds docker images based on templates. Triggered by github webhooks
• Pushes successfully build images to dockerhub with the commit hash as tag

You can run unit tests in the following way:

nosetests mc/tests/unittests
nosetests mc/tests/livetests (this will create/run real containers)

A Vagrantfile and puppet manifest are available for development within a virtual machine. To use the vagrant VM defined here you will need to install Vagrant and VirtualBox.

• Vagrant
• VirtualBox

To load and enter the VM: vagrant up && vagrant ssh

To add a service to mission-control for a specific feature, you will need to follow the instructions below:

• Automatic builds to Docker registry on GitHub commit:

  1. Add the service to the "WATCHED_REPOS" list in mc/config.py
  2. Update and restart the mission_control web service

• CI Testing

  1. You will need to include stub data for the relevant dependencies that the tests will use:
  2. Solr
  3. Postgres
  4. Consul
  5. Update the config/adsws/staging/WEBSERVICES in the Consul key-value store mc/templates/consul/adsws/adsws.config.json for service discovery.
  6. Check it is in the KNOWN_SERVICES in the Postgres provisioner (this should be removed soon).

1. Check the YAML file has the correct configuration for what you want to test against:

   dependencies:
     - name: redis
       image: redis:2.8.21
   services:
     - name: graphics_service
       repository: adsabs
       tag: dd905b927323e1ecf2a563a80d2bc5d9d98b62b4

   This example will start redis using the 2.8.21 image, and the graphics service for the relevant commit hash. The following keywords are available:

   • dependencies: this is a stand alone service that our infrastructure relies on, possible values: redis, postgres, solr, consul, registrator
     • name: name of the service
     • image: docker image to use (docker convention is used)
     • requirements: which containers does the service need to know exist
     • build_requirements: which containers does the service need to know exist ON BUILD
   • services: this is our application tier, and so any service that is part of our microservice layer, possible values: graphics_service, metrics_service, recommender_service, solr-service, export_service, myads, orcid-service, citation_helper_service, vis-services, biblib-service, adsws
     • name: name of the service
     • repository: Docker registry repository it belongs to
     • tag: commit hash/tag of the container
   • tests: this refers to what functional tests to run on the test cluster, possible values: adsrex

2. Run the tests bash python mc/manage.py test_cluster -c run

3. Check the results bash cat .mc.ci

Note: The YAML list is order dependent, much like in the TravisCI .travis.yml file.

1. The image should have been built and pushed to dockerhub. If the docker image is for whatever reason inaccessible, the deployment (last step) will fail. One can manually build commits via

   python mc/manage.py dockerbuild --repo $REPO --commit $HASH

2. Print the JSON task definition:

   json=$(python mc/manage.py print_task_def --containers metrics_service:a07fc3bf3e1307fff7ac31f7aba85a8576395128 staging 300 --family metrics_service)

   (300: Megabytes to allocate to the container. This is the criterion ECS uses to allocate services amongst the cluster, so it should be well-understood)

3. Register the JSON task definition the AWS-ECS:

   python mc/manage.py register_task_def --task "$json"

4. Update the running ECS service with the newly updated task definition:

   python mc/manage.py update_service --cluster staging --service metrics_service --desiredCount 1 --taskDefinition metrics_service

• Consul back-up