Lighter solves the problem of automating deployments to Marathon and handling of differences between multiple environments. Given a hierachy of yaml files and environments, Lighter can expand service config files and deploy them to Marathon.

For even tighter integration into the development process, Lighter can resolve Marathon config files from Maven and merge these with environment specific overrides. This enables continuous deployment whenever new releases or snapshots appear in the Maven repository. Optional version range constraints allows patches/minor versions to be rolled out continuously, while requiring a config change to roll out major versions.

usage: lighter COMMAND [OPTIONS]...

Marathon deployment tool

positional arguments:
  {deploy,verify}       Available commands
    deploy              Deploy services to Marathon
    verify              Verify and generate Marathon configuration files

optional arguments:
  -h, --help            show this help message and exit
  -n, --noop            Execute dry-run without modifying Marathon [default:
                        False]
  -v, --verbose         Increase logging verbosity [default: False]
  -t TARGETDIR, --targetdir TARGETDIR
                        Directory to output rendered config files
  -p PROFILES, --profile PROFILES
                        Extra profile file(s) to be merged with service
                        definitions.

usage: lighter deploy [OPTIONS]... YMLFILE...

Deploy services to Marathon

positional arguments:
  YMLFILE               Service files to expand and deploy

optional arguments:
  -h, --help            show this help message and exit
  -m MARATHON, --marathon MARATHON
                        Marathon URL like "http://marathon-host:8080/".
                        Overrides default Marathon URL's provided in config
                        files
  -f, --force           Force deployment even if the service is already
                        affected by a running deployment [default: False]
  --canary-group CANARYGROUP
                        Unique name for this group of canaries [default: None]
  --canary-cleanup      Destroy canaries that are no longer present [default:
                        False]

usage: lighter verify YMLFILE...

Verify and generate Marathon configuration files

positional arguments:
  YMLFILE           Service files to expand and deploy

optional arguments:
  -h, --help        show this help message and exit
  --verify-secrets  Fail verification if unencrypted secrets are found
                    [default: False]

Given a directory structure like

config-repo/
|   globals.yml
|   myprofile.yml
└─ production/
|   |   globals.yml
|   |   myfrontend.yml
|   └─ mysubsystem/
|          globals.yml
|          myservice-api.yml
|          myservice-database.yml
└─ staging/
    |   globals.yml
    |   myfrontend.yml

Running lighter deploy -p myprofile1.yml -p myprofile2.yml staging/myfrontend.yml will

• Merge myfrontend.yml with environment defaults from config-repo/staging/globals.yml, config-repo/globals.yml, myprofile1.yml and myprofile2.yml
• Fetch the json template for this service and version from the Maven repository
• Expand the json template with variables and overrides from the yml files
• Post the resulting json configuration into Marathon

Yaml files may contain a marathon: section with a default URL to reach Marathon at. The -m/--marathon parameter will override this setting when given on the command-line.

globals.yml

marathon:
  url: 'http://marathon-host:8080/'

The maven: section specifies where to fetch json templates from which are merged into the configuration. For example

globals.yml

maven:
  repository: "http://username:password@maven.example.com/nexus/content/groups/public"

myservice.yml

maven:
  groupid: 'com.example'
  artifactid: 'myservice'
  version: '1.0.0'
  classifier: 'marathon'

The Maven 'classifier' tag is optional.

Versions can be dynamically resolved from Maven using a range syntax.

maven:
  groupid: 'com.example'
  artifactid: 'myservice'
  version: '[1.0.0,2.0.0)'

For example

Yaml files may contain a service: tag which specifies a Marathon json fragment to use as the service configuration base for further merging. This allows for services which aren't based on a json template but rather defined exclusively in yaml.

myservice.yml

service:
  id: '/myproduct/myservice'
  container:
    docker:
      image: 'meltwater/myservice:latest'
  env:
    DATABASE: 'database:3306'
  cpus: 1.0
  mem: 1200
  instances: 1

Yaml files may contain an override: section that will be merged directly into the Marathon json. The structure contained in the override: section must correspond to the Marathon REST API. For example

override:
  instances: 4
  cpus: 2.0
  env:
    LOGLEVEL: 'info'
    NEW_RELIC_APP_NAME: 'MyService Staging'
    NEW_RELIC_LICENSE_KEY: '123abc'

An YAML and JSON file upwards-recursive deep merge is performed when parsing service definitions. Precedence is defined by the directory structure

• myservice.yml has the highest precedence
• globals.yml files are merged with decreasing precedency upwards in the directory structure
• myservice-1.0.0-marathon.json if fetched from Maven has the lowest precedence

Lists, dicts and scalar values are deep merged

• Dicts are deep merged, the result containing the union of all keys
• Lists are appended together
• Scalar values coalesce to the not-null value with highest precedence

The default behaviour is to append lists together, however specific list items can be overriden and deep merged using a dict with integer keys. For example

myservice-1.0.0-marathon.json

{
    "container": {
        "docker": {
            "portMappings": [
                {"containerPort": 8080, "servicePort": 1234},
                {"containerPort": 8081, "servicePort": 1235}
            ]
        }
    }
}

myservice-override-serviceport.yml

override:
  container:
    docker:
      portMappings:
        # Override service ports 1234,1235 with port 4000,4001
        0:
          servicePort: 4000
        1:
          servicePort: 4001

Booleans, integers and floats in the env section are converted to strings before being posted to Marathon. Non-scalar environment variables like dicts and lists are deep merged and automatically serialized to json strings.

myservice.yml

override:
  env:
    intvar: 123
    boolvar: TRUE
    dictvar:
      mykey:
       - 1
       - 'abc'

Would result in a rendered json like

{
  "env": {
    "intvar": "123",
    "boolvar": "true",
    "dictvar": "{\"mykey\": [1, \"abc\"]}"
  }
}

Yaml files may contain an variables: section containing key/value pairs that will be substituted into the json template. All variables in a templates must be resolved or it's considered an error. This can be used to ensure that some parameters are guaranteed to be provided to a service. For example

variables:
  docker.registry: 'docker.example.com'
  rabbitmq.host: 'rabbitmq-hostname'

And used from the json template like

{
    "id": "/myproduct/myservice",
    "container": {
        "docker": {
            "image": "%{docker.registry}/myservice:1.2.3"
        }
    },
    "env": {
        "rabbitmq.url": "amqp://guest:guest@%{rabbitmq.host}:5672"
    }
}

Lighter also allows specifying environment variables as values in the configuration yaml files.

With the following configuration: myservice.yml

service:
  id: '/myproduct/myservice'
  container:
    docker:
      image: 'meltwater/myservice:%{env.VERSION}'
  env:
    DATABASE: 'database:3306'
  cpus: 1.0
  mem: 1200
  instances: 1

And Running VERSION=1.1.1 lighter deploy myservice.yml, lighter will deploy the docker image meltwater/myservice:1.1.1 to marathon.

To avoid interpolating some string like %{id} when you really want it, use %%{id}

If an image is rebuilt with the same Docker tag, Marathon won't detect a change and hence won't roll out the new image. To ensure that new snapshot/latest versions are deployed use %{lighter.uniqueVersion} and forcePullImage like this

myservice.yml

override:
  container:
    docker:
      forcePullImage: true
  env:
    SERVICE_BUILD: '%{lighter.uniqueVersion}'

Lighter calls the Docker Registry API to resolve %{lighter.uniqueVersion} when it's used in a non-Maven based service. This is only enabled if the %{lighter.uniqueVersion} variable is actually referenced from the service config.

For authenticated reprositories you must supply read-access credentials to be used when calling the registry API. You can find the base64 encoded credentials in your ~/.docker/config.json or ~/.dockercfg files. Note that Docker Hub is not supported at this time.

globals.yml

docker:
  registries:
    'registry.example.com':
      auth: 'dXNlcm5hbWU6cGFzc3dvcmQ='

Yaml files may contain a facts: section with information about the service surroundings

staging/globals.yml

facts:
  environment: 'staging'

Lighter has support for Secretary which can securely distribute secrets to containers.

*someenv/globals.yml *

secretary:
  url: 'https://secretary-daemon-loadbalancer:5070'
  master:
    publickey: 'someenv/keys/master-public-key.pem'

someenv/myservice.yml

override:
  env:
    DATABASE_PASSWORD: "ENC[NACL,NVnSkhxA010D2yOWKRFog0jpUvHQzmkmKKHmqAbHAnz8oGbPEFkDfyKHQHGO7w==]"

Note

Make sure the environment variable name is a valid shell script identifier and supported by Secretary, only alphanumeric characters and underscores are supported, starting with an alphabetic or underscore character. e.g DATABASE.PASSWORD is invalid but DATABASE_PASSWORD is valid.

Lighter together with Proxymatic supports canary deployments using the --canary-group parameter. This parameter makes Lighter rewrite the app id and servicePort to avoid conflicts and automatically add the metadata labels that Proxymatic use for canaries. The --canary-cleanup parameter destroys canary instances when they are removed from configuration.

Take care that the --canary-group parameter is unique to the deployment job and branch that executes the canary deployment. Lighter will clean out canaries with the same group name if they aren't being generated anymore, and if multiple deployment jobs share a group name they'd conflict and destroy each others canaries.

This example use a *-canary-* filename convention to separate canaries from normal services. In this workflow you would copy the regular service file myservice.yml, and make any tentative changes in this new myservice-canary-somechange.yml. When the canary has served its purpose you'd git mv back or git rm the canary file.

# Deploy regular services
lighter deploy -f -m "http://marathon-host:8080/" $(find . -name \*.yml -not -name globals.yml -not -name \*-canary-\*)

# Deploy and prune canaries
lighter deploy -f -m "http://marathon-host:8080/" --canary-group=generic --canary-cleanup $(find . -name \*-canary-\*.yml)

This usage would run lighter -t /some/output/dir verify ... on a PR and again on its base revision. Then diff -r the rendered json files to figure out what services were modifed in the PR. The modified services would be deployed as canaries with lighter deploy --canary-group=mybranchname --canary-cleanup ... whenever the PR branch is changed. When the PR is closed or merged the canaries would be destroyed using lighter deploy --canary-group=mybranchname --canary-cleanup

Lighter adds a Docker label com.meltwater.lighter.canary.group which can be used to separate out container metrics from the canaries.

Place a lighter script in the root of your configuration repo. Replace the LIGHTER_VERSION with a version from the releases page.

#!/bin/bash
set -e

LIGHTER_VERSION="x.y.z"

BASEDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
LIGHTER="$BASEDIR/target/lighter-`uname -s`-`uname -m`-${LIGHTER_VERSION}"

if [ ! -x "$LIGHTER" ]; then
    mkdir -p $(dirname "$LIGHTER")
    curl -sfLo "$LIGHTER" https://github.com/meltwater/lighter/releases/download/${LIGHTER_VERSION}/lighter-`uname -s`-`uname -m`
    chmod +x "$LIGHTER"
fi

# Ligher will write the expanded json files to /tmp/output
exec "$LIGHTER" -t "`dirname $0`/target" $@

Use the script like

cd my-config-repo

# Deploy/sync all services (from Jenkins or other CI/CD server)
./lighter deploy $(find staging -name \*.yml -not -name globals.yml)

# Deploy single services
./lighter deploy staging/myservice.yml staging/myservice2.yml

Lighter can push deployment notifications to a number of services.

Yaml files may contain an hipchat: section that specifies where to announce deployments. Create a HipChat V2 token that is allowed to post to rooms. The numeric room ID can be found in the room preferences in the HipChat web interface.

hipchat:
  token: '123abc'
  rooms:
    - '123456'

The default message of the hipchat notification is :

Deployed /my-image-id with image example-registry.io/image:1.0.0 to staging (marathon-url.example.com).

If you would like to post releases notes in addition to the above message, you have 2 options :

1. You can add the following block to your service config

hipchat:
  releaseNotes: "my amazing release notes"

2. You can add the release notes with a label on the docker image itself and then indicate which label lighter should use to get the release notes :

dockerfile

FROM registry.io/image:tag
LABEL com.example.component.release-notes="my amazing release notes"

service config

hipchat:
  message.image.label: "com.example.component.release-notes" # image label to use for hipchat message

To send New Relic deployment notifications supply your New Relic REST API key (different from the license key given to the agent).

globals.yml

newrelic:
  token: '123abc'

myservice.yml

override:
  env:
    NEW_RELIC_LICENSE_KEY: 'abc123'
    NEW_RELIC_APP_NAME: 'MyService'

To send Datadog deployment events supply your Datadog API key. Lighter will add Marathon appid and canary group as Docker container labels in order for Datadog to tag collected metrics, see: collect_labels_as_tags.

globals.yml

datadog:
  token: '123abc'
  tags:
    - subsystem:example

Datadog Puppet Config

datadog::docker:
  docker_daemon:
    instances:
      - url: "unix://var/run/docker.sock"
        new_tag_names: true
        collect_labels_as_tags: ["com.meltwater.lighter.appid", "com.meltwater.lighter.canary.group"]

To send Graphite deployment events supply your Graphite plaintext and HTTP endpoints.

globals.yml

graphite:
  address: 'graphite-host:2003'
  url: 'http://graphite-host:80/'
  prefix: 'lighter'
  tags:
    - subsystem:example

• Giuliano Manno

  • Lighter logo