1. Design 2. Launch 3. Analyze

Experimenter is a platform for managing experiments in Mozilla Firefox.

https://experimenter-app.dev.mozaws.net/

https://experimenter.stage.mozaws.net/

https://experimenter.services.mozilla.com/

An experiment is a way to measure how a change to your product affects how people use it.

An experiment has three parts:

1. A new feature that can be selectively enabled
2. A group of users to test the new feature
3. Telemetry to measure how people interact with the new feature

1. Build a new feature behind a pref flag
2. Define an experiment for that feature in Experimenter
3. Send it to Shield
4. After Shield reviews and approves it, it is sent to Firefox
5. Firefox clients check whether they should enroll in the experiment and configure themselves accordingly
6. Telemetry about the experiment is collected
7. Dashboards are created to visualize the telemetry
8. Analyze and collect the results to understand how the new feature impacted users
9. Do it again!

1. Install docker on your machine

2. Clone the repo

   git clone <your fork>
   

3. Copy the sample env file

   cp .env.sample .env
   

4. Set DEBUG=True for local development

   vi .env
   

5. Create a new secret key and put it in .env

   make secretkey
   

6. Run tests

   make test
   

7. Setup the database

   make refresh
   

8. Run a dev instance

   make up
   

9. Navigate to it and add an SSL exception to your browser

   https://localhost/
   

Done!

Experimenter uses docker for all development, testing, and deployment.

The following helpful commands have been provided via a Makefile:

Build the application container by executing the build script

Build the supporting services (nginx, postgresql) defined in the compose file

Start a dev server listening on port 80 using the Django runserver

Run the Django test suite with code coverage

Run flake8 against the code

Run both test and lint

Apply all django migrations

Create an admin user in the local dev instance

Populates locales and countries

Populates db with dummy experiments

Start an ipython shell inside the container (this lets you import and test code, interact with the db, etc)

Start a bash shell inside the container (this lets you interact with the containerized filesystem)

Create dummy SSL certs to use the dev server over a locally secure connection. This helps test client behaviour with a secure connection. This task is run automatically when needed.

Stop and delete all docker containers. WARNING: this will remove your database and all data. Use this to reset your dev environment.

Run kill, migrate, load_locales_countries load_dummy_experiments

Start up a normandy and delivery console instance. Prereqs. Symlink normandy and delivery console eg. ln -s ../normandy normandy, ensure user is assigned superuser status

Run the integration test suite inside a containerized instance of Firefox. You must also be already running a make up dev instance in another shell to run the integration tests.

Start a linux VM container with VNC available over vnc://localhost:5900 with password secret. Right click on the desktop and select Applications > Shell > Bash and enter tox -c tests/integration/ to run the integration tests and watch them run in a Firefox instance you can watch and interact with.

API documentation can be found here

Please see our Contributing Guidelines

Experimenter uses the Mozilla Public License