• Prerequisites
• Getting Started: The Quick Version
• Getting Started: The Unabridged Edition
• Management Commands
• Updating Requirements
• Advanced Actions Against the Database
• Other Commands
• Troubleshooting

The installation instructions below assume you have the following software on your machine:

• Docker
• Pipenv (if not using Docker for Mac)

To get started, run:

make dev-init  # one-time command
docker-compose up  # long-running process to run application server, every time

# In a separate shell:
docker-compose exec django ./manage.py createdevdata  # one-time command

Visit http://localhost:8000/ to see the site or http://localhost:8000/admin/ for the Wagtail admin.

The development environment uses Docker Compose to run the application server, database, and webpack compilation processes. If you are using Docker for Mac, this comes preinstalled. You may skip the Pipenv related code. Otherwise, you can install it for this project using Pipenv:

pipenv install

When installed this way, you will need to activate the env before running all docker-compose commands with:

pipenv shell
# or
pipenv run $command $args

These must be run from the project directory.

Before development you must run this one-time command.

make dev-init

To start the environment, run the following your first run:

docker-compose up

This is how you start the server every time you are working on the project. This will start a long-running process. You can exit this process with ctl-c. You may wish to open a second shell to run one-off commands while the server is running.

To populate the project with data suitable for development and testing.

docker-compose exec django ./manage.py createdevdata

You should be able to hit the web server interface at http://localhost:8000/. You can access the Wagtail admin at http://localhost:8000/admin/.

To learn more about Docker Compose, see the docker-compose CLI docs

In addition to the management commands provided by Django and Wagtail, the project has a set of its own custom management commands. All commands listed should be prefaced by docker-compose exec django ./manage.py.

These commands are meant to be used once at the beginning of development. They can be run individually or all at once using the createdevdata command. They should not be run in production as they create fake data.

• createdevdata [--delete]

  Runs all of the other create* commands and creates fake data. The delete flag deletes the current homepage and creates a new one.

• createblogdata <number_of_posts>

  Creates a blog index page and the indicated number of posts.

• createdirectory <number_of_instances>

  Creates a directory page and theindicated number of SecureDrop instances.

• createresultgroups [--delete]

  Creates the initial text for the scan results shown on the details page of a securedrop instance. The delete flag removes current result groups and result states.

• createfootersettings

  Creates the initial default text, menus, and buttons for the footer.

• createnavmenu [--delete]

  Creates the main nav menu and links it to the appropriate pages. Creates a DirectoryPage, BlogIndexPage, and MarketingIndexPage if they do not yet exist. The delete flag destroys the existing nav menu.

• createsearchmenus [--delete]

  Creates default search menus. The delete flag destroys any existing search menus.

• scan [securedrops]

  Scan one or more SecureDrop landing pages (specified by space-separated domain names) for security. By default, scans all pages in the directory.

• update_docs_index [--rebuild]

  Crawl the SecureDrop documentation pages on https://docs.securedrop.org/en/stable/ and update the corresponding SearchDocument entries. Pass --rebuild to this command to delete existing entries for documentation pages before fetching new data, which is useful if out-of-date information or pages are in the index. Rebuild is usually the behavior that you will want. Note that this command depends on a particular arrangement and format of HTML and links on the above 3rd party web URL. If these change in the future, then the command will potentially fail and report zero or only a few documents indexed.

• update_wagtail_index [--rebuild]

  Crawl Wagtail pages and create SearchDocuments for each one. This command should only be run once when the repo is initialized, as thereafter SearchDocuments will be updated via get_search_content which is run when pages are created, updated, or deleted. Note that if pages are changed outside of the Wagtail interface, their search documents will not be updated and this command will need to be run again. Pass --rebuild to this command to delete existing entries for Wagtail pages before fetching new data, which is useful if out-of-date information or pages are in the index.

• update_discourse_index [--rebuild]

  Crawl the SecureDrop forum pages on https://forum.securedrop.club/ and update the corresponding SearchDocument entries. Pass --rebuild to this command to delete existing entries for documentation pages before fetching new data, which is useful if out-of-date information or pages are in the index. Rebuild is usually the behavior that you will want.

  This command depends on two settings: DISCOURSE_HOST which should be set to the name of the Discourse server without the protocol (forum.securedrop.club) and DISCOURSE_API_KEY. If you require these for development, acquire them securely from a Discourse forum administrator and stash them in securedrop/settings/local.py.

New requirements should be added to *requirements.in files, for use with pip-compile. There are two Python requirements files:

• requirements.in production application dependencies
• dev-requirements.in development container additions (e.g., debug toolbar)

Add the desired dependency to the appropriate .in file, then run:

make update-pip-dependencies

All requirements files will be regenerated based on compatible versions. Multiple .in files can be merged into a single .txt file, for use with pip. The Makefile target handles the merging of multiple files.

The developer environment dependencies are handled via pipenv and documentation for that project can be found here.

Drop a Postgres database dump into the root of the repo and rename it to import.db. To import it into a running dev session (ensure docker-compose up has already been started) run make dev-import-db. Note that this will not pull in images that are referenced from an external site backup.

The postgresql service is exposed to your host on a port that will be displayed to you in the output of docker-compose port postgresql 5432. If you have a GUI database manipulation application you'd like to utilize point it to localhost with the correct port, username securedrop, password securedroppassword, dbname securedropdb

You can mimic a production environment where django is deployed with gunicorn, a reverse nginx proxy, and debug mode off using the ci-docker-compose.yaml file. Note that build time for this container takes much longer than the developer environment:

docker-compose -f prod-docker-compose.yaml up

It is not run using live-code refresh so it's not a great dev environment but is good for replicating issues that would come up in production.

When developing, it is often required to switch branches. These different branches can have mutually incompatible changes to the database, which can render the application inoperable. It is therefore helpful to be able to easily restore the database to a known-good state when making experimental changes. There are two commands provided to assist in this.

make dev-save-db: Saves a snapshot of the current state of the database to a file in the db-snapshots folder. This file is named for the currently checked-out git branch.

make dev-restore-db: Restores the most recent snapshot for the currently checked-out git branch. If none can be found, that is, make dev-save-db has never been run for the current branch, this command will do nothing. If a saved database is found, all data in database will be replaced with that from the file. Note that this command will terminate all connections to the database and delete all data there, so care is encouraged.

Workflow suggestions. I find it helpful to have one snapshot for each active branch I'm working on or reviewing, as well as for master. Checking out a new branch and running its migrations should be followed by running make dev-save-db to give you a baseline to return to when needed.

When checking out a new branch after working on another, it can be helpful to restore your snapshot from master, so that the migrations for the new branch, which were presumably based off of master, will have a clean starting point.

In order to ensure that all commands are run in the same environment, we have added a make flake8 command that runs flake8 in the docker environment, rather than on your local env.

Sometimes when dependencies are changed or a Docker image needs to be updated for other reasons, the containers will need to be manually triggered to rebuild. These commands, listed in order of destructiveness can resolve most container issues:

docker-compose up --build

Adding the --build flag tells Docker Compose to detect and update any images that require new changes. You can safely add the --build flag under most circumstances without adverse effects.

docker-compose up --build --force-recreate

Adding the --force-recreate flag tells Docker Compose to recreate all containers that are part of the application.

If neither of the above fix the issues you're encountering, ensure all docker containers are stopped (ctl-c if containers are running in a shell, docker-compose kill if they are running detached) and run the following commands. These commands will remove all images ad containers and rebuild from scratch. Any data in your database will be wiped.

docker-compose rm
docker-compose up --build

If you want to use the PDB program for debugging, it is possible. First, add this line to an area of the code you wish to debug:

import ipdb; ipdb.set_trace()

Second, attach to the running Django container. This must be done in a shell, and it is within this attached shell that you will be able to interact with the debugger. The command to attach is docker attach <ID_OF_DJANGO_CONTAINER>, and on UNIX-type systems, you can look up the ID and attach to the container with this single command:

docker attach $(docker-compose ps -q django)

Once you have done this, you can load the page that will run the code with your import ipdb and the debugger will activate in the shell you attached. To detach from the shell without stopping the container press Control+P followed by Control+Q.