CLEAR LINUX* DISSECTOR
======================

This is a small Django-based web application that enables comparing
the contents of Linux-based images. The initial set of functionality
enables comparing images produced by OpenEmbedded / Yocto Project tools
to Clear Linux.

There are two main methods of setting up this application - within
a set of Docker containers, or standalone. The Docker-based setup
is more suited for production whereas standalone is a bit easier
for development. This document will consider only the Docker-based
setup; for standalone please see README.devel.


Docker Setup
------------

The dockersetup.py script will set up and configure a cluster of 5 or 6
docker containers:

  - layersapp: the application
  - layersdb: the database
  - layersweb: NGINX web server (as a proxy and for serving static content)
  - layerscelery: Celery (for running background jobs)
  - layersrabbit: RabbitMQ (required by Celery)
  - layerscertbot: Runs certbot to keep letsencrypt certificates up-to-date
                   (optional, default disabled)

The script will edit all necessary configuration files, build and launch all
containers, and do the initial database setup. It is advised that you start
with a .sql database file to prepopulate your database. The following
instructions will walk you through the setup.

1) Install docker and docker-compose per instructions:

        https://docs.docker.com/compose/install/

   ** Note: for latest docker-compose version follow the directions above,
      rather than using a perhaps outdated one provided by your distribution.

2) Run the setup script (dockersetup.py). You can optionally supply your
   hostname, proxy settings, a sql database file of layer mappings to import,
   and a host to container port mapping. For more information, run:

        ./dockersetup.py -h

   Example command to run containers with a proxy and with a database to
   import:

        ./dockersetup.py -d ~/databasedump.sql -p http://<proxyserver>:<port>

   NOTE: If you want email sending to work (required for user registration and
   password resets), you should use the -e/--email-host option to specify your
   outgoing email server.

   NOTE: If you import an existing database dump, you will get whatever data
   was in the database, however you will not have the corresponding Clear Linux
   sources which means that file diffs for version comparisons won't be
   possible. To obtain the sources please see "Importing/Updating Clear Linux
   data" below.

   During the setup you will be asked for a username, email and password to
   set up a super user for the database.

3) Once the script completes, open a web browser and navigate to the URL
   printed out by the script. By default that would be: https://localhost:8081

4) If you have chosen to not supply a prepopulated database and are instead
   starting fresh, you should now follow the instructions in the
   "Database Setup" section of the main README.

5) If you need to rerun this script for any reason a second time, you'll need
   to choose between two modes:

   A) Updating (-u/--update) - updates the code and runs any database upgrades
      if applicable, or

   B) Reinstalling (-r/--reinstall) - deletes the containers and reinstalls
      from scratch. Be warned that this will throw away all data in the
      database.

   Note that updating with -u/--update will only work if the configuration
   changes originally made by dockersetup.py upon installation (e.g. passwords,
   hostname, etc.) are still present in the source tree.



TROUBLESHOOTING:

- Network issues behind a proxy when building container: On some systems
  (particularly where dnsmasq is installed), /etc/resolv.conf is set to
  127.0.0.x, rather than your local DNS server. Docker will look there for
  your DNS server, and when it fails to find it it will default to using a
  public one (frequently 8.8.8.8). Many corporate proxies blocks public DNS
  servers, so you will need to manually supply the DNS server to docker using
  /etc/docker/daemon.json:

	{"dns": ["xx.xx.xx.xx] }



Database Setup
--------------

For a fresh database (without importing any database dump) you will need layer
data. To import the full set for the master branch from the public instance at
layers.openembedded.org you can run the following:

docker-compose run --rm layersapp /opt/layerindex/layerindex/tools/import_layers.py -d https://layers.openembedded.org

Alternatively, you can populate the database manually or use the separate
import_layer.py script to import layers individually.

NOTE: Please also see "Security Considerations" below.


Updating OpenEmbedded data
--------------------------

You may wish to update the OpenEmbedded layer information on a regular basis.
To do that:

   Incremental update:

       docker-compose run --rm layersapp /opt/layerindex/layerindex/update.py

   Reload all data (should only be needed if the database structure has
   changed as part of an upgrade of the application code):

       docker-compose run --rm layersapp /opt/layerindex/layerindex/update.py -r


Importing/Updating Clear Linux data
-----------------------------------

Importing data from Clear Linux (or a suitably-structured derivative) can
be done through the web UI by users with the
"layerindex | classic recipe | Can update comparison branches" permission
via the "Import Distro Data" button on the main page. However, if you wish
to import data on the backend you can do so as follows:

docker-compose run --rm layersapp /opt/layerindex/layerindex/tools/import_clear.py -d -p /opt/dissector -o /opt/sources -b <branchname>

Where <branchname> should be replaced by the name of the Clear Linux branch
to use (e.g. "clearlinux"). You can import into a new branch if you wish,
and specify a specific version with -r. Use the --help option to see more.

Importing a derivative Clear Linux distribution is similar; you will need
the corresponding source tarball:

docker-compose run --rm layersapp /opt/layerindex/layerindex/tools/import_clear.py -d -p /opt/dissector -o /opt/sources -b <branchname> -g <derivative tarball URL>

Typically when importing a derivative you would specify a *different* branch
than your main Clear Linux branch.



Upgrading from an earlier version
---------------------------------

To upgrade the Docker-based setup from a previous release, simply run:

./dockersetup.py -u

This will update the code and run any required database migrations.



Setting up other distro comparisons
-----------------------------------

The Layer Index also provides optional functionality to enable comparison
with other distributions (currently RPM-based only) in a similar manner to
OE-Classic comparison documented above. To set this up you need to perform
the following steps:

1. From the admin interface, set up the appropriate entries:

   1.1. Create a Branch with the following values:
        - Bitbake branch: <any dummy value>
        - Enable updates: NOT enabled
        - Comparison: enabled

   1.2. Create a Layer. Typically this would have the same name as
        the branch although that is not a requirement. The "Comparison"
        checkbox should be ticked. If the packages are in separate
        repositories (one per package, as is typical in RPM-based
        distributions such as Fedora) then in order to make the links
        through to files work correctly you may need to use repository web
        interface URLs similar to these:

        Repository web interface tree base URL:
          https://github.com/organisationname/%pathelement[0]%/tree/master/%pathelement[1:]%
        Repository web interface file base URL:
          https://github.com/organisationname/%pathelement[0]%/blob/master/%pathelement[1:]%

   1.3. Create a LayerBranch to link the Branch and Layer that you
        created in the previous steps. You don't need to enter anything
        special here.

2. Run the import script, specifying the branch and layer names and the path
   to the base of the packages (where each subdirectory contains a package,
   notably a spec file describing the package):

   layerindex/tools/import_otherdistro.py import-pkgspec <branchname> <layername> <pkgpath>

3. Update the comparison status of recipes based on layers in the database:

   layerindex/tools/update_classic_status.py -l <layername> -b <branchname>

4. Optionally enable the Update button in the UI by setting COMPARISON_UPDATE
   in settings.py to map each other distro branch to the command that should
   be run in the background when the button is pressed. For example:

COMPARISON_UPDATE = [
    {
        'branch_name': 'otherlinux',
        'update_command': 'layerindex/tools/import_otherdistro.py import-pkgspec otherlinux otherlinux /path/to/pkgs -u %update%; layerindex/tools/update_classic_status.py -b otherlinux -l otherlinux -u %update%',
    },
]

If you refresh the main page of the website, the other distro data should
now show up at the bottom of the branch drop-down menu. On a periodic
basis you can repeat steps 2 and 3 to refresh the data as changes occur on
both sides. Users with sufficient permissions can also manually update the
migration status on the other distro recipe detail pages within the website,
which is useful for example when there's an equivalent recipe in another
layer that doesn't have the same name, so the update_classic_status.py
script wouldn't be able to pick it up.

If you want to show links to additional upstream pages associated with
packages in the other distro, you can add "Layer recipe extra URL" entries
for each type of link you want to be shown. For example, Fedora provides
a summary page for each package - acl's one is at
https://apps.fedoraproject.org/packages/acl, so you would create a
Layer Recipe Extra URL entry with the template URL
"https://apps.fedoraproject.org/packages/%pn%" and then links would be
shown for this under the detail page for each package in the other distro.
If you have the rrs application enabled the link will also be shown in the
"Distros" section of the maintenance detail page for the covering recipe.



Security Considerations
-----------------------

Some things to be aware of from a security perspective:

* By default, anyone can register themselves an account. If you wish to
  disable new user registration and manage users manually through the admin
  interface instead, then add the following line to docker/settings.py:

REGISTRATION_OPEN = False

  Then, assuming you have already run dockersetup.py to install the
  application, run the following command to update it:

./dockersetup.py -u

* By default, dockersetup.py enables connection to the web server via
  HTTPS using a self-signed certificate; connections via HTTP are
  re-directed to HTTPS. However, the self-signed certificate is only
  intended to provide a minimum level of security, but will result in
  browser warnings and is not recommended for production - instead,
  obtain and use your own certificate/key pair corresponding to the
  domain which will be used to access the application in production,
  or alternatively if the application is accessible to the internet you
  can use Let's Encrypt.

* If you provide your own certificates for HTTPS, you should probably
  also enable HSTS in your configuration. Refer to the Django Security
  guide for details on how to do that:

  https://docs.djangoproject.com/en/1.11/topics/security/#ssl-https

* To reset a forgotten account password, you can either use the password
  reset function ( /accounts/password_reset/ ) or alternatively from the
  backend you can run the following command:

  docker-compose exec layersapp /opt/layerindex/manage.py changepassword <username>

* The web-based password reset function will ask the user answers to
  security questions they selected and answered when they created the
  account. Admins can configure the selectable security questions in
  the admin interface under "Security questions"; however, be cautious
  about deleting or substantially changing a question if you already
  have user accounts that have given answers to that question, as doing
  so will invalidate the user-set answers. You can check this if you go
  to delete the security question in the admin interface - any user
  answers will show up as "Security question answers" listed to be
  deleted along with the question.

  Note: the superuser created during setup will not have answers to
  security questions set, so if you think you might need to use the
  password reset function later you will need to set these by logging
  into the application and then going to Edit Profile on the top-right
  user drop-down menu.

* Security question answers are stored using the same mechanism as
  for passwords, i.e. a secure one-way hash; thus answers cannot be
  retrieved from the database once set. Additionally, if a user wants
  to change one of their answers via the Edit Profile function, they
  will be required to re-specify all of them.

* Account lockout: this application will lock out the user in two ways:

  - By IP address (using django-axes) after too many invalid login attempts.
    (default 4, resets after an hour). The lockout can be removed
    immediately using the following command:

    docker-compose exec layersapp /opt/layerindex/manage.py axes_reset

    If you wish to disable this, remove or comment out "axes" in
    INSTALLED_APPS. For more information on configuring axes, see:

    https://django-axes.readthedocs.io/en/latest/

  - By account for too many incorrect password reset attempts. To remove
    this lockout, log into the admin interface, go to Users, click on the
    the user, tick the Active checkbox and then save.

* The REST API inherited from the OpenEmbedded layerindex upon which this
  application is based has been disabled, since it has no access controls
  on querying data (since the OE layer index operates entirely on publicly
  accessible information, whereas this application may not in actual usage).


Maintenance
-----------

The code for this application is maintained by Intel, as a fork of the
OpenEmbedded layer index software maintained by the Yocto Project.

The latest version of the code can always be found here:

  https://github.com/intel/clear-linux-dissector-web

Contributions are welcome. Please send changes in the form of pull requests
on the github repository linked above.


License
-------

This application is based upon the Django project template, whose files
are covered by the BSD license and are copyright (c) Django Software
Foundation and individual contributors.

Bundled Bootstrap (including Glyphicons) is redistributed under
the MIT license.

Bundled jQuery is redistributed under the MIT license.

Bundled Chart.js is redistributed under the MIT license.

Bundled TableSorter is redistributed under the MIT license.

Bundled diff2html is redistributed under the MIT license.

Bundled and modified django-registration-templates is redistributed
under the MIT license.

All other content is copyright (C) 2013-2019 Intel Corporation and
licensed under the MIT license (unless otherwise noted) - see
COPYING.MIT for details.


Trademarks
----------

* Other names and brands may be claimed as the property of others.