OE Layer Index web interface
============================

This is a small Django-based web application that provides a way to
manage an index of OpenEmbedded metadata layers for use on top of
OE-Core.

There are two main methods of setting up this application - within
a set of Docker containers, or standalone. The Docker-based setup
is more suited for production whereas standalone is a bit easier
for development.


Docker Setup
------------

To use layerindex-web hosted within Docker containers, refer to
docker/README keeping in mind you'll need to set up Docker properly
as part of the setup process. Once you've run through those
instructions you'll need to return to this document and follow the
instructions in the section "Database Setup" below.


Standalone Setup
----------------

In order to run this application standalone you will need:

* Python 3.4+
* Django 1.11.x - tested with 1.11.12; newer versions may work, but
  the application has not been tested with 2.0 or newer.
* RabbitMQ 3.6.x - tested with 3.6.10.
* For production usage, a web server set up to host Django applications
  (not needed for local-only testing/development)
* A database supported by Django (SQLite, MySQL, etc.). Django takes
  care of creating the database itself, you just need to ensure that the
  database server (if not using SQLite) is configured and running.
* Third-party Django/python modules as detailed by requirements.txt.
  If you're using pip, you can install these by running:
    pip install -r requirements.txt
* On the machine that will run the backend update script (which does not
  have to be the same machine as the web server, however it does still
  have to have Django installed, have the same or similar configuration
  in settings.py and have access to the database used by the web
  application):
  * Python 2.7.6+ / Python 3.4+ to match with the version of BitBake
    for the OpenEmbedded branch being parsed (for modern versions it's
    Python 3.)
  * Python dependencies as per requirements.txt (we still need Django
    etc. here since we interact with the database through Django's ORM.)

Example virtualenv-based setup for the above:

Python's virtualenv provides an easy way to isolate the python dependencies
of applications such as the layer index. Here's an example of setting up a
virtualenv for the layer index that's particularly useful for development.
(This assumes a Debian-based distribution, adjust accordingly for other
distros).

1. Install required host distro packages (some of these are required by
   pip to build the dependencies; it's also assumed you want MariaDB as
   the database backend):

   sudo apt-get install virtualenv libmariadb-client-lgpl-dev build-essential python3-dev libjpeg-dev libz-dev libfreetype6-dev mariadb-server rabbitmq-server

2. Work around path issues (you may not need this):

   sudo ln -s /usr/bin/mariadb_config /usr/bin/mysql_config

3. Create a Python 3 virtualenv (path can be anywhere you like):

   virtualenv -p python3 /path/to/desired/venv

4. Activate the virtualenv:

   . /path/to/desired/venv/bin/activate

5. Install requirements:

   pip install -r requirements.txt


Configuration and running:

1. Edit settings.py to specify a database, EMAIL_HOST, SECRET_KEY and
   other settings specific to your installation. Ensure you set
   LAYER_FETCH_DIR to an absolute path to a location with sufficient
   space for fetching layer repositories. Modify RABBIT_BROKER
   and RABBIT_BACKEND to reflect the settings used by your RabbitMQ
   server.

2. Run the following commands within the layerindex-web directory to
   initialise the database:

   python3 manage.py migrate
   python3 manage.py createsuperuser

3. You can test the web application locally by setting DEBUG = True in
   settings.py and running the following:

   python3 manage.py runserver

   Then visit http://127.0.0.1:8000/layerindex/ with your browser. As
   with all Django applications there is an admin interface available
   at http://127.0.0.1:8000/admin/ also. The initial login and password
   will be those you entered in the step above when creating an admin
   account.

   NOTE: This local server should only be used for testing - for
   production you need to use a proper web server and have DEBUG set
   to False (or use the Docker-based setup).

   3.1. In order to process asynchronous tasks like sending email,
        you will need to run a Celery worker:

        celery -A layerindex.tasks worker --loglevel=info



Database Setup
--------------

Once the application is running you'll need to do a bit of further
setup within it:

1. You'll need to add at least the openembedded-core layer to the
   database, or some equivalent that contains conf/bitbake.conf for
   the base system configuration. To add this, follow these steps:

   1.1. With the server running, go to the main page (
        http://127.0.0.1:8000/admin/ or equivalent depending on
        your setup) and click on the "Log in" button on the top
        right. Use the login/password for the admin account you
        added previously.

   1.2. Click on the "Submit Layer" button in the top right and
        enter the details for the core layer. To use the real
        openembedded-core layer, use these values:

        Layer name:               openembedded-core
        Layer type:               Base
        Summary:                  Core metadata
        Description:              Core metadata
        Repository URL:           git://git.openembedded.org/openembedded-core
        Repository subdirectory:  meta

        Once you have filled in the required values, click on the
        "Submit Layer" button.

        NOTE: The name of the layer must be "openembedded-core",
        unless you change CORE_LAYER_NAME in settings.py to match
        whatever alternative name you use here.

   2.3. The layer has been added but is not yet published. (For the
        public index this provides some protection against spam and
        malformed entries.) To publish it, click on the orange number
        next to your login name at the top right, click on the
        newly added layer entry, and then click on "Publish Layer".

2. If you need to support multiple branches of OpenEmbedded/BitBake
   where some require Python 2.x and others require Python 3.x, then
   you will need to set up "Python environment" records through the
   admin interface to correspond to these so that the right Python
   version gets used to parse the branch, and then set the
   "Update environment" field on each branch record to point to the
   appropriate environment. If you're using virtualenv you will need
   separate virtual environments set up for Python 2 and 3 which you
   should point to in the Python environment record.

3. Set the site name (as displayed in the top bar and page titles) by
   going into the admin interface (http://127.0.0.1:8000/admin/
   or equivalent), clicking on "Sites" at the bottom, and editing the
   first entry, setting "Display name" to the desired name.

4. You may wish to customise some of the page templates to suit your
   installation, in particular:
   * templates/base.html
   * templates/layerindex/about.html



Usage
-----

On a regular basis you need to run the update script:

path/to/layerindex/update.py

This will fetch all of the layer repositories, analyse their contents
and update the database with the results. Run the script with --help for
further information on available options.


Upgrading from an earlier version
---------------------------------

This application uses Django's migrations functionality to allow changes
in the database structure whilst preserving existing data. When upgrading
with an existing database, you just need to invoke the migration command.

If you're migrating from a version prior to the Django 1.8 upgrade, you
need to use the following command so that the initial migration (which
takes care of creating the database structure and initial data) is
skipped:

python3 manage.py migrate --fake-initial

Otherwise, you can just run the plain migration command:

python3 manage.py migrate


Support for OE-Classic
----------------------

The Layer index optionally provides a means to index OE-Classic on a
one-off import basis and then compare what was there to what you have
now in the indexed layers (with some graphs showing how much of it has
been migrated/superseded). If you want to enable this, do the following:

1. From the admin interface, create a Branch record with the following
   values:
     - Name: oe-classic (must be exactly this!)
     - Bitbake branch: 1.12
     - Enable updates: NOT ticked
     - Comparison: ticked
     - Update environment: if you have set up Python environments (for
       python2 vs python3 across different branches) then you'll need
       to select the python2 environment that you created here

2. Clone OE-Classic somewhere locally on the machine running the
   layer index:

   git clone git://git.openembedded.org/openembedded

3. Clone a bitbake somewhere locally and check out the 1.12 branch:

   git clone git://git.openembedded.org/bitbake -b 1.12

4. Run import_classic.py, specifying the path to OE-Classic and
   the bitbake you checked out:

   layerindex/tools/import_classic.py /path/to/bitbake112 /path/to/oeclassic

5. Update the migration status of OE-Classic recipes based on other
   layers in the database:

   layerindex/tools/update_classic_status.py

If you refresh the main page of the website, the OE-Classic data should
now show up at the bottom of the branch drop-down menu. On a periodic
basis you can repeat the last step to update the migration status in case
new recipes are brought across (or replacements are created). Users with
sufficient permissions can also manually update the migration status on
the OE-Classic recipe detail pages within the website, which is useful
for example when there's a replacement recipe in another layer that
doesn't have the same name, so the update_classic_status.py script
wouldn't be able to pick it up.


Setting up other distro comparisons
-----------------------------------

The Layer Index also provides optional functionality to enable comparison
with other distributions (currently RPM-based only) in a similar manner to
OE-Classic comparison documented above. To set this up you need to perform
the following steps:

1. From the admin interface, set up the appropriate entries:

   1.1. Create a Branch with the following values:
        - Bitbake branch: <any dummy value>
        - Enable updates: NOT enabled
        - Comparison: enabled

   1.2. Create a Layer. Typically this would have the same name as
        the branch although that is not a requirement. The "Comparison"
        checkbox should be ticked. If the packages are in separate
        repositories (one per package, as is typical in RPM-based
        distributions such as Fedora) then in order to make the links
        through to files work correctly you may need to use repository web
        interface URLs similar to these:

        Repository web interface tree base URL:
          https://github.com/organisationname/%pathelement[0]%/tree/master/%pathelement[1:]%
        Repository web interface file base URL:
          https://github.com/organisationname/%pathelement[0]%/blob/master/%pathelement[1:]%

   1.3. Create a LayerBranch to link the Branch and Layer that you
        created in the previous steps. You don't need to enter anything
        special here.

2. Run the import script, specifying the branch and layer names and the path
   to the base of the packages (where each subdirectory contains a package,
   notably a spec file describing the package):

   layerindex/tools/import_otherdistro.py import-pkgspec <branchname> <layername> <pkgpath>

3. Update the comparison status of recipes based on layers in the database:

   layerindex/tools/update_classic_status.py -l <layername> -b <branchname>

4. Optionally enable the Update button in the UI by setting COMPARISON_UPDATE
   in settings.py to map each other distro branch to the command that should
   be run in the background when the button is pressed. For example:

COMPARISON_UPDATE = [
    {
        'branch_name': 'otherlinux',
        'update_command': 'layerindex/tools/import_otherdistro.py import-pkgspec otherlinux otherlinux /path/to/pkgs -u %update%; layerindex/tools/update_classic_status.py -b otherlinux -l otherlinux -u %update%',
    },
]

If you refresh the main page of the website, the other distro data should
now show up at the bottom of the branch drop-down menu. On a periodic
basis you can repeat steps 2 and 3 to refresh the data as changes occur on
both sides. Users with sufficient permissions can also manually update the
migration status on the other distro recipe detail pages within the website,
which is useful for example when there's an equivalent recipe in another
layer that doesn't have the same name, so the update_classic_status.py
script wouldn't be able to pick it up.

If you want to show links to additional upstream pages associated with
packages in the other distro, you can add "Layer recipe extra URL" entries
for each type of link you want to be shown. For example, Fedora provides
a summary page for each package - acl's one is at
https://apps.fedoraproject.org/packages/acl, so you would create a
Layer Recipe Extra URL entry with the template URL
"https://apps.fedoraproject.org/packages/%pn%" and then links would be
shown for this under the detail page for each package in the other distro.
If you have the rrs application enabled the link will also be shown in the
"Distros" section of the maintenance detail page for the covering recipe.




Maintenance
-----------

The code for this application is maintained by the Yocto Project.

The latest version of the code can always be found here:

  http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/

Contributions are welcome. Please send patches / pull requests to
yocto@yoctoproject.org with '[layerindex-web]' in the subject.


License
-------

This application is based upon the Django project template, whose files
are covered by the BSD license and are copyright (c) Django Software
Foundation and individual contributors.

Bundled Bootstrap (including Glyphicons) is redistributed under
the MIT license.

Bundled jQuery is redistributed under the MIT license.

Bundled uitablefilter.js is redistributed under the MIT license.

Bundled nv.d3.js is redistributed under the Apache License 2.0.

Bundled d3.js is redistributed under the BSD License.

All other content is copyright (C) 2013-2018 Intel Corporation and
licensed under the MIT license (unless otherwise noted) - see
COPYING.MIT for details.