lsst-build, a builder and continuous integration tool for LSST
==============================================================

Provides the following capabilities:

  * Given one or more top-level packages, intelligently clone their git
    repositores and check out the requested branches into a build directory:

      lsst-build prepare 
         [--repository-pattern=format_pattern_for_repo_URLs]
         [--exclusion-map=exclusions.txt]
         [--version-git-repo=versiondbdir]
         [--ref=branch1 [--ref=branch2 [...]]]
         <builddir> <product1> [product2 [product3 [...]]]

    Run `lsst-build prepare -h' to see the full list of options.

  * Given that build directory, and an EUPS stack, intelligently build and
    install those packages:

      lsst-build build <builddir>

    Run `lsst-build build -h' to see the full list of options.

Example:

    git clone git@github.com:lsst/versiondb.git
    export REPOSITORY_PATTERN="git://git.lsstcorp.org/LSST/DMS/%(product)s.git|git://git.lsstcorp.org/LSST/DMS/devenv/%(product)s.git|git://git.lsstcorp.org/LSST/DMS/testdata/%(product)s.git|git://git.lsstcorp.org/LSST/external/%(product)s.git"

    cat > exclusions.txt <<-EOF
        # Exclusion map. Format:
        # dependency regex      product regex
        cuda_toolkit            .*
        cuda_sdk                .*
        scipy                   .*
        healpy                  .*
        condor                  .*
    EOF

    mkdir build

    lsst-build prepare \
        --exclusion-map=exclusions.txt
        --version-git-repo=versiondb
        build lsst_distrib

    lsst-build build build

lsst-build prepare
------------------

`lsst-build prepare' clones the given top level product(s) into <builddir>,
discovers its depenencies by reading its ups/$PRODUCT.table file, and
recursively clones them until all dependencies are exhausted.  Following the
clone, it writes out <builddir>/manifest.txt file, which lists all cloned
products, their versions and dependency relationships.


It maps product names to git repository URLs via a repository pattern, a
'|'-separated string of python string format patterns passed by
--repository-pattern option, or existing in a $REPOSITORY_PATTERN
environment variable. Each format pattern is evaluated in order with
%(product) replaced by the product name to construct the URL, and a
git-clone is attempted until a clone is successful.

lsst-build skips any dependencies matching a rule in an exclusion map given
--via exclusion-map option.  The exclusion map is a text file with two
entries per line: the dependency regex and the product regex. Any product
that matches the dependency regex, while being considered for cloning as a
dependency of a product that matches product regex, will be skipped.

Once cloned, a branch/tag/commit given by --ref is checked out. If multiple
--ref options are given, each ref is tested for existence until one that
exists is found and checked out. If no requested refs exist, 'master' is
checked out. This is commonly used to prepare and test the build of a
ticket branch, falling back to master if that branch doesn't exist in some
repositories (i.e., lsst-build prepare --ref ticket/1234 --ref master ...).

Upon completing the clone and ref checkouts of all packages in the product
tree, lsst-build prepare writes out a "build manifest" in
<builddir>/manifest.txt.  This is a topologically sorted
whitespace-separated list of (product, sha1, version, dependencies), with
one line per cloned product.  Given the topological sorting, it is possible
to build the packages in order of appearance in the file. The 'dependencies'
entry is a comma-separated list of dependency product names (first level
only).

In addition to these tuples, the manifest may also contain variable
assignments of the form VARNAME=VARVALUE.  Current implementation of
lsst-build defines only one, 'BUILD', which is a locally unique identifier
identifying this particular set of packages (i.e., a "build number").

The construction of version string in the build manifest depends on whether
--version-git-repo option has been used. If it is not, the version string is
constructed as <pkgautoversion>+<deps_sha1>, where pkgautoversion is the
output of EUPS' pkgautoversion tool when run in the product's git
repository, and <deps_sha1> is the abbreviated SHA1 of a sorted,
space-separated list of (product, version) tuples of its dependencies. This
guarantees that two disconnected runs of lsst-build with exactly the same
input repositories will compute the same versions.

If --version-git-repo=<versiondb> is given, then the versions are of the
form <pkgautoversion>+<N>, where N is a monotonically increasing integer. 
Nevertheless, N is guaranteed to be the same for the same set of
dependencies.  To achieve this, the files in <versiondb> directory record
the mapping from deps_sha1, computed as described in the previous paragraph,
and the integer N.  Given the same source repositories, and the same
versiondb repository, two disconnected lsst-build prepare runs are
guaranteed to generate the same versions.  Note that <versiondb> must be a
specially formatted git repository (see the 'VersionDB Repository' section
for more).

The construction of the build ID (BUILD=xxx entry in the manifest) also
depends on whether --version-git-repo option is in effect. If no, then this
ID is computed by looking for the highest EUPS tag matching 'b*' and
incrementing it by one (e.g., if EUPS tags b1, b2, and b3 already exist,
BUILD=b4 would be written into the manifest). If --version-git-repo is used,
the same algorithm is applied but on *git* tags of the versiondb repository.
Also, if --version-git-repo is used the manifest will be stored in versiondb
repository, the repository will be git-committed and tagged with the build
ID.

If --version-git-repo is used, it is advisable to git-push the repository
upstream upon a successful lsst-build prepare (esp. before lsst-build build
is run). This ensures that no versions can exist in the installed stack that
are not present in the versiondb repo on the central server.


lsst-build build
----------------

`lsst-build build' builds, installs and EUPS-declares, each product found in
the list in <builddir>/manifest.txt unless the product with the same name
and version is already EUPS-declared in the stack. The build is done using
the eupspkg mechanism; products not using eupspkg cannot be built with
lsst-build build. Each successfully installed product is EUPS-tagged with
the build identifier (the BUILD=xxxx entry in manifest.txt).


For each product found in <builddir>/manifest.txt, lsst-build build first
checks if it already exists on the stack in EUPS_PATH. If not, it enters
its directory, sets up its dependencies (roughly, using "setup -r ."), and
runs eupspkg {prep, config, build, install} sequence. If the build is
successful, it declares the product to EUPS. Either way, the product is
tagged with the BUILD value given in manifest.txt.

At the end of a successful run, all products listed in
<builddir>/manfest.txt will have been build, declared and installed into the
active EUPS stack (the first entry on $EUPS_PATH), and tagged with the value
of BUILD (the "build number").

Environment Variables
---------------------

lsst-build may use the following environment variables:

 * REPOSITORY_PATTERN

    A '|'-separated string of python string format patterns used to map
    product names to URLs of their git repositories.  Each format pattern is
    evaluated with %(product) replaced by the product name to construct the
    URL.  A git-clone is attempted on each one (in order), until a clone is
    successful. Setting this variable is equivalent to specifying the patter
    via --repository-pattern option.

 * EUPS_PATH

   The colon-separated path to EUPS-managed software stacks. lsst-build will
   use these (via EUPS) to determine whether products need to be built or
   have already been built, and will install new products into this
   location (using eupspkg).

VersionDB Repository
--------------------

VersionDB is a specially-formatted git-managed directory that serves as a +N
dependency number and build ID number database for lsst-build with
--version-git-repo option (see above).

It consists of three subdirectories:

 * ver_db/

    A directory of text files, one per product, with space-separated
    (product_version, N, deps_sha1) pairs.  Used by lsst-build to always
    assign the same +N suffix to same set of dependencies (both products and
    their versions).

 * dep_db/

    A directory of text files, one per product, with space-separated list of
    (product_version, N, dep_name, dep_version) tuples.  May be used by the
    user to quickly look up a set of dependencies corresponding to some +N
    suffix.  Note that this information is also available in the product's
    expanded table file.

 * manifests/

    A directory where build manifests are stored. Each manifest is named
    $BUILD.txt, where $BUILD is the unique build ID.  Every time lsst-build
    is run with --version-git-repo, it will add the resulting manifest to
    this directory, git-commit it, and tag it with the build ID, unless a
    manifest with matching content already exists. In the latter case,
    that build ID will be reused and no new commits will be added to
    VersionDB.

Note that it is not necessary to understand this internal format to use this
repository; in fact, one should *not* depend on its internal format, as it
may change as lsst-build itself is improved.