Skip to content
This repository has been archived by the owner on Sep 24, 2018. It is now read-only.

NGenetzky/espa

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Author: David V. Hill, SGT
        Ron Dilley, Innovate!

License: NASA Open Source Agreement 1.3
USGS Designation: EROS Science Processing Architecture (ESPA)

Change Notes
Version 2.8.5 (July 2015)
        Added additional error condition to network errors 
            - retry on socket timeouts

Version 2.8.4 (July 2015)
        Added additional error conditions to the auto error resolver
            - sixs failures now retry
            - additional warp failures result in status unavailable

Version 2.8.3 (July 2015)
        Modified RSS feed queries to use raw SQL due to performance overhead
        with large orders.
        Created separate uwsgi ini configurations for each environment to
        ease deployment logic.
        Removed deploy_install.sh and replaced with deploy_install.py,
        which now works with github instead of subversion on Google Projects.
        Updated lsrd_stats.py to work off of separate credentials to support
        reporting from the historical db.
        Enhanced processing system to support using a local path for output
        product delivery and code cleanup associated with supporting the
        local path.
        Modified warping parameters for UTM South to match UTM North with both
        now using the WGS84 datum.

Version 2.8.1 (May 2015)
        Added additional error conditions to the master error handler
        Cosmetic changes for field labels
        Removed inventory restrictions on L8SR TIRS failure scenes

Version 2.8.0 (May 2015)

Version 2.7.2 (March 4, 2015)
        Added DSWE to ECV section for staff only
        Corrected bugs with validation messages
        UI updates for images on index page

Version 2.7.1 (January 6, 2015)
        Added status update retry logic to processing tier.
        Corrected set_product_retry in app tier to include log file.

Version 2.7.0 (December 22, 2014)
        Added auto-retry logic.
        Added landsat 8 surface reflectance.
        Added timeout for SUDS ObjectCache.
        Utilizing EE HTTP services for ordering and level 1 download.

Version 2.5.0 (August 27, 2014)
        Added console application for staff personnel.
        Added priority to Orders in conjunction with Hadoop Queueing to prevent
        large orders from dominating all compute resources.
        Moved espa schema file to espa-common to prevent network effects from
        causing processing failures.
        Added select MODIS products as customizable input products.
        Corrected error with invalid checksum generation.
        Multiple bug fixes relating to image reprojection and resampling.
        Added trunk/espa_common to hold shared libs between web and processing.

Version 2.4.0 (July 29, 2014)
        Transitioned to binary raster processing for science algorithms.
        Introduced schema constrained xml metadata for all products.
        Format conversion has been introduced.
        Reprojected products now have properly populated metadata entries.

Version 2.3.0 (May 30, 2014)

        *Note -- All changes for this release are for the web tier only.
                 No modifications were made to the back end processing code.

        -----------------
        major features
        -----------------
        Added ability to authenticate users against EarthExplorer
            - reorganized lta.py, added service clients for RegistrationService

        L1T order submission changed from LTA Massloader to
        LTA OrderWrapperService

        Added temporary logic to search for user email addresses in both the
        Django auth User table and the Order email field to support migration
        of EE auth.
            - Order.email field will be removed with the next release.

        -----------------
        platform
        -----------------
        Upgraded to Django 1.6

        -----------------
        project structure
        -----------------
        Upgraded project structure to be Django 1.6 compliant

        Reorganized the orderservice directory
            - now called 'web'
            - moved contents of 'htdocs' into 'web/static'
            - populated 'web/static' with Django admin static content,
              eliminating need for softlinks
                - regenerate these with ./manage.py collectstatic

        -----------------
        settings.py
        -----------------
        Modified settings.py to include the static files app and set STATIC_ROOT, STATIC_URL
        Added LOGIN_URL, LOGIN_REDIRECT_URL
        Added URL_FOR() lambda function as service locator function
            - was previously implemented in lta.py, multiple modules now need this and its a critical dependency 
        Added ESPA_DEBUG flag to settings.py
            - looks for ESPA_DEBUG = TRUE in environment variables, enables DEBUG and TEMPLATE_DEBUG

        Moved all configuration items to settings.py
            - configurable items can be set in espa-uwsgi.ini or as environment variables
            - private key & database credentials must be provided on the local filesystem in ~/.cfgno

        -----------------
        espa-uwsgi.ini
        -----------------
        Removed the /media mapping in espa-uwsgi.ini as it is no longer necessary (merged into /static mapping)

        -----------------
        Misc
        -----------------

        django.wsgi removed in favor of django generated wsgi.py

        Deleted scene_cache.py from the espa/ codebase
            - Moved the scene cache replacement from trunk/prototype/new_scene_cache to trunk/scenecache.

        Moved all Order & Scene related operations to the models.Order and models.Scene classes as either class or static methods

        Removed all partially hardcoded urls from template and view code, replaced with named urls, {% url %}, {% static %} and reverse() tags

        Moved all cross-application functionality into espa_web

        Added espa_web/context_processors.py to include needed variables in templates
            - must include in settings.py TEMPLATE_CONTEXT_PROCESSORS

        Added espa_web/auth_backends.py for EE authentication 
            - Django auth system plugin
            - must include in settings.py AUTHENTICATION_BACKENDS


Version 2.2.5 (March 11, 2014)
        Made change to support variable cloud pixel threshold for calls
        to cfmask.

Version 2.2.4 (December 20, 2013)
        Bug fix to parse_hdf_subdatasets() in cdr_ecv.py to correct
        reprojection failures.

Version 2.2.3
        Altered cdr_ecv to use straight FTP rather than SCP (performance)

Version 2.2.2
        Added SCA & SWE to internal ordering
        Corrected unchecked reprojection errors
        Corrected Bulk Ordering Navigation
        Fail job submission if no products selected
        Added MSAVI to spectral indices
        Merged all vegetation indices to a single raster
        Removed standalone CFMask option
        Removed solr index generation option
        Corrected cross browser rendering issues

Version 2.2.1
        Fixed bug reprojecting spectral indices
        Fixed bug reprojecting cfmask
        Modified reprojected filename extensions to '.tif'
        Defaulted pixel size to 30 meters or dd equivalent when necessary.
        Corrected Albers subsetting failures.
        Modified code to properly clean up HDFs when reprojecting/reformatting
        to GeoTiff
        Optimized calls to warp outputs in the cdr_ecv.
        Added first set of prototype system level tests for cdr_ecv.py

Version 2.2.0
        Added code to perform reprojection.
        Added code to perform framing.
        Added code to perform subsetting.

Version 2.1.6
        Reorganized the codebase to accomodate LPVS code.
        Implemented a standardized logging format.
        Added webpages for marketing.

Version 2.1.5
        Added a lockfile to the scene_cache update to prevent multiple
        processes from running, which causes a parallel contention issue

Version 2.1.4
        Bug fix for duplicate orders from EE.  Modified core.py
        load_orders_from_ee() to look for an order + scene in the db before
        attempting to add a new one

Version 2.1.3
        Fix to clear logs upon non-error status update.
        Initial version of api.py added.
        Integration for SI 1.1.2, cfmask 1.1.1, ledaps 1.3.0,
        cfmask_append 1.0.1, swe 1.0.0
        Updated code to interact with online cache over private 10Gig network
        Added load balancing code mapper.py for contacting the online cache

Version 2.1.2
        Updates for new cfmask and appending cfmask to sr outputs.
        Multiple bug fixes related to UI css files when viewed of https
Version 2.1.1
        Update internal email verbage to remove "Surface Reflectance"
        Bug fixes for LTA integration when processing mixed orders
        Fixed product file names
        Corrected bug where espa user interface wouldn't accept a Glovis
        scenelist
        Added all available spectral indices to espa_internal
        Integrated cfmask 1.0.4
        Integrated updated Spectral Indices

Version 2.1
        Updated web code to verify submitted scenes against the inventory
        before accepting them.
        Updated web code to pull orders from EE on demand for surface
        reflectance only.

Version 2.0.4
        Update espa to work with new ledaps and cfmask
        Corrected issue with bad tarballs being distributed.
        Generated cksum files and distribute those with the product.

Version 2.0.3
        Updated espa to work with ledaps 1.2 & cfmask 1.0.1
        Added ability for all users to select product options.
        Added ability to order cfmask for espa_internal and espa_admin
        Removed seperate source file distribution (can be selected as an option)

Version 2.0.2
        Fixed solr generation to account for landsat metadata field name
        changes
        Changed espa TRAM ordering priority
        Tested setting ESPA work directory to "." or cwd()

Version 2.0.1
        Added espa_internal for evaluation.

Version 2.0.0
        Modified architecture to remove the Chain of Command structure from
        processing.
        Modified espa to work with ledaps 1.1.2.
        Added ability for end users to select which products they want in
        their delivery (admin acct only at this time.)
        Added mapper.py and removed any external calls from espa.py except for
        scp calls to obtain data.  Mapper.py handles
        all status notifications to the espa tracking node now.

Version 1.3.8
        Updated espa to work with ledaps 1.1.1.

Version 1.3.7
        Updated espa to work with ledaps 1.1.0.  Added scene_cache.py to speed up scene submission.
        Multiple bugfixes and cosmetic website/email changes.
Version 1.3.5
        Updated 'lndpm' binary to work with new Landsat metdata

        Corrected metadata file lookup to account for Landsat's metadata format
        changes.
Version 1.3.4
        Corrections to reprojection code in espacollection.py when creating
        browse images.
Version 1.3.3
        Modifications to the NDVI code to release memory more quickly, reduce
        memory footprint.
Version 1.3.2
        Correction to the solr index to be semicolon seperated

Version 1.3.1
        Bug fixes for NDVI and CleanupDirs

Version 1.3
        Updated LEDAPS to the latest version from November 2011.  Added ability to generate NDVI for output products
        Multiple additions to collection processing to create browse and solr index.

Version 1.2
        Replaced Python's tar and untar commands in espa.py with the native operating system commands.
        Prior to this there were intermittent corrupt archives making it into the distribution node.
        Native os commands are also far faster than the python implementation.

Version 1.1
        Added RSS Status capability to ordering interface

Version 1.0
        First major stable version of ESPA released that will process Landsat L1T's to Surface Reflectance

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Python 93.9%
  • HTML 3.4%
  • JavaScript 1.4%
  • CSS 1.2%
  • Other 0.1%