The DATA Act Broker repository is the API, which communicates to the web front end.

For instructions on contributing to this project or running your own copy of the DATA Act broker, please refer to the documentation in the DATA Act core responsitory.

The repository has two major directories: scripts and handlers.

dataactbroker/
├── scripts/        (Install and setup scripts)
└── handlers/       (Route handlers)

The /dataactbroker/scripts folder contains the install scripts needed to setup the broker API for a local install. For complete instructions on running your own copy of the API and other DATA Act broker components, please refer to the documentation in the DATA Act core responsitory.

The dataactbroker\handlers folder contains the logic to handle requests that are dispatched from the loginRoutes.py, fileRoutes.py, and 'userRoutes.py' files. Routes defined in these files may include the @permissions_check tag to the route definition. This tag adds a wrapper that checks if there exists a session for the current user and if the user is logged in, as well as checking the user's permissions to determine if the user has access to this route. If user is not logged in to the system or does not have access to the route, a 401 HTTP error will be returned. This tag is defined in dataactbroker/permissions.py. Cookies are used to keep track of sessions for the end user. Only a UUID is stored in the cookie.

accountHandler.py contains the functions to check logins and to log users out.

fileHandler.py contains functions for managing user file interaction. It creates all of the jobs that are part of the user submission and has query methods to get the status of a submission. In addition, this class creates downloadable links to error reports created by the DATA Act Validator.

In addition to these helper objects, the following sub classes also exist within the directory: UserHandler, JobHandler, ErrorHandler, and 'InterfaceHolder'. These classes extend the database connection objects that are located in the Core Repository. Extra query methods exist in these classes that are used exclusively by the Broker API.

In general, status codes returned are as follows:

• 200 if successful
• 400 if the request is malformed
• 401 if the username or password are incorrect, or the session has expired
• 500 for server-side errors

This route confirms that the broker is running

Example input:

None

Example output:

"Broker is running"

This route checks the username and password against a credentials file. Accepts input as json or form-urlencoded, with keys "username" and "password".

Example input:

{
    "username": "user",
    "password": "pass"
}

Example output:

{
    "message": "Login successful",
    "user_id": 42,
    "name": "John",
    "title":"Developer",
    "agency": "Department of Labor",
    "permissions" : [0,1]
}

Logs the current user out, only the login route will be accessible until the next login. If not logged in, just stays logged out. Returns 200 in both cases.

Example input:

None

Example output:

{
    "message": "Logout successful"
}

Checks that the session is still valid. Returns 200, and JSON with key "status" containing True if the session exists, and False if it doesn't.

Example input:

None

Example output:

{
    "status": "True"
}

Gets the information of the current that is login to the system.

Example input:

None

Example output:

{
    "user_id": 42,
    "name": "John",
    "title":"Developer",
    "agency": "Department of Labor",
    "permissions" : [0,1]
}

Permissions for the DATA Act Broker are list based. Each integer in the list corresponds with a permission.

Registers a user with a confirmed email. A call to this route should have JSON or form-urlencoded with keys "email", "name", "agency", "title", and "password". If email does not match an email that has been confirmed, a 400 will be returned. This route can only be called after the confirm_email_token route. After a successful submission this route will require confirm_email_token to be called again.

Example input:

{
   "email":"user@agency.gov",
   "name":"user",
   "agency":"Data Act Agency",
   "title":"User Title",
   "password":"pass"
}

Example output:

{
  "message":"Registration successful"
}

Changes a user's status, used to approve or deny users. This route requires an admin user to be logged in. A call to this route should have JSON or form-urlencoded with keys "uid" and "new_status". For typical usage, "new_status" should be either "approved" or "denied".

Example input:

{
   "uid":"1234",
   "new_status":"approved"
}

Example output:

{
  "message":"Status change successful"
}

Create a new user and sends a confirmation email to their email address. A call to this route should have JSON or form-urlencoded with key "email".

Example input:

{
   "email":"user@agency.gov"
}

Example output:

{
  "message":"Email Sent"
}

Checks the token sent by email. If successful, updates the user to email_confirmed. A call to this route should have JSON or form-urlencoded with key "token". If the token is invalid a failure message is returned along with the error code. The email address will also be returned upon success.

Example input:

{
   "token":"longRandomString"
}

Success Example output:

{
  "errorCode":0,
  "message":"success",
  "email" : "emailAddress@email.com"
}

Failure Example output:

{
  "errorCode":3,
  "message":"Link already used"
}

The following is a table with all of the messages and error code

Checks the token sent by email for password reset. A call to this route should have JSON or form-urlencoded with key "token". If the token is invalid a failure message is returned along with the error code. The email address will also be returned upon success.

Example input:

{
   "token":"longRandomString"
}

Success Example output:

{
  "errorCode":0,
  "message":"success",
  "email" : "emailAddress@email.com"
}

Failure Example output:

{
  "errorCode":3,
  "message":"Link already used"
}

The following is a table with all of the messages and error code

List all users with specified status, typically used to review users that have applied for an account. Requires an admin login. A call to this route should have JSON or form-urlencoded with key "status".

Example input:

{
   "status":"awaiting_approval"
}

Example output:

{
  "users":[{"uid":1,"name":"user","email":"agency@user.gov","title":"User Title","agency":"Data Act Agency"},{"uid":2,"name":"user2","email":"","title":"","agency":""}]
}

List all submissions by currently logged in user.

Example input:

None

Example output:

{
  "submission_id_list":[1,2,3]
}

Change specified user's password to new value. User must have confirmed the token they received in same session to use this route. A call to this route should have JSON or form-urlencoded with keys "uid" and "password".

Example input:

{
   "token":"longRandomString"
}

Example output:

{
  "message":"Password successfully changed"
}

Remove current password and send password with token for reset. A call to this route should have JSON or form-urlencoded with key "email".

Example input:

{
   "email":"user@agency.gov"
}

Example output:

{
  "message":"Password reset"
}

This route confirms that the broker is running

Example input: None Example output: "Broker is running"

This path will return files located in the local folder. This path is only accessible for local installs due to security reasons.

Example Route /Users/serverdata/test.csv for example will return the test.csv if the local folder points to /Users/serverdata.

Input for this route should be a post form with the key of file where the uploaded file is located. This route only will return a success for local installs for security reasons. Upon successful upload, file path will be returned.

Example Output:

{
   "path": "/User/localuser/server/1234_filename.csv"
}

This route is used to retrieve S3 URLs to upload files. Data should be JSON with keys: ["appropriations", "award_financial", "award", "program_activity"], each with a filename as a value, and submission metadata keys: ["agency_name","reporting_period_start_date","reporting_period_end_date","existing_submission_id"]. If an existing submission ID is provided, all other keys are optional and any data provided will be used to correct information in the existing submission.

This route will also add jobs to the job tracker DB and return conflict free S3 URLs for uploading. Each key put in the request comes back with an url_key containing the S3 URL and a key_id containing the job id. A returning submission_id will also exist which acts as identifier for the submission.

A credentials object is also part of the returning request. This object provides temporarily access to upload S3 Files using an AWS SDK. It contains the following: SecretAccessKey, SessionToken, Expiration, and AccessKeyId. It is important to note that the role used to create the credentials should be limited to just S3 access.

When upload is complete, the finalize_submission route should be called with the job_id.

Example input:

{
  "appropriations":"appropriations.csv",
  "award_financial":"award_financial.csv",
  "award":"award.csv",
  "program_activity":"program_activity.csv",
  "agency_name":"Name of the agency",
  "reporting_period_start_date":"03/31/2016",
  "reporting_period_end_date":"03/31/2016",
  "existing_submission_id: 7 (leave out if not correcting an existing submission)
}

Example output:

{
  "submission_id": 12345,

  "bucket_name": "S3-bucket",

  "award_id": 100,
  "award_key": "2/1453474323_awards.csv",

  "appropriations_id": 101,
  "appropriations_key": "2/1453474324_appropriations.csv",

  "award_financial_id": 102,
  "award_financial_key": "2/1453474327_award_financial.csv",

  "program_activity_id": 103,
  "program_activity_key": "2/1453474333_program_activity.csv",

  "credentials": {
    "SecretAccessKey": "ABCDEFG",
    "SessionToken": "ABCDEFG",
    "Expiration": "2016-01-22T15:25:23Z",
    "AccessKeyId": "ABCDEFG"
  }
}

A call to this route should have JSON or form-urlencoded with a key of "upload_id" and value of the job id received from the submit_files route. This will change the status of the upload job to finished so that dependent jobs can be started.

Example input:

{
  "upload_id":3011
}  

Example output:

{
  "success": true
}

A call to this route should have JSON or form-urlencoded with a key of "submission_id" and value of the submission id received from the submit_files route. The response object will be JSON with keys of "job_X_error_url" for each job X that is part of the submission, and the value will be the signed URL of the error report on S3. Note that for failed jobs (i.e. file-level errors), no error reports will be created.

Example input:

{
   "submission_id":1610
}

Example output:

{
  "job_3012_error_url": "https...",
  "job_3006_error_url": "https...",
  "job_3010_error_url": "https...",
  "job_3008_error_url": "https...",
  "cross_file_error_url": "https..."
}

A call to this route will provide status information on all jobs associated with the specified submission. The request should have JSON or form-urlencoded with a key "submission_id". The response will contain a list of status objects for each job under the key "jobs", and other submission-level data.

Example input:

{
  "submission_id":1610
}

Example output:

{  
  "jobs": [
    {
    "job_id": 3005,
    "job_status": "invalid",
    "file_type": "appropriations",
    "job_type": "file_upload",
    "filename": "approp.csv",
    "file_status" : "header_error",
    "missing_headers": ["header_1", "header_2"],
    "duplicated_headers": ["header_3", "header_4"],
    "file_size": 4508,
    "number_of_rows": 500,
    "error_type": "header_error",
    "error_data": []
    },
    {
    "job_id": 3006,
    "job_status": "finished",
    "file_type": "appropriations",
    "job_type": "file_upload",
    "filename": "approp.csv",
    "file_status" : "complete",
    "missing_headers": [],
    "duplicated_headers": [],
    "file_size": 4508,
    "number_of_rows": 500,
    "error_type": "record_level_error",
    "error_data":  [
	{
	"field_name": "allocationtransferagencyid",
	"error_name": "type_error",
	"error_description": "The value provided was of the wrong type",
	"occurrences": 27,
	"rule_failed": ""
	},
	{
	"field_name": "availabilitytypecode",
	"error_name": "rule_failed",
	"error_description": "",
	"occurrences": 27,
	"rule_failed": "Failed rule: Indicator must be X, F, A, or blank"
	}
	]
    }
  ],
  "agency_name": "Name of the agency",
  "reporting_period_start_date": "03/31/2016",
  "reporting_period_end_date": "03/31/2016",
  "number_of_errors": 54,
  "number_of_rows": 446,
  "created_on": "04/01/2016"
}

Returns a signed URL to the current RSS file. Requires a logged in user.

Example output:

{
  "rss_url":"https://rssfilehere"
}

To run the broker API unit tests, navigate to the main project folder (data-act-broker) and type the following:

    $ python tests/runTests.py

Before running test cases, make sure the validator is running.

To generate a test coverage report from the command line:

1. Make sure you're in the main project folder (data-act-broker).
2. Run the tests using the coverage command: coverage run tests/runTests.py.
3. After the tests are done running, view the coverage report by typing coverage report. To exclude third-party libraries from the report, you can tell it to ignore the site-packages folder: coverage report --omit=*/site-packages*.