Command line tools for interacting with your Jut instance.

• Requirements
• Installation
• Upgrading
• Config Command
  • First time configuration
  • Using multiple configurations
  • Removing configurations
• Jobs Command
  • Check which jobs are running
  • Kill a running job
  • Connect to a persisten job * Run Command
  • Getting JSON data out of Jut
  • Getting a list of things out of Jut
  • Reconstructing log lines from all your hosts with Jut
  • Getting a desktop notification with Jut
• Upload Command
  • Upload a JSON file
  • Uploading a directory of JSON files
• Programs Command
  • Pull all your programs
• Development
  • Running Tests
  • Running a specific test

• Python 2.7.9+

We ask for 2.7.9+ because otherwise you may run into the following security message:

InsecurePlatformWarning: A true SSLContext object is not available. 
This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. 
For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.

Installation is as easy as**(*)**:

pip install jut-tools

or

easy_install jut-tools

That will get you the latest official build from pypi but if you want to use the bleeding master code you can also use:

(*) You may need to use sudo with the above commands if you are not using a virtualenv or don't have a userspace python installation.

If attempting to upgrade to the latest and greatest build just add the -U option like so:

pip install -U jut-tools

or

easy_install -U jut-tools

Once you've installed the jut python tools you should immediately run:

jut config add

in order to add your first configuration. We will save the client_id and client_secret for that configuration under ~/.jut/ and will generate access tokens as needed. You can add multiple user accounts to your jut-tools local setup so that you can easily swap between them when doing things on the command line.

If you have configured multiple configurations (ie jut config add) you can easily switch between them with a simple:

jut config defaults -u username

Removing a configuration is also very straightforward with:

jut config rm

And then follow the prompts to choose the correct configuration to remove.

To see the list of jobs currently running:

jut jobs list

You'll need to run jut jobs list to get the job id or possibly have a job id from persisting a job through the jut run command at which point you can kill the running job like so:

jut jobs kill [your_job_id]

When you start a persistent job you may find yourself wondering if its hitting any errors or warnings and in order to check on it you can use the jut jobs connect command to reconnect to a running job and snoop on the output.

Lets say you were running a persistent program running

jut run "read -type 'metric' name='cpu.idle' | reduce -every :hour: value=avg(value) | put name='cpu.idle.per.hour' | write -space 'collectd_rollups'" -p
dcee7afa

Notice we used the -p argument so that to persist the program and leave it running. This specific program is calculating 5 minute average rollups for cpu.idle metrics and writing them to our collectd.rollup space. Now lets say we were wondering if the job with the id dcee7afa was writing out any warnings or errors ? Well using the following we could snoop on the output like so:

jut jobs connect dcee7afa

And you'd now start seeing any error or warning messages. You won't see any data points since the program ended with a write sink. If you really wanted to see points then you'd have to include a little debug trick like so in your program:

read -type 'metric' name='cpu.idle' | reduce -every :hour: value=avg(value) | put name='cpu.idle.per.hour' | ( write -space 'collectd_rollups' ; pass)

So your program would end with a splitting of the stream where one of them goes into a write while the other just passes through and pushes points back to a client that may be listening for data.

jut also allows you to easily run juttle from the command line and use the results with the various other CLI tools you're use to using. Here are some example uses:

Let's see how many points you ingested into the 'default' space in the last 5 minutes:

jut run "read -last :5 minutes: -space 'jut_internal' name='import.success' AND import_space='default' | reduce sum(value)"

Output:

[
{
    "sum": 24758
}
]

List of host names that have reported metrics to the 'collectd' space in the past minute

jut run -f text "read -space 'collectd' -last :1 minutes: | reduce count() by host | keep host | uniq host | sort host"

Output:

browses3
pypi
qa-selenium-node1
qa-selenium-node2
qa-selenium-node3
qa-selenium-node4
qa-selenium-node5
qa-selenium-node6
qa-selenium-node7
qa-selenium-node8
sauce0

With the following you juttle you could easily put together all error log lines in order from all of the systems currently sending their log lines to your Jut instance:

jut run -f text "read -type 'event' -last :15 minutes: 'error' | keep time, message"

May look something like this:

2015-10-02T01:04:40.807Z _make_request error  getaddrinfo ENOTFOUND
2015-10-02T01:04:42.051Z Error obtaining access token: getaddrinfo ENOTFOUND
2015-10-02T01:04:48.413Z Error obtaining access token: getaddrinfo ENOTFOUND
2015-10-02T01:04:50.868Z ERROR [main] 2015-10-02 01:00:56,860 CassandraDaemon.java:202 - Directory /opt/jut/data/cassandra/commit_logs doesn't exist
2015-10-02T01:04:50.868Z ERROR [main] 2015-10-02 01:00:56,836 CassandraDaemon.java:202 - Directory /opt/jut/data/cassandra/data doesn't exist
2015-10-02T01:04:50.868Z ERROR [main] 2015-10-02 01:00:56,861 CassandraDaemon.java:202 - Directory /opt/jut/data/cassandra/cache doesn't exist
2015-10-02T01:04:50.870Z [2015-10-02 01:02:26,268][INFO ][cluster.metadata         ] [node-6b6002a5-8462-48de-a93d-c24dd0ffcb98] [events-jut_internal@2015.10.02] creating index, cause [auto(bulk api)], templates [schema-template-iis, schema-template-jira, schema-template-log4j, schema-template-github_push, schema-template-event, schema-template-jut_server, schema-template-github_commit, schema-template-jut_job_event, schema-template-jut_event, schema-template-cisco, schema-template-linux_logfile, schema-template-pagerduty, schema-template-ruby_logger, schema-template-jira_change, schema-template-syslog, schema-template-web_access, schema-template-web_error], shards [1]/[0], mappings [schema-iis, _default_, schema-jut_server, schema-syslog, schema-pagerduty, schema-jira_change, schema-jut_event, schema-jut_job_event, schema-log4j, schema-cisco, schema-linux_logfile, schema-event, schema-jira, schema-web_error, schema-github_commit, schema-web_access, schema-github_push, schema-ruby_logger]
2015-10-02T01:04:54.061Z Error obtaining access token: getaddrinfo ENOTFOUND
2015-10-02T01:05:00.056Z Error obtaining access token: getaddrinfo ENOTFOUND
2015-10-02T01:05:07.045Z _make_request error  getaddrinfo ENOTFOUND
2015-10-02T01:05:07.045Z Error obtaining access token: getaddrinfo ENOTFOUND
2015-10-02T01:05:10.702Z _make_request error  getaddrinfo ENOTFOUND
2015-10-02T01:05:10.702Z Error obtaining access token: getaddrinfo ENOTFOUND
2015-10-02T01:05:10.808Z Error obtaining access token: getaddrinfo ENOTFOUND
2015-10-02T01:05:10.808Z _make_request error  getaddrinfo ENOTFOUND
2015-10-02T01:05:11.708Z Error obtaining access token: getaddrinfo ENOTFOUND
2015-10-02T01:05:18.453Z Error obtaining access token: getaddrinfo ENOTFOUND
2015-10-02T01:05:18.960Z Error obtaining access token: getaddrinfo ENOTFOUND
2015-10-02T01:05:24.569Z (22846) error during import: Error: All connections on all I/O threads are busy
2015-10-02T01:05:24.570Z (22846) request 127.0.0.1 "POST /api/v1/import/default HTTP/1.1" 503 71 "-" "-" 110.375 ms
2015-10-02T01:05:24.571Z (22922) _make_request error  Received status code 503 from server.

To get a desktop notification when any collectd CPU usage is higher than 90% for over a minute, you could use something like so:

jut run "read -type 'metric' name='cpu.idle' | put value=100-value | reduce -every :1 minute: value=avg(value) | filter value > 90 | keep time, host, value"

Now the program is starting to get a little long so it may be best to simply stick the program in a file like so:

read -type 'metric' name='cpu.idle'
| put value=100-value
| reduce -every :1 minute: value=avg(value)
| filter value > 90
| put message="${time}: CPU usage on host ${host} over 90% at ${value} for the past minute"
| keep message

Lets call it collectd_cpu_alert.juttle and then you can simply run this juttle like so:

jut run examples/collectd_cpu_alert.juttle 

And with a desktop notification:

Try this first to test out the notification displays correctly

• on linux:

  jut run -f text "emit -limit 1 | put message='Hello World'" | xargs -I {} notify-send 'Testing' '{}'
  

• on mac:

  jut run -f text "emit -limit 1 | put message='Hello World'" | xargs -I {} osascript -e 'display notification "%" with title "Testing"'
  

Then you can run the high CPU usage juttle like so:

• on linux:

  jut run -f text examples/collectd_cpu_alert.juttle | xargs notify-send "High CPU Usage" %
  

• on mac:

  jut run -f text examples/collectd_cpu_alert.juttle | xargs osascript -e 'display notification "%" with title "High CPU Usage"' 
  

To upload a JSON file containing multiple data points just use jut like so:

jut upload data.json [-space some-other-space]

The above will simply upload the data points contained in the JSON file to the webhook connector listening on the default configuration.

You can also pipe JSON data to jut and it will handle uploading points just fine:

curl 'https://api.github.com/repos/nodejs/node/issues?page=1&per_page=100&state=all' -o issues1.json | \
jut upload

Would upload those last 100 issues returned from the github issues API for the node project.

Using --dry-run you can easily see what data you would have actually uploaded and also how using additional options affects your data. Take for example the following simple JSON data:

[
  {
    "sha": "6908574bc2a0511084ee76f9e411bb9d",
    "name": "Rod The Dog",
    "email": "rod@jut.io",
    "date": "2015-08-28T06:24:29Z",
    "message": "Oh this is not going go well..."
  }
]

You could simply use --dry-run to see what the data would look like:

> jut upload --dry-run                                                                        
POST: [
    {
        "date": "2015-08-28T06:24:29Z", 
        "sha": "6908574bc2a0511084ee76f9e411bb9d", 
        "message": "Oh this is not going go well...", 
        "name": "Rod The Dog", 
        "email": "rod@jut.io"
    }
]

Looking at that we realized we wanted to do a few things to our data points:

1. remove the email field
2. anonymize the name field
3. rename date to the required time field

Luckily jut handles all of this with ease:

> jut upload data.json --dry-run \
  --remove-fields email --anonymize-fields name --rename-fields date=time
POST: [
    {
        "sha": "6908574bc2a0511084ee76f9e411bb9d", 
        "message": "Oh this is not going go well...", 
        "name": "0c33366cb1abf53c9200930210024b79", 
        "time": "2015-08-28T06:24:29Z"
    }
]

Removing the --dry-run and you'll push your data up to Jut in a jiffy.

Instead of building this type of feature into jut-tools we felt it was easier for you to use your trusty command line tools like so:

for I in `ls *.json`
do
    jut upload $I
done

This is quite simple and can be achieved by pulling all of your programs into a single directory like so (make sure to use admin user to pull ALL of your programs, otherwise you'll only pull your users programs):

jut programs pull local_directory --all

Which would dump each program into its own file with the name of the program followed by the .juttle. extension.

If on the other hand you wanted to break up the programs per user into their own directory then supply the --per-user-directory like so:

jut programs pull local_directory --all --per-user-directory

When developing there are some unittests that can be used to verify your changes have not broken existing functionality as well as for you to add more tests to verify your changes. To run these you need to supply a username/password combo that has admin access to a real deployment. You can run it like so:

JUT_USER=username JUT_PASS=password python setup.py test

You can run just a test by name like so:

JUT_USER=username JUT_PASS=password python -m unittest tests.jut_upload_tests.JutUploadTests.test_jut_upload_to_url

Would run that one test within the upload tests called test_jut_upload_to_url and if you wanted to run just the upload suite of tests then the invocation would look like so:

JUT_USER=username JUT_PASS=password python -m unittest tests.jut_upload_tests