Please read the "Dependencies" chapter below.
Probespawner uses a number of great software and it depends directly on a number of them. Links are provided for every of them. Some packages, namely jar files, are available here for convenience but you should upgrade them should you use this project.

All the probespawner's source is public domain - see LICENSE.md file

Below you'll find the instructions to install and use Probespawner in a *NIX environment.
[Click here to check instruction on how to install and use in Windows] (https://github.com/filipealmeida/probespawner/blob/master/INSTALL.windows.md)

1. Download java (1.7+) - https://java.com/en/download/
2. Install java
3. Download Jython - http://www.jython.org/downloads.html
4. Have java on your path (1.7+)
5. Install Jython: java -jar jython-installer-2.7.0.jar -s -d targetDirectory
6. Grab probespawner from github or download it from where's available - https://github.com/filipealmeida/probespawner/
7. Expand the tarball/zip you’ve downloaded.
8. Enter probespawner's directory
9. Have jython on your path
10. Run ./probespawner.sh <YOURCONFIG.json> if in linux/mac or probespawner.bat <YOURCONFIG.json> if with windows

YOURCONFIG.json example here

Probespawner is a small jython program initially designed to repeat JDBC queries periodically and write it’s output to an Elasticsearch cluster.
Now it's kind of a crossbreed of a logshipper with crontable.
It can periodically perform JDBC queries, JMX queries and operations and/or command executions, outputting it's parsed data (usually as JSON) to Elasticsearch, RabbitMQ/AMQP queues, OpenTSDB, files and/or STDOUT.
It's no substitute of a log shipper but comes in handy and packs a number of interesting examples in jython on how to achieve just that.
Tough immature and not production ready, it's kind of easy to adapt/extend and it already has been real useful for monitoring and troubleshooting systems, databases and java applications (so far).

The simple answer is "just because".
Probespawner got written initally to perform some tasks that elasticsearch-river-jdbc feeder did not address and to come around the bugs and difficulties if setting up one such river/feeder (plus rivers are apparently now deprecated).
Other work extended from there to help troubleshooting, monitoring and performance statistics on the OS and applications.
See the examples folder for some practical uses.
An effort do document some of the things done using probespawner will be made but some are:

• Collect AWR from OracleDB, DMV data from Microsoft SQL Server and performance schema data from MySQL's.
  Index data on Elasticsearch. Insight through kibana.
  SQLServer DMVs example here.
• Collect netstat information periodically, send through RabbitMQ to Elasticsearch. D3JS to perform force directed graphs from the information with brush date/time interval selector. This animates the graph of the network conversations as you slide throught a time interval.
  Example here.
• Collect top information, ship through pipeline to Elasticsearch. Kibana dashboard allows for quick browse trough the processes history, correlate with machine resources, document blocking conditions and wait events.
  Example here.
• Collect stack traces periodically from application servers while monitoring resources of a JVM using JMXProbe. Data shipped through pipeline (RabbitMQ) made available for performance engineers, application testers, master troubleshooters and developers for the many reasons you might imagine.
  Example here.
• Collect vmstat info, write metrics to OpenTSDB via socket

Probespawner reads a JSON configuration file stating a list of inputs and the outputs, much like logstash. The inputs provided are either JMX (probing a JVM), JDBC (querying a database) or execution of programs in different platforms.
Each is called a probe.
The data acquired cyclically from these input sources are sent to Elasticsearch, RabbitMQ, OpenTSDB, stdout or file.

Basically, for each input you have defined, probespawner will launch a (java) thread as illustrated in the concurrency manual of jython.
Each thread is an instance of a probe that performs:

• Periodical acquisition of records from a database, writes these to an Elasticsearch cluster (using Elasticsearch’s JAVA api).
• Periodical acquisition of JMX attributes from a JVM instance, outputs to an index of your choice on your Elasticsearch cluster and to a file on your filesystem.
• Periodical top command parse, sends data to a RabbitMQ queue
• Send metrics data from command execution to OpenTSDB
• Periodically executes any task you designed for your own probe and do whatever you want with the results, for instance, write them to STDOUT.

1. Jython 2.5.3+ - http://www.jython.org/downloads.html
2. Jyson 1.0.2+ - http://opensource.xhaus.com/projects/jyson
3. JodaTime 2.7+ - https://github.com/JodaOrg/joda-time

1. Tomcat’s 7.0.9+ (connection pool classes) - http://tomcat.apache.org/download-70.cgi
2. Elasticsearch 1.5.0+ - https://www.elastic.co/downloads
3. RabbitMQ 3.5.3+ - https://www.rabbitmq.com

1. Mysql - http://dev.mysql.com/downloads/connector/j/
2. Oracle - http://www.oracle.com/technetwork/apps-tech/jdbc-112010-090769.html
3. MSSQL - http://www.microsoft.com/en-us/download/details.aspx?displaylang=en&id=11774, http://go.microsoft.com/fwlink/?LinkId=245496

About the use of Tomcat’s connection pool, zxJDBC could’ve been used to attain the same objective. Since some code was around using it, so it stood.

CLASSPATH=jyson-1.0.2.jar:joda-time-2.7.jar

CLASSPATH=jyson-1.0.2.jar:joda-time-2.7.jar:tomcat-jdbc.jar:tomcat-juli.jar

CLASSPATH=jyson-1.0.2.jar:joda-time-2.7.jar:tomcat-jdbc.jar:tomcat-juli.jar:mysql-connector-java-5.1.20-bin.jar

JSON input configuration:

...
"probemodule": { "module": "cooldbprobe", "name" : "DatabaseProbe" },
"url": "jdbc:mysql://localhost:3306/mysql",
"driverClassName": "com.mysql.jdbc.Driver",
"username": "root",
"password": "password",
...

CLASSPATH=jyson-1.0.2.jar:joda-time-2.7.jar:tomcat-jdbc.jar:tomcat-juli.jar:ojdbc6.jar

JSON input configuration:

...
"probemodule": { "module": "cooldbprobe", "name" : "DatabaseProbe" },
"url": "jdbc:mysql://localhost:3306/mysql",
"driverClassName": "com.mysql.jdbc.Driver",
"username": "root",
"password": "password",
...

CLASSPATH=jyson-1.0.2.jar:joda-time-2.7.jar:tomcat-jdbc.jar:tomcat-juli.jar:sqljdbc4.jar

CLASSPATH=compress-lzf-1.0.2.jar:elasticsearch-2.1.1.jar:guava-18.0.jar:hppc-0.7.1.jar:jackson-core-2.6.2.jar:jboss-client.jar:joda-time-2.7.jar:jsr166e-1.1.0.jar:jyson-1.0.2.jar:lucene-core-5.3.1.jar:mysql-connector-java-5.1.20-bin.jar:netty-3.10.5.Final.jar:ojdbc6.jar:rabbitmq-client.jar:sqljdbc4.jar:t-digest-3.0.jar:tomcat-jdbc.jar:tomcat-juli.jar:wlclient.jar:wljmxclient.jar

JSON input configuration:

...
"probemodule": { "module": "cooldbprobe", "name" : "DatabaseProbe" },
"url": "jdbc:sqlserver://suchhost:1433;databaseName=Master",
"driverClassName": "com.microsoft.sqlserver.jdbc.SQLServerDriver",
"username": "suchuser",
"password": "suchpassword",
...

CLASSPATH=jyson-1.0.2.jar:joda-time-2.7.jar:tomcat-jdbc.jar:tomcat-juli.jar:jboss-client.jar

JSON input configuration:

...
"url":"service:jmx:remoting-jmx://localhost:8000",
"username": "admin",
"password": "password",
"alias": "hostname.project",
"queries" : [
 {
  "object_name" : "java.lang:type=Memory",
  "attributes" : [ "NonHeapMemoryUsage", "HeapMemoryUsage" ]
 }, {
  "object_name" : "com.conceptwave.AVM:type=CWAVM"
 }, {
  "object_name" : "com.conceptwave.AVM:type=PEQueues,name=Participants Queue"
 }, {
  "object_name" : "java.lang:type=GarbageCollector,name=*",
  "attributes" : [ "CollectionCount", "CollectionTime" ]
 }
],
"description": "Obtains JMX metrics from JVM instance, stores on elasticsearch",
"probemodule": { "module": "jmxprobe", "name" : "JMXProbe" },
...

CLASSPATH=/foo/bar/opt/elasticsearch-2.1.1/lib/elasticsearch-2.1.1.jar:/foo/bar/opt/elasticsearch-2.1.1/lib/lucene-core-5.3.1.jar:/foo/bar/opt/elasticsearch-2.1.1/lib/guava-18.0.jar:/foo/bar/opt/elasticsearch-2.1.1/lib/jsr166e-1.1.0.jar:/foo/bar/opt/elasticsearch-2.1.1/lib/hppc-0.7.1.jar:/foo/bar/opt/elasticsearch-2.1.1/lib/jackson-core-2.6.2.jar:/foo/bar/opt/elasticsearch-2.1.1/lib/compress-lzf-1.0.2.jar:/foo/bar/opt/elasticsearch-2.1.1/lib/t-digest-3.0.jar:/foo/bar/opt/elasticsearch-2.1.1/lib/netty-3.10.5.Final.jar

JSON output configuration:

"class": "elasticsearch",
"outputmodule": { "module": "jelh2", "name" : "Elasticsearch" },

The package contains the following files:

1. dummyprobe.py - Do not be mistaken judging by the name that this is a dummy probe. It’s been a poor choice of name but this is immature code so it’s staying like that. This is the class from which the other inherit and it’s the real skeleton for probes.
2. cooldbprobe.py - Same as databaseprobe.py but uses JodaTime for time parameters, timestamp is in milliseconds, datetime strings are ISO8601 format looking like “2015-11-21T15:14:15.912+05:00”.
3. jelh.py - Jython’s Elasticsearch Little Helper, the module responsible for the bulk requests to elasticsearch.
4. jmxprobe.py - This probe executes JMX requests in a JVM and reports it’s results to the outputs configured in the JSON setup. NOTE: It discards bad objects/attributes in case of failure but keeps working if it happens. Adjust to your needs (find out the try/except in the “tick” method)
5. probespawner.ini - Logging configuration.
6. probespawner.py - It’s the croupier, it reads and delivers the configuration for the probes. Lastly, it instantiates all of them and wait’s for them to finish. It’s a simple threadpool but it would be great if it dealt with interrupts and failures from it’s workers.
7. probespawner.sh - Launches probespawner with it’s argument as configuration.
8. probespawner.bat - Launches probespawner with it’s argument as configuration in case you’re using Microsoft Windows.
9. testprobe.py - The real dummy probe, it merely states that is a test and sends a sample dictionary to the outputs configured.
10. example.json - Sample JSON configuration for a JDBC, a JMX input, any input. Click here for contents
11. zxdatabaseprobe.py - Same as databaseprobe.py but dispensing the use of Tomcat’s connection pool (uses zxJDBC)
12. databaseprobe.py - Amazingly, this does not depend from “dummyprobe.py”, it was the initial code of a more monolitic probe that existed in the past. This probe executes any given query in a database and reports it’s results (if any) to the outputs configured in the JSON setup. It's here for legacy reasons, you should ignore this probe.
13. execprobe.py - Executes “command” every cycle and reports it’s output
14. linuxtopprobe.py - Executes top command on linux boxes every cycle, parses and reports it’s output in an elasticsearch friendly fashion
15. netstats.py - Executes “netstat -s” command on linux boxes every cycle, parses and reports it’s output in an elasticsearch friendly fashion.
16. netstatntc.py - Executes “netstat -ntce” command on linux boxes every cycle, parses and reports it’s output in an elasticsearch friendly fashion.
17. rmqlh.py - Jython’s RabbitMQ Little Helper, the module responsible for pushing to RabbitMQ queues.
18. opentsdblh.py - Jython’s OpenTSDB (time-series database) Little Helper, the module responsible for pushing to such backend.
19. jelh2.py - Jython’s Elasticsearch 2.0 Little Helper, the module responsible for the bulk requests to elasticsearch versions 2+.

Instead of reading this section you can refer to the file example.json file in the repo/zip. There you'll find examples of most configurations.

Where omitted a description of a field it means it has no action.
The list of possible fields for inputs and outputs is shown below:

There are two possible modules, “cooldbprobe”, “databaseprobe” and “zxdatabaseprobe”
First two use Tomcat’s JDBC connection pool.

Parameters you can use for your prepared statements (? in the statement definition, see example.json).

cooldbprobe packs a few extras and has some differences from the above:

Now you need to assemble a JSON file indicating which inputs are to be spawned, what probes are to be launched. Hence the “input” field.
This field indicates probespawner which inputs are to be processed by the probe threads.

Refer to the file example.json.
There you will find an example with many combinations of the above options.
That should suffice for most everything you have in mind for recipes with probespawner.

Below a sample classpath for using probespawner and a few JDBC drivers: export CLASSPATH=/home/suchuser/opt/apache-tomcat-7.0.59/lib/tomcat-jdbc.jar:/home/suchuser/opt/apache-tomcat-7.0.59/bin/tomcat-juli.jar:/home/suchuser/var/lib/java/jyson-1.0.2/lib/jyson-1.0.2.jar:/home/suchuser/var/lib/java/mysql-connector-java-5.1.20-bin.jar:/home/suchuser/var/lib/java/sqljdbc_4.0/enu/sqljdbc4.jar:/home/suchuser/var/lib/java/sqljdbc_4.0/enu/sqljdbc.jar:/home/suchuser/opt/rabbitmq-java-client-bin-3.5.3/rabbitmq-client.jar

jython probespawner.py --configuration=example.json

This one calls “find” in the CWD and adds all *jar files to the classpath before calling jython.
probespawner.sh example.json

The batch file lists all jars in the “jars” directory and adds them to the classpath before calling jython.
probespawner.bat example.json

Just in case you want to develop a probe for your convenience, be warned: this is immature code with no support. Having said that, take a look at the “testprobe.py” file.

This is the minimum probe, the only thing it does is writing the { ‘test’: ‘yes’ } dictionary to the configured outputs in your JSON file. For that to be accomplished, your class must override the “tick” method which is called at every cycle, meaning every “interval” seconds you defined in your input.

The method “initialize” is another popular overridden one, called before the probe starts cycling. It’s usually in this method you get the configuration keys from the JSON file, set up and initialize your probe.

username = self.getInputProperty("username")
This grabs the “username” content of your defined input in the JSON file and returns it’s value (or None).