The next generation cloud-based science data service platform
- Follow instructions for installing nexusproto
- Follow instructions for installing data-access
- Follow instructions for setting up nexus-ingest
- Ingest some data
- Follow instructions for running analysis
- Install and configure Apache Zookeeper 3.4.x
- Create chroots
solr
,xd
, andkafka
-
Install and configure Apache Solr Cloud 5.3.x
-
Download JTS Topology Suite v1.13 and extract the zip.
-
From the exploded JTS zip, copy
$JTS_ZIP/lib/jts-1.13.jar
and$JTS_ZIP/lib/jtsio-1.13.jar
into$SOLR_INSTALL_DIR/server/lib/ext
on all Solr nodes. -
Configure Solr Cloud to use the
/solr
chroot of zookeeper -
On one of the Solr cloud nodes, upload the
nexustiles
configuration (located in data-access/config/schemas/solr) as a configset./zkcli.sh -cmd upconfig -z $ZK_SERVERS/solr -confname nexustiles -confdir /path/to/nexustiles/conf
-
Create a new collection with the name nexustiles. Use the nexustiles configset uploaded previously.
curl "http://<SOLR_HOST>/solr/admin/collections?action=CREATE&name=nexustiles&collection.configName=nexustiles"
-
Repeat steps 5 & 6 for the datasets collection.
- Install and configure Apache Cassandra 2.2.x
- Execute the DDL located in nexustiles.cql
- Install and run HSQLDB 2.3.x
- Install and run Redis 3.0.x
- Install and configure Apache Kafka 2.11-0.9.0.1
- Configure Kafka to use the
/kafka
chroot of zookeeper
- Install Spring XD 1.3.1.RELEASE
- Optionally install Flo for Spring XD
- Configure to use previously installed Apache Kafka as messaging bus
- Configure to use previously installed Redis for analytics
- Configure to use previously installed HSQLDB for Job Repository
- Configure to use previously installed Apache Zookeeper using the
xd
chroot - Choose and configure a location for the custom module registry
- Create a directory called
none
in$SPRING_XD_HOME/xd/lib
In order for streams to use the custom python and groovy scripts found in nexus-ingest, the machines that will be running Spring XD containers need to have some software installed on them.
-
Install Anaconda 4.0.0 with Python 2.7
-
Create an Anaconda environment
conda create --name nexus-xd-python-modules python
-
Install conda dependencies
conda install libnetcdf conda install netcdf4 conda install numpy
-
Install nexusproto
-
Install spring-xd-python
-
Install nexus-xd-python-modules
- Install Java 8 JDK. Either Oracle or OpenJDK.
-
The nexus-messages JAR needs to be available via Maven. There are a number of ways to do this. One way is to explicitly install the dependency into the local maven cache on each XD Container node.
mvn install:install-file -DpomFile=nexus-messages-VERSION.xml -Dfile=nexus-messages-VERSION.jar
- Place the groovy-scripts in a location that will be accessible to all XD Container nodes
- Upload the dataset-tiler custom module
- Upload the tcpshell custom module
- Upload the nexus-sink custom module