Vertigo is a distributed event processing framework built on the Vert.x application platform. Following a concept and structure similar to Storm, Vertigo allows real-time problems to be broken down into smaller tasks (as Vert.x verticles) and distributed across one or many Vert.x instances, managing communication between components in a predictable and reliable manner.

• Supports simple pipelines or complex networks of Vert.x modules/verticles, including remote procedure calls spanning multiple Vert.x verticle instances
• Supports message ack/fail/timeout mechanisms, providing data sources with feedback on the state of processing simple or complex message trees
• Monitors networks for failures and automatically reassigns/redeploys failed verticles and modules
• Can distribute verticle/module instances among a single Vert.x instance or across a cluster of Vert.x instances
• Network components can be written in any Vert.x supported language, with current integration for Java and Python
• Integrates seemlessly with existing Vert.x applications

Vertigo is not a replacement for Storm. Rather, Vertigo is a lightweight - albeit less reliable - alternative that is intended to be embedded in larger Vert.x applications.

A network is the representation of a collection of components - special Vert.x verticle implementations- and the connections between them. Put together, a network processes streams of data in real-time. Vertigo puts no limitations on network structures. Each component may be connected to zero or many other components, and circular relationships may be created as well. Networks are defined using the NetworkDefinition API and deployed using Via clusters.

This is an example of a complex network. Given a set of Vert.x verticles or modules, Vertigo uses a code-based representation of the network structure to define connections between network components, start component verticles or modules, monitor components, and manage communication between them in a fast, reliable manner.

See the complex network example

A component represents a single vertex in a Vertigo network graph. Each network may contain any number of components, and each component may have any number of instances running within the network (each of which may be assigned to different machines around a cluster). Within the context of Vert.x a component can be defined as a verticle that may receive messages from zero or many verticles and send messages to one or many verticles. What happens within the verticle depends entirely where they appear in a network graph and how the component is implemented. Fortunately, Vertigo provides several types of components for common tasks.

Feeders are components whose sole responsibility is to feed data to a network. Data generated by feeders may come from any source, and Vertigo provides a number of feeder implementations for integrating networks with a variety of Vert.x and other APIs.

Java

BasicFeeder feeder = vertigo.createBasicFeeder();
feeder.start(new Handler<AsyncResult<BasicFeeder>>() {
  public void handle(AsyncResult<BasicFeeder> result) {
    BasicFeeder feeder = result.result();

    feeder.setMaxQueueSize(1000);
    final JsonObject data = new JsonObject().putString("body", "Hello world!");

    feeder.feed(data, new Handler<AsyncResult<Void>>() {
      public void handle(AsyncResult<Void> result) {
        if (result.failed()) {
          // Processing failed. Re-emit the message.
          feeder.feed(data);
        }
        else {
          // Processing succeeded!
        }
      }
    });
  }
});

Python

feeder = vertigo.create_basic_feeder()

def start_handler(error, feeder):
  if not error:
    def ack_handler(error):
      if not error:
        pass # Message was successfully processed!
    feeder.feed({'body': 'Hello world!'}, ack_handler)

feeder.start(start_handler)

Worker components are the primary units of processing in Vertigo. Workers are designed to both receive input and emit output (though workers can do one or the other). The processes that are performed in between depend entirely on the implementation.

Java

final Worker worker = vertigo.createWorker();
worker.messageHandler(new Handler<JsonMessage>() {
  public void handle(JsonMessage message) {
    JsonArray words = message.getArray("words").size();
    if (words != null) {
      // Emit the number of words.
      worker.emit(new JsonObject().putNumber("count", words), message);
      worker.ack(message);
    }
    else {
      // This is an invalid message. Fail it.
      worker.fail(message);
    }
  }
}).start();

Python

worker = vertigo.create_worker()

@worker.message_handler
def message_handler(message):
  if message.body.has_key('words'):
    worker.emit({'count': len(message.body['words'])}, parent=message)
    worker.ack(message)
  else:
    worker.fail(message)

Note that the API provides additional reliability features - message acking. See how message acking works

Executors are components that execute part or all of a network essential as a remote procedure invocation. Data emitted from executors is tagged with a unique ID, and new messages received by the executor are correlated with past emissions.

Networks are defined in code using an object-oriented API. The network definition API supports defining networks of virtually any structure, and Vertigo puts no limitations on network topologies. Just as network components can be written in any Vert.x supported language, networks can be defined using any Vert.x language.

Some examples demonstrate how the network definition API works.

One of the primary responsibilities of Vertigo is managing communication between network components in a consistent, predictable, and reliable manner. Vertigo uses the Vert.x event bus for inter-component communication, but Vertigo also provides many reliability features on top of the event bus.

Network components communicate with each other directly over the event bus rather than being routed through a central message broker. When a network is created, Vertigo assigns unique addresses to each component (verticle) instance which that verticle uses to communicate with the verticles around it. Thus, each component instance essentially maintains a direct connection to its neighbors, ensuring fast messaging between them.

With each component instance maintaining its own unique event bus address, Vertigo needs a way to determine which component messages emitted from one component are dispatched to. Each component may indicate a grouping which determines how messages are distributed among multiple instances of the component. For instance, one component may need to receive all messages emitted to it, while another may be need to receive messages in a round-robin fashion. Vertigo provides groupings for various scenarios, including consistent-hashing based groupings. Custom component groupings may also be provided.

Vertigo messages contain metadata in addition to the message body. And just as with grouping component instances, sometimes components may be only interested in receiving messages containing specific metadata. For this, Vertigo provides message filters which allow components to define the types of messages they receive. As with groupings, custom filters may be provided.

Vertigo's implicit control over the messaging infrastructure of component verticles allows it to provide some unique reliability features. When a Vertigo network is deployed, a special verticle called the auditor verticle is deployed along with it. The Vertigo auditor is tasked with monitoring messages within the network, tracking acks and nacks throughout the network, and notifying feeders when a message tree fails.

When a feeder component creates and emits a new message, the feeder's OutputCollector will notify the network's auditor of the newly created message. It is the responsibility of each worker component to ack or fail the message. If a worker creates a child of the message, the auditor is told about the new relationship. If a descendant of a message is failed, the original feeder will be notified of the failure. The feeder may optionally replay failed messages through the network. Once all descendants of a message have been successfully acked, the original feeder will be notified. The auditor maintains a timer for each message tree (the timers are actually shared among several message trees). If the timer is triggered prior to the entire message tree being acked, the tree will be failed and the original feeder will be notified.

Vertigo provides APIs for deploying networks both locally (within a single Vert.x instance) as well as distributing network components across multiple Vert.x instances using Via, a distributed deployment framework for Vert.x (Via was specifically developed for Vertigo).

final Cluster cluster = new LocalCluster(vertx, container);
cluster.deploy(network, new Handler<AsyncResult<NetworkContext>>() {
  public void handle(AsyncResult<NetworkContext> result) {
    if (result.succeeded()) {
      NetworkContext context = result.result();
      cluster.shutdown(context);
    }
  }
});