The aim of this project is to provide unified service modeling, management and visualization platform agnostic of delivery method. It creates virtual representations of any software services or physical resources and allows control over their entire life-cycle. The name of project comes from Architect program in Matrix movie series:

In the Matrix the Architect is a highly specialized, humorless program of the machine world as well as the creator of the Matrix. As the chief administrator of the system, he is possibly a collective manifestation, or at the very least a virtual representation of the entire Machine mainframe.

The Architect service consists of several core compontents:

Inventory Component

Inventory is the Architect's metadata engine. It encapsulates and unifies data from various metadata sources to provide inventory/metadata for various orchestration services.

Manager Component

Manager is the Architect's orchestration engine. The aim of this module is to enforce infrastructure topologies models and acquire live infrastructure topology data from any resource provider for further relational and quantitative analysis and visualisations.

Monitor Component

The structure of infrastructure resources is directed graph that can be subject for further analysis. We can perform several transformation functions on this graph data in Monitor component.

Following figure shows high-level achitecture of Architect system.

Installation instructions, getting started guides, and API documentation.

https://architect-api.readthedocs.io/

A quick summary of properties, capabilities and integrations of each component.

Inventory is the Architect's metadata engine. It encapsulates and unifies data from various metadata sources to provide inventory/metadata for various orchestration services. Basically serves as metadata proxy with clients. Currently supported metadata engines are:

• reclass (python3 version)
• salt-formulas

The following inventory providers are to be intergrated in near future.

• varstack
• hiera

There is a plan to integrate simple UI workflow (multi-step form wizards) defitions to simplify creation of complex service models.

Manager is the Architect's orchestration engine. The aim of this module is to enforce infrastructure topologies models and acquire live infrastructure topology data from any resource provider for further relational and quantitative analysis and visualisations.

The pull approach for querying endpoint APIs is supported at the moment, the processing push from target services is supported for SaltStack events. Currently supported resource providers are:

• Kubernetes clusters
• OpenStack clouds
• Amazon web services
• SaltStack infrastructures
• Terraform templates
• Jenkins pipelines

The following resource providers are to be intergrated in near future.

• GCE and Azure clouds
• Cloudify TOSCA blueprints
• JUJU templates

The structure of infrastructure resources is directed graph that can be subject for further analysis. We can perform several transformation functions on this graph data in Monitor component.

Currently supported relational analysis visualizations:

• Adjacency Matrix
• Arc Diagram
• Force-directed Layouts
• Hierarchical Edge Bundling
• Hive Plot
• Node-link Trees (Reingold-Tilford, Dendrograms)
• Partition Layouts (Sunburst, Icicle Diagrams, Treemaps)

Following steps show how to deploy various components of the Architect service and connections to external services.

The core service responsible for handling HTTP API requests and providing simple UI based on Material design. Release version of architect-api is currently available on Pypi, to install it, simply execute:

pip install architect-api

To bootstrap latest development version into local virtualenv, run following commands:

git clone git@github.com:cznewt/architect-api.git
cd architect-api
virtualenv venv
source venv/bin/activate
python setup.py install

You provide one configuration file for all service settings. The default location is /etc/architect/api.yaml.

Following configuration for individual inventories/managers/models can be stored in config files or in the database.

Each manager endpoint expects different configuration. Following samples show the required parameters to setup individual inventory backends.

Following configuration points to the reclass inventory storage on local filesystem.

class_dir: /srv/salt/reclass/classes
node_dir: /srv/salt/reclass/nodes
storage_type: yaml_fs
filter_keys:
  - _param

Each manager endpoint expects different configuration. Following samples show the required parameters to setup each endpoint type.

AWS manager uses boto3 high level AWS python SDK for accessing and manipulating with AWS resources.

region: us-west-2
aws_access_key_id: {{ access_key_id }}
aws_secret_access_key: {{ secret_access_key }}

Kubernetes requires some information from kubeconfig file. You provide the parameters of the cluster and the user to the manager. These can be found under corresponding keys.

scope: global
cluster:
  certificate-authority-data: |
    {{ ca-for-server-and-clients }}
  server: https://{{ kubernetes-api }}:443
user:
  client-certificate-data: |
    {{ client-cert-public }}
  client-key-data: |
    {{ client-cert-private }}

Configuration for keystone v2.0 and keystone v3 clouds. Configuration sample for single tenant access.

scope: local
region_name: RegionOne
compute_api_version: '2.1'
auth:
  username: {{ user-name }}
  password: {{ user-password }}
  project_name: {{ project-name }}
  domain_name: 'default'
  auth_url: https://{{ keystone-api }}:5000/v3

Config for managing resources of entire cloud, including hypervisors, tenants, etc in given region.

scope: global
region_name: RegionOne
auth:
  username: {{ admin-name }}
  password: {{ admin-password }}
  project_name: admin
  auth_url: https://{{ keystone-api }}:5000/v2.0

Configuration for manager connection to Salt API.

auth_url: http://{{ salt-api }}:8000
username: {{ user-name }}
password: {{ user-password }}

Following figure shows how SaltStack integrates with Architect Inventory and Manager. Please note that you can use Inventory integration independently of the Manager integration.

Configuration for parsing Hashicorp Terraform templates.

dir: ~/terraform/{{ terraform-dir }}

Following config snippets show configuration for supported types of visualization. Currently we support Network graphs, hierarchical structures for quantitative analysis.

The manager endpoint is used as source of relational data. The data can be sliced and diced as shown in the example.

name: Hive-plot
chart: hive
data_source:
  default:
    manager: openstack-project
    layout: graph
    filter_node_types:
    - os_server
    - os_key_pair
    - os_flavor
    - os_network
    - os_subnet
    - os_floating_ip
    - os_router
    filter_lone_nodes:
    - os_key_pair
    - os_flavor

The manager endpoint is used as source of relational data. This data can be traversed to create hierarchies. The hierarchical data has it's own family of visualization techniques.

name: Tree Structure (cluster > namespace > pod > service)
height: 1
chart: tree
data_source:
  default:
    manager: k8s-demo
    layout: hierarchy
    hierarchy_layers:
      0:
        name: Kubernetes Root
        kind:
      1:
        kind: k8s_namespace
      2:
        kind: k8s_pod
        target: in_k8s_namespace
      3:
        kind: k8s_service
        target: in_k8s_pod

Managers that do not expose any form of API can be controlled locally by using architect-adapter worker that wrap the local orchestration engine (Ansible, Cloudify, TerraForm).

To enable Salt Master inventory, you need to install http_architect Pillar and Top modules and add following to the Salt Master configuration files.

http_architect: &http_architect
  project: local-salt
  host: architect.service.host
  port: 8181

ext_pillar:
  - http_architect: *http_architect

master_tops:
  http_architect: *http_architect

You can control salt master infrastructure and get the status of managed hosts and resources. The Salt engine architect relays the state outputs of individual state runs and architect runners and modules provide the capabilities to interface with salt and architect functions. The Salt Master is managed through it's HTTP API service.

http_architect: &http_architect
  project: newt.work
  host: 127.0.0.1
  port: 8181

Different data require different diagram visualization. Diagrams are symbolic representation of information according to some visualization technique. Every time you need to emphasise different qualities of displayed resources you can choose from several layouts to display the data.

For most of the cases we will be dealing with network data that do not have any single root or beginning.

Force-directed graph drawing algorithms are used for drawing graphs in an aesthetically pleasing way. Their purpose is to position the nodes of a graph in two-dimensional or three-dimensional space so that all the edges are of more or less equal length and there are as few crossing edges as possible, by assigning forces among the set of edges and the set of nodes, based on their relative positions, and then using these forces either to simulate the motion of the edges and nodes or to minimize their energy.

The hive plot is a visualization method for drawing networks. Nodes are mapped to and positioned on radially distributed linear axes — this mapping is based on network structural properties. Edges are drawn as curved links. Simple and interpretable.

An arc diagram is a style of graph drawing, in which the vertices of a graph are placed along a line in the Euclidean plane, with edges being drawn as semicircles in one of the two halfplanes bounded by the line, or as smooth curves formed by sequences of semicircles. In some cases, line segments of the line itself are also allowed as edges, as long as they connect only vertices that are consecutive along the line.

An adjacency matrix is a square matrix used to represent a finite graph. The elements of the matrix indicate whether pairs of vertices are adjacent or not in the graph.

Danny Holten presents an aesthetically pleasing way of simplifying graphs and making tree graphs more accessible. What makes his project so useful, however, is how he outlines the particular thought process that goes into making a visualization.

Directed graph traversal can give os acyclic structures suitable for showing parent-child relations in your subraphs.

Dendrograms are node-link diagrams that place leaf nodes of the tree at the same depth. Dendograms are typically less compact than tidy trees, but are useful when all the leaves should be at the same level, such as for hierarchical clustering or phylogenetic tree diagrams.

The partition layout produces adjacency diagrams: a space-filling variant of a node-link tree diagram. Rather than drawing a link between parent and child in the hierarchy, nodes are drawn as solid areas (either arcs or rectangles), and their placement relative to other nodes reveals their position in the hierarchy. The size of the nodes encodes a quantitative dimension that would be difficult to show in a node-link diagram.

We display resources as circles with lower-level resources as inner circles. Circle packing in a circle is a two-dimensional packing problem with the objective of packing unit circles into the smallest possible larger circle.

A directed acyclic graph (DAG), is a finite directed graph with no directed cycles. That is, it consists of finitely many vertices and edges, with each edge directed from one vertex to another, such that there is no way to start at any vertex v and follow a consistently-directed sequence of edges that eventually loops back to v again. Equivalently, a DAG is a directed graph that has a topological ordering, a sequence of the vertices such that every edge is directed from earlier to later in the sequence.

Layered graph drawing or hierarchical graph drawing is a type of graph drawing in which the vertices of a directed graph are drawn in horizontal rows or layers with the edges generally directed downwards. It is also known as Sugiyama-style graph drawing after Kozo Sugiyama, who first developed this drawing style.