Installation and Provisioning of servers for Juniper Contrail Virtual Network System
====================================================================================
This document describe a mechanism to install and provision a set of servers that work in the Juniper Contrail Virtual Network System. The main tasks are

* Define the testbed by creating a file at /opt/contrail/utils/fabfile/testbeds/testbed.py
on the server that works as config node.
* Execute [fabric](http://www.fabfile.org) tasks to install and provision the system.

The sections below explain how to install and provision

* A typical testbed in which servers have a single interface and where all the user operations are initiated from the server behaving as a config node. 
* A testbed in which servers have more than one interface and the different interfaces are used for management(of physical host), control(communication between contrail services) and data(from Guest VM) traffic.
* A testbed in which servers have more than one interface and the interfaces are bonded.


Testbeds with single interface
------------------------------
### Step 1: Define the testbed.py file
	
The testbed file contains the following information

* Credentials to access the servers in the system
* Roles assigned to servers

A sample testbed definition consisting of a single server is at:

    /opt/contrail/utils/fabfile/testbeds/testbed_singlebox_example.py
    
A sample testbed definition consisting of multiple servers is at:

    /opt/contrail/utils/fabfile/testbeds/testbed_multibox_example.py
    
Copy one of these files as testbed.py fill in the ip-addresses and root credentials for the servers.

### Step 2: Copy the contrail_install_packages to other servers(in multiple server setups):
On the server running config node:

    cd /opt/contrail/utils
    fab install_pkg_all:/path/to/contrail-install-packages.rpm | /path/to/contrail-install-packages.deb
    
### Step 3: Install, Provision and start the services
The following commands are required on the server running config node:

    cd /opt/contrail/utils
    fab upgrade_kernel_all  #Only in case ofr Ubuntu to upgrade kernal(Optional)
    <all nodes will reboot>
    fab -c fabrc install_contrail
    <compute nodes will reboot>
    <relogin if config node was also running as compute node>
    cd /opt/contrail/utils
    fab -c fabrc setup_all
    <compute nodes will reboot>
    
Testbeds with separate interfaces for Management And Control + Data
-------------------------------------------------------------------
### Step 1: Define the testbed.py file
The testbed file contains the following information

* Credentials to access the servers in the system
* Roles assigned to servers
* Interface information on servers (which interface is used for management and which for control + data)

A sample testbed definition consisting of a single server is at:

    /opt/contrail/utils/fabfile/testbeds/testbed_singlebox_example.py
    
A sample testbed definition consisting of multiple servers is at:

    /opt/contrail/utils/fabfile/testbeds/testbed_multibox_example.py
    
Copy one of these files as testbed.py fill in the ip-addresses and root credentials for the servers.

### Step 2: Copy the contrail_install_packages to other servers:
On the server running config node:

    cd /opt/contrail/utils
    fab install_pkg_all:/path/to/contrail-install-packages.rpm | /path/to/contrail-install-packages.deb

### Step 3: Install Packages and fixup testbed.py
The following commands are required on the server running config node:

    cd /opt/contrail/utils
    fab upgrade_kernel_all  #Only in case ofr Ubuntu to upgrade kernal(Optional)
    <all nodes will reboot>
    fab -c fabrc install_contrail
    <Compute nodes will reboot>
    <Relogin if config node was also running as compute node>
    <Update testbed.py with renamed interface names in uncommented 'OPTIONAL SEPARATION OF MANAGEMENT AND CONTROL + DATA' section>

### Step 4: Provision and start the services
    cd /opt/contrail/utils
    fab -c fabrc setup_network
    fab -c fabrc setup_all
    <compute nodes will reboot>

Testbeds with multiple interfaces that are bonded
-------------------------------------------------
### Step 1: Define the testbed.py file
The testbed file contains the following information

* Credentials to access the servers in the system
* Roles assigned to servers
* Interface information on servers (which physical interfaces are bonded and the ip address on the bond interface)

A sample testbed definition consisting of a single server is at:

    /opt/contrail/utils/fabfile/testbeds/testbed_singlebox_example.py
    
A sample testbed definition consisting of multiple servers is at:

    /opt/contrail/utils/fabfile/testbeds/testbed_multibox_example.py
    
Copy one of these files as testbed.py fill in the ip-addresses and root credentials for the servers. 

### Step 2: Copy the contrail_install_packages to other servers:
On the server running config node:

    cd /opt/contrail/utils
    fab install_pkg_all:/path/to/contrail-install-packages.rpm | /path/to/contrail-install-packages.deb

### Step 3: Install Packages and fixup testbed.py
The following commands are required on the server running config node:

    cd /opt/contrail/utils
    fab upgrade_kernel_all  #Only in case ofr Ubuntu to upgrade kernal(Optional)
    <all nodes will reboot>
    fab -c fabrc install_contrail
    <Compute nodes will reboot>
    <Relogin if config node was also running as compute node>
    <Uncomment the 'OPTIONAL BONDING CONFIGURATION' section and fill in the device and ip address for the bonding>

### Step 4: Provision and start the services
    cd /opt/contrail/utils
    fab -c fabrc setup_network
    fab -c fabrc setup_all
    <compute nodes will reboot>
    

Upgrade single node contrail VNS
---------------------------------
### Step 1: Define the testbed.py file; this file will already exists
if the system is up and running. 
	
The testbed file contains the following information

* Credentials to access the servers in the system
* Roles assigned to servers

A sample testbed definition consisting of a single server is at:

    /opt/contrail/utils/fabfile/testbeds/testbed_singlebox_example.py
    
Copy one of these files as testbed.py fill in the ip-addresses and root credentials for the servers.

### Step 2: Copy the new contrail-install-packages.rpm | contrail-install-packages.deb to the config node:
Use SCP to do this
    yum localinstall contrail-install-packages.rpm | dpkg -i contrail-install-packages.deb
Run setup script,
    /opt/contrail/contrail_packages/setup.sh

    
### Step 3: Upgrade the contrail packages and restart the services
The following fab command upgrades the live system:

    cd /opt/contrail/utils
    fab upgrade_contrail:<base_release>,/path/to/contrail-install-packages.rpm | /path/to/contrail-install-packages.deb
        <base_release> can be (1.05 or 1.06) in case of ubuntu and (1.05) in case of centos.
Above fab command will reboot the node; so login again and issue the following command.
    cd /opt/contrail/utils
    fab restart_openstack_compute

### Step 4: Soft Reboot all the VM's through horizon dashbord.
    To reboot  all the VM's in all tenants following is the fab task.
    cd /opt/contrail/utils
    fab reboot_vm:mode=hard  #Default mode is soft reboot.

    To reboot any single VM following is the fab task
    cd /opt/contrail/utils
    fab reboot_vm:vmid=<uuid of a vm>mode=hard  #Default mode is soft reboot.


Upgrade multinode contrail VNS
------------------------------
### Step 1: Define the testbed.py file; this file will already exists
if the system is up and running. 
        
The testbed file contains the following information
    
* Credentials to access the servers in the system
* Roles assigned to servers
    
A sample testbed definition consisting of a single server is at:

    /opt/contrail/utils/fabfile/testbeds/testbed_multibox_example.py

Copy one of these files as testbed.py fill in the ip-addresses and root credentials for the servers.

### Step 2: Copy the new contrail-install-packages.rpm | contrail-install-packages.deb to the config node:
Use SCP to do this
    yum localinstall contrail-install-packages.rpm | dpkg -i contrail-install-packages.deb
Run setup script,
    /opt/contrail/contrail_packages/setup.sh

### Step 3: Upgrade the contrail packages and restart the services
The following fab command upgrades the live system, should be run from the config node.:

    cd /opt/contrail/utils
    fab upgrade_contrail:<base_release>,/path/to/contrail-install-packages.rpm | /path/to/contrail-install-packages.deb
        <base_release> can be (1.05 or 1.06) in case of ubuntu and (1.05) in case of centos.

### Step 4: Soft Reboot all the VM's through horizon dashbord.
    To reboot  all the VM's in all tenants following is the fab task.
    cd /opt/contrail/utils
    fab reboot_vm:mode=hard  #Default mode is soft reboot.

    To reboot any single VM following is the fab task
    cd /opt/contrail/utils
    fab reboot_vm:vmid=<uuid of a vm>mode=hard  #Default mode is soft reboot.


Adding a new compute node to existing cluster
---------------------------------------------
    Add all the irequired infromation about the new nodes to testbed.py
    cd /opt/contrail/utils
    fab install_pkg_node:/path/to/contrail-install-packages.rpm | /path/to/contrail-install-packages.deb,<user@newnode1ip>,<user@newnode2ip>
    fab add_vrouter_node:<user@newnode1ip>,<user@newnode2ip>


Removing an existing compute node from  existing cluster
--------------------------------------------------------
    cd /opt/contrail/utils
    fab detach_vrouter_node:<user@oldnode1ip>,<user@oldnode2ip>
    Remove the infromation about the detached nodes from testbed.py

Advanced Information
--------------------
### Testbed Options
* #### env.interface_rename
    Defaults to True, contrail-interface-name package will be installed and ethernet interfaces are renamed such that their names never change across reboots.

    When set to False contrail-interface-name package will not be installed.The assumption is the Linux distribution being used guarantees that the ethernet interfaces on the server always come up with the same name across reboots.
    
* #### multi_tenancy
     Defaults to False, configuration API operations are not checked for resource and API permissions.
     
     When set to True, the checks are performed.

* #### do_parallel
     Defaults to False, serial excution of parallel task in multiple nodes.
     
     When set to True, the fabric task is executed in parallel.

### Task Composition
The fabric tasks are defined in such a way that user can do a high-level operation (for eg. install_and_setup_all) if there is no need for custom workflow. A custom workflow can be achieved by invoking the low-level tasks directly and doing appropriate fixups in between.

Invoke `fab list` to see the list of all tasks and a brief description. The composition of task are as below:

    install_and_setup_all
        install_contrail
            pre_check
            create_install_repo
            install_database
            install_openstack
            install_cfgm
            install_control
            install_collector
            install_webui
            install_vrouter
            upgrade_pkgs
            update_keystone_log
            install_interface_name [if env.interface_rename is True]
                => This task will reboot the compute nodes

        setup_all
            setup_ha
            setup_rabbitmq_cluster
            increase_limits
            increase_ulimits
            setup_database
            verify_database
            setup_openstack
            if Openstack HA setup,
                sync_keystone_ssl_certs
                setup_cluster_monitors
            setup_cfgm
            verify_cfgm
            setup_control
            verify_control
            setup_collector
            verify_collector
            setup_webui
            verify_webui
            setup_vrouter
            prov_control_bgp
            prov_external_bgp
            prov_metadata_services
            prov_encap_type
            setup_remote_syslog
            compute_reboot
                => This task will reboot the compute nodes
            verify_compute


    The composition of tasks related to storage are as below:

    a. To install storage packages:

	install_storage
	    install_storage_master
	    install_storage_compute

    b. To setup storage:
        setup_storage