MultiTwin package for the analysis of multipartite graphs.

Authors: Eduardo Corel & Jananan S. Pathmanathan -- 2014-2018.

• From this repository: download or clone the zipped archive MultiTwin-master.zip into an INSTALL_DIR.

  $ cd INSTALL_DIR
  $ unzip MultiTwin-master.zip
  $ cd MultiTwin-master	
  

  $ ls
  install.sh                           Installer script
  INSTALL.md                           Install instructions
  README.md                            this file
  BlastProg/                           program sources in C++
  data/                                test data 
  python-scripts/                      python scripts

$ ls python-scripts/

Except for bt_launcher.py, fg_launcher.py, dt_launcher.py, ds_launcher.py, utils.py, xmlform.py, all files should be executable (if not, change the status with $ chmod +x *.py)

$ ls BlastProg/

cleanblast and familydetector should be executable (same remark as above)

$ ls data/
genome2seq.txt
seq.annot
seq.blastp

This is a toy example to test the installation.

detect_twins.py, factorgraph.py, bitwin.py and description.py have a graphical interface mode, invoked on the command line as program_name.py -G. Any argument that is passed according to the syntax detailed below will appear in the corresponding field of the graphical interface, where it can also be modified.

See the detailed description of the executables below for more explanations.

A graph is specified by one compulsory and several optional files

• edge file (compulsory)
• node type file (optional: for multipartite graphs, i.e. tripartite or higher)
• trail file (optional)

The edge file describes the graph by its edges in a standard way:

Node1 TAB Node2

This format is most convenient for storing large graph files, although its main drawback is the unability to store singletons. Nodes can be any string (with or without whitespace, although they should definitely avoid TAB characters)

UNIPARTITE GRAPHS: no further specification needed.

BIPARTITE GRAPHS: Bipartite graphs are graphs containing two sets of nodes (conventionally called "white" nodes and "black" nodes), such that every edge connects only nodes of different sets.

For bipartite graphs, our code implicitly considers all nodes appearing in the same column to be of the same colour:

Node1 TAB Node2
Node3 TAB Node4
Node1 TAB Node4
(..)

Node1 and Node3 are then, say, white, and Node2 and Node4, black.

NOTE: Any inconsistency in the input file will NOT be caught! If a node appears in both columns, then the graph is not bipartite, but this should be specified (usually by a -k 1 option).

MULTIPARTITE GRAPHS: k-partite graphs are graphs whose set of nodes V, is partitioned in k subsets V_1,...,V_k such that an edge only connects nodes from different subsets.

They must be described with an additional node type file:

Node1 TAB nodeType1
Node2 TAB nodeType1
Node3 TAB nodeType2
Node4 TAB nodeType2
Node5 TAB nodeType3
(...)

1. detect_twins.py [options] edgeFile
   Twins are nodes in a graph that have exactly the same set of neighbours.

• Input: a graph
• Output (by default): a single file containing the twins in the graph.
  Node1 TAB TwinClass1
Node2 TAB TwinClass1
Node3 TAB TwinClass2
Node4 TAB TwinClass2
Node5 TAB TwinClass2
  (…)

Nodes that form a singleton (i.e. whose set of neighbours is unique) are not given a twin identifier: the output file can therefore have less lines than the number of nodes.

• Options:

2. factorgraph.py [options] -i inNetworkFile -o outNetworkFile -T outTrailingFile

This script is the central piece of the MultiTwin software: it is meant to construct a factor graph (or quotient graph) from an input graph, and allow to track the modifications that have been applied to its nodes.
The factor graph is constructed from a community file: its nodes correspond to the communities, and an edge is drawn between two communities whenever members of these communities were connected.
If no community file is supplied however, it simply recasts the input graph by renaming its nodes, and outputs the dictionary of the renaming as an outTrailingFile .
The essential usefulness of this script is provided through the -c (and -t ) options.

A community file simply specifies the community id of the nodes in the graph, according to the syntax

Node1 TAB identifier1
Node2 TAB identifier1
Node3 TAB identifier2
Node4 TAB identifier2
(…)

A community file and a trail file follow the same syntax, except for a two-line header in the trail file that recalls the process leading to the current graph.

The trail file is of central importance: it links the original node ID to the current node ID it has been mapped to, starting from the root graph.

Note: It is possible to change the ROOT graph at any step. If one supplies no trailFile to factorgraph.py , then it is implicitly assumed that the edgeFile provided is the (new) ROOT graph (and its node identifiers the reference ones).

• Options:

3. description.py [options] edgeFile annotFile

This script produces a complete summary of the contents of the graph contained in the edgeFile, in terms of the attributes contained in an annotation file annotFile.

The annotFile is a flat TAB-separated file with columns containing the attributes of the nodes in the graph. It has the following syntax:

The expected output of this script is the content in all attributes of all nodes in the graph. This script was conceived to be as flexible as possible: it has therefore a lot of parameterizing options!
Basically, it works as a two-step procedure:

1. Construct an XML configuration file CONFIG_FILE specifying
   • what are the components (usually elements of a terminal clustering)
   • what nodes are considered (which node types)
   • which attributes are assigned (list of attributes)
   • what levels of clustering should be included (trail files)
2. Run the script with this configuration file produces description output files, either
   • a plain text file (readable but difficult to parse)
   • an XML output file (parsable but more difficult to read).

The configuration file is generated as a template (named config.xml by default) when the code is run without one. It is complete (i.e contains all attributes, for all node types and for all levels) when called as follows:

$ description.py -c COMP_FILE -H TRAIL_FILE [-X CONFIG_FILE] EDGE_FILE ANNOT_FILE

In the CONFIG_FILE, the following elements will be generated:

• corresponding to the COMP_FILE

  * one <mod> element comprising
  * one <filename> field (COMP_FILE)
  * one <name> field ("Module" by default)
  * as many <key> elements as node types, containing each
         - one <type> attribute (the node type, say 1)
         - one <display> attribute (boolean `True/False`)
         - one <name> field ("NodeType1" by default)
  

• corresponding to the TRAIL_FILE

   * as many <trail> elements as levels from the root graph 
     (see the output of $ trailhistory.py TRAIL_FILE) each comprising
          - one <rank> field (the index in the trail file sequence)
          - one <name> field (the trailFile name)
          - as many <key> elements as node types (same as before)
  

• corresponding to the ANNOT_FILE

    *  as many <attr> elements as column names in the ANNOT_FILE header comprising each
          - one <name> field (the column name)
          - as many <node> elements as node types comprising
                  * one <type> attribute (the node type)
                  * one <display> attribute (True/False)
  

The configuration file can be manually or graphically (-G option) edited to restrict to some elements, according to the following rules:

• if no COMP_FILE is given, every node of the current graph will form one component.
• trail files can be skipped, but take care to adapt the <rank> field accordingly (as consecutive integers starting from 1). Alternatively, one can set all the displayattributes to False.
• the <display> field should be set to <display="True"> when the corresponding key/attribute should be taken into account.
• the <name> field of the <mod> element can be set at will to describe the components, depending on their nature (e.g. Connected component, Twin component…)
• the <name> field of the <key> element can also be chosen to describe the nature of the node types (e.g. Genome for NodeType1, and Twin or Gene Family for NodeType2 in the bitwin.py example).

When passed to the script along with an edgeFile and an annotFile, the code

• parses all components in the terminal clustering of the current graph (-c COMP_FILE ; if none is given, then we parse each node of the current graph)

• for each one, visits the previous levels of clustering up to the ROOT graph (by visiting the tree representing the sequence of trail files in a depth-first search).

• summarizes all the values of the attributes contained in each node at each specified level of clustering.

• The output can be any one of:

  • a plain text file (readable but difficult to parse) (-o option)
  • an XML-formatted file (less readable but parsable) (-O option)

NB: The complete description files can be VERY verbose.

• Options:

4. subgraph.py [options] in_networkFile out_subnetworkFile

Computes a subgraph of the input graph, with respect to a subset of nodes.

• Options:

5. trailhistory.py [options] trailFile

Reconstructs the trail history since the ROOT graph, including all the calls to factorgraph.py and the names of the intermediate trailFiles. This is an independent utility script.
Input: the last trailFile.

• Options:

6. transfer_annotations.py [options] annotationFile trailFile outFile

Updates the annotationFile with the new IDs specified in the trailFile. In case several old IDs are given the same new ID, the output file will have as many rows as in the original annotationFile.

• Options:

7. cluster.py [options] edgeFile

Wrapper for several clustering algorithms implemented in igraph: produces a community file formatted clustering output for the graph specified in the edgeFile.
Ensures that the community file has the correct node identifiers (this can be a problem since igraph automatically renumbers node IDs).

• Options:

• fg: community_fastgreedy(self, weights=None)
  Community structure based on the greedy optimization of modularity.
• im: community_infomap(self, edge_weights=None, vertex_weights=None, trials=10)
  Finds the community structure of the network according to the Infomap method of Martin Rosvall and Carl T.
• le: community_leading_eigenvector(clusters=None, weights=None, arpack_options=None)
  Newman’s leading eigenvector method for detecting community structure.
• lp: community_label_propagation(weights=None, initial=None, fixed=None)
  Finds the community structure of the graph according to the label propagation method of Raghavan et al.
• ml: community_multilevel(self, weights=None, return_levels=False)
  Community structure based on the multilevel algorithm of Blondel et al.
• om: community_optimal_modularity(self, *args, **kwds)
  Calculates the optimal modularity score of the graph and the corresponding community structure.
• eb: community_edge_betweenness(self, clusters=None, directed=True, weights=None)
  Community structure based on the betweenness of the edges in the network.
• sg: community_spinglass(weights=None, spins=25, parupdate=False, start_temp=1, stop_temp=0.01, cool_fact=0.99, update_rule='config', gamma=1, implementation='orig', lambda_=1)
  Finds the community structure of the graph according to the spinglass community detection method of Reichardt & Bornholdt. Attention: only works on connected graphs.
• wt: community_walktrap(self, weights=None, steps=4)
  Community detection algorithm of Latapy & Pons, based on random walks.

8. bitwin.py [options] -b blastFile -g genome2sequenceFile

Standalone program generating the twin and articulation points analysis for the genome-gene family bipartite graph. It takes sequence data and sequence-to-genome assignment as input.

• Options:

9. blast_all.py [-h] -i I [-db DB] [-evalue EVALUE] -out OUT -th TH [-fasta_spl FASTA_SPL]

Runs a parallel BLAST on TH threads for the FASTA input file I, and stores the output in file OUT.

NB: It is recommended that you run your own BLAST for large data (you know your machine best!). Also consider using diamond.

10. simplify_graph.py [options] in_networkFile out_subnetworkFile

Removes gene family nodes having bounded degree (default by 1).

• Options: