Studies about Graph management, Pregel, Spark, Frames

1. Construct graph
   1. Vertices
   2. Edges
   3. Other strategy for creating edges
2. Batch management
3. Adaptative batch management
4. Application for GraphFrames
5. Results
6. A topology for alerts and properties

We split the space as a square grid of cells (currently: 100 x 100)

Vertices are created randomly, and a position (x, y) is assigned to them Then we compute the cells

conf.g = 100
cell_id = lambda x, y: int(x*conf.g) + conf.g * int(y*conf.g)

We partition vertices against cell ids (300 partitions)

Edges are created first using an edge_iterator:

• for all vertices:
  • select a random # of connected vertices from all vertices (up to max_degree)
  • ignore self edges
• join [source vertices, edges, dest vertices]:
  • source vertex src.id == edge.src
  • dest vertex dest.id == edge.dst
  • source vertex src.id != dest.id
  • source vertex cell is neighbour of dest vertex cell

cell neighbours:

two cells are said neighbours when:

• when they are adjacent by sides, or by corners
• including a continuous spheric space (left-right and top-down)

We use the mapPartition mechanism to visit partitions

• we divide the space in a matrix (grid) of cells

• we assume that the connection beteen objects will exist only when objects are close enough to each other, ie. up to a maximum distance

• in fact, we construct the cell matrix so as the cell size = the max distance

• thus, each object may only be associated with objects found in the same cell or in immediate neighbour cells (neighbours by sides or by corners)

• we create a "cell_iterator" able to visit all immediate neighbour cells around a given cell

• we define a (lambda) function able to visit all cells plus all neighbour cells of all visited cells

    visit_cells = lambda x: [(src_id, x, y, cell, row, col) for i in x]
  
    # visit all neighbour cells for each cell,
    #
    f2 = lambda x: [[(src_id, x, y, cell_src, _row * grid_size + _col) 
                        for _, _row, _col in cell_iterator(_row, _col, 2, grid_size)] 
                        for src_id, x, y, cell_src, _row, _col in visit_cells(x)]
  

• to operate the visit, we apply mapPartitions to the vertices, and construct the complete list of visited cells and make a dataframe

    def func(p_list):
        yield p_list
        
    vertices_rdd = vertices.rdd.mapPartitions(func)
    full_visit = vertices_rdd.map(lambda x: f2(x))
    all_visited_cells = full_visit.flatMap(lambda x: x).flatMap(lambda x: x)
    all_edges = sqlContext.createDataFrame(all_visited_cells, ['src_id', 'src_cell', 'src_x', 'src_y', 'dst_cell'])
  

• Then this list of edges is filled with objects using a join by the dest vertices (adding an edge id) considering that a sample

        df = all_edges.join(dst, (dst.dst_cell == all_edges.dst_cell) &
                            (all_edges.src_id != dst.dst_id)).\
            select('src_id', 'dst_id'). \
            withColumnRenamed("src_id", "src"). \
            withColumnRenamed("dst_id", "dst"). \
            withColumn('id', monotonically_increasing_id())
  

• optionally, we may limit the max degree by make the dest dataframe a sample

    dst = vertices. \
        withColumnRenamed("id", "dst_id"). \
        withColumnRenamed("cell", "dst_cell"). \
        withColumnRenamed("x", "dst_x"). \
        withColumnRenamed("y", "dst_y"). \
        sample(False, fraction)
  
    degree = np.random.randint(0, degree_max)
    fraction = float(degree) / N
    dst = vertices.sample(False, fraction)
  

• this operation may also performed by batches by hashing the cell id of the source cell

        df = all_edges.join(dst, ((all_edges.src_cell % batches) == batch) &
                            (dst.dst_cell == all_edges.dst_cell) &
                            (all_edges.src_id != dst.dst_id)).\
            select('src_id', 'dst_id'). \
            withColumnRenamed("src_id", "src"). \
            withColumnRenamed("dst_id", "dst"). \
            withColumn('id', monotonically_increasing_id())
  

• Of course we also improve the edge construction by considering the real distance between objects.

        df = all_edges.join(dst, ((all_edges.src_cell % batches) == batch) &
                            (dst.dst_cell == all_edges.dst_cell) &
                            (all_edges.src_id != dst.dst_id) &
                            (dist(src.x, src.y, dst.x, dst.y) < max_distance)).\
            select('src_id', 'dst_id'). \
            withColumnRenamed("src_id", "src"). \
            withColumnRenamed("dst_id", "dst"). \
            withColumn('id', monotonically_increasing_id())
  

To check the algorithm we make a graphical representation

    points = vertices.toPandas()
    x_points = points["x"]
    y_points = points["y"]
    
    edges = df.toPandas()
    e_src_x = edges["src_x"]
    e_src_y = edges["src_y"]
    e_dst_x = edges["dst_x"]
    e_dst_y = edges["dst_y"]
    
    plt.scatter(x_points, y_points, s=1)
    e = [plt.plot((e_src_x[i], e_dst_x[i]), (e_src_y[i], e_dst_y[i])) for i, x in enumerate(e_src_x)]
    plt.show()

Construction of large set of vertices and edges can be split in batches

• subset dataframes are created
• written (append mode) to hdfs(parquet)

The batch management is meant to cope with limited memory in some complex operations that implies large shuffle or aggregations.

Sometimes it's difficult to figure out or foresee the real memory needs (for instance, when the operation is hidden in a library)

The following adaptative pattern helps:

• we assume that the full set is keyed by a completely identifying key

• first we split the full set of elements in the dataframe (using some filtering technique on the key)

  • subset_size = int(full_size / batches)
  • construct subsets using filter with condition: int(key/subset_size) == batch

• we iterate on:

  • we apply the aggregation operation onto each subset

    • if we detect memory error, we increase (x2) the batch number (= we lower the subset size)
    • we adapt the batch number (x2)
    • we continue the iteration with the new conditions

• this pattern may be saved at any step by saving the intermediate results and conditions and restarted later on

example for the triangle count operation:

full_set = N                            # by construction
batches = conf.batches_for_triangles
total_triangles = conf.count_at_restart  # by default = 0
batch = conf.batch_at_restart            # by default = 0

subset = int(full_set / batches)
while batch < batches:
    try:
        # ---------- extract the subset     
        gc.collect()
        g1 = g.filterVertices("int(cell/{}) == {}".format(subset, batch))
        triangles = g1.triangleCount()
        # ---------- apply the aggregation
        gc.collect()
        triangle_count = triangles.agg({"cell":"sum"}).toPandas()["sum(cell)"][0]
        total_triangles += triangle_count
        print("batch=", batch, "total=", total_triangles, "partial", triangle_count)
        batch += 1
    except:
        print("memory error")
        # ----------- new conditions        
        batches *= 2
        batch *= 2
        subset = int(full_set / batches)
        print("restarting with batches=", batches, "subset=", subset, "at batch=", batch)
        if batches >= 1:
            continue
        # ----------- no way to get enough ressources: kill the iteration
        break

1. Varying the number of vertices and the max-degree for edges (and batches for edges)

2. Varying the number of edge-batches (same number of vertices [1000000] and degree[10000])

3. Varying the number of vertices and the max-degree for edges (and batches for edges)

(see the paragraph above concerning batch oriented measurement)

Once vertices and edges are created, graphframes are assembled

A second application read vertex and edge dataframes and re-assemble graphframes and apply algorithms

• we have N alerts

• we have P properties

• every alert gets p properties (p in [0..P])

• we define "zones" : the set of properties associated with an alert define a "zone"

• we can define a distance between zones:

  • number of properties NOT in common to the two zones (symmetric_difference)
  • weighted by the sum of property numbers of the two zones
    • diff(z1, z2)/(len(z1) + len(z2))

• since we have a finite set of different properties, the set of possible distance values is finite

  • the length of a symmetric_difference is within the [0..P] range
  • the maximum distance is 1 when there are no properties in common

• we can compute the distance between two objects = distance of their zones

• we can select all neighbour objects according a range of distance

• we set a link between objects, when their zones is apart by a given distance or a range of distance

An graphical application shows:

• alerts and properties
• links setup when selecting a range of distance
• one color per distance value

see:

• Dimensionality Reduction in Spark