HGSHM: VM Shared Memory for MapReduce like Applications

With wide spread acceptance of virtualization, virtual machines are heavily used today for better resource utilization. They find their presence in various applications. Many of these applications run multiple copies to enhance both availability and performance. Though true high availability requires these virtual machines (VMs) to be instantiated on different compute nodes, to insulate customers from software bugs and also for performance, we can find multiple copies of virtual machines being instantiated on the same compute node. One such application that has not yet caught in the wave of virtualization is MapReduce. For example: say a video stream needs to be split into objects, compressed and stored. This can be effectively implemented using MapReduce where the mapper will distribute slices of video stream to the reducers, which in-turn compresses and stores them. This generates lots of network traffic as the data needs to be sent to another VM and involves multiple memory copies. To solve this issue, I have developed HGSHM, a shared memory subsystem that allows mapper to direct the data to a memory area that can also be accessed by the reducers. Once the data is filled in the buffer, a signal is sent out to the reducer that can start processing the data.