After the advent of convolutional and recurrent neural networks, the content based intelligent classification is being given high priority. But the biased feature learning by recurrent networks and window restricted feature learning by convolutional networks restrict the performance of these networks. In this project, three variants of deep learning based text classification system for combining the convolutional and recurrent networks in parallel manner that overcomes their individual limitations is proposed. We also propose a novel activation function Inverse Exponential Linear Unit (IELU) that allows the network to learn more features in comparison with traditional activation functions like Rectified Linear unit and Exponential Linear unit. The throughput of the system is analysed by considering a variety of datasets and performance metrics into consideration. It has been inferred that all the three parallel architectures have higher performance when compared with the existing classification systems. Also, our IELU performs better than the widely used activation functions in text classification application.