This thesis investigates which sentence features are encoded by Recurrent Neural Networks (RNNs). We train two state-of-the-art RNN sentence encoders and com- pare them against a CBOW (continuous-bag-of-words) baseline, which represents sentences by averaging pre-trained word vectors. We evaluate the models on three tasks: Predicting (i) sentence length (ii) word content, and (iii) dependency tags. Our findings are that on tasks where information about the full sentence or the order of words is not important (e.g. the word content task) the CBOW baseline performs on par with the RNN encoders. And on tasks where information about the full sen- tence or order of words is important (e.g. the sentence length and dependency tag tasks), the RNNs outperform the CBOW baseline. The main contribution of this thesis is to show that dependency tags can be retrieved from a classifier which has only access to the sentence embedding, and the embed- dings of two words. This provides some evidence that RNNs are capable of learning syntactic relationships between words, even if this was not part of the objective they were trained on. Upon further inspection of sentence embeddings, we also illustrate that certain di- mensions carry information about specific aspects of the sentence. We show that there are differences across models with respect to how strongly, and in how many dimensions such aspects are encoded.