-
Notifications
You must be signed in to change notification settings - Fork 0
/
corpora-and-regex.py
60 lines (51 loc) · 1.94 KB
/
corpora-and-regex.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
# Corpora and Regex
import nltk
import os
import sys
# nltk.download('brown')
from nltk.corpus import brown # Brown University Stanford Corpus of Present-Dat American English
# x = nltk.data.load('big.txt', format='raw')
# print(x[:500])
# importing in text format makes things 1000x better
x = nltk.data.load('big.txt', format='text')
# print(x[:500])
# and you can do things like this
# print("the" in x)
# from nltk.corpus.reader import WordListCorpusReader
# reader = WordListCorpusReader('./', ['wordlist.txt', 'wordlist2.txt'])
# print(reader.words())
# print(reader.fileids())
### Lemmatization & Stemming
# nltk.download('wordnet')
# nltk.download('punkt')
# from nltk.stem.porter import *
# stemmer = PorterStemmer()
# print(stemmer.stem('loving'))
# from nltk.stem import WordNetLemmatizer
# wnl = WordNetLemmatizer()
# print(wnl.lemmatize('monsters'))
'''
In each of the above cases we have handled one word.
Now print the stemmed and lemmatized versions of all the words in the document computerscience.txt
Preview the document. Here is an overview of what you need to do:
1. Load the file into a reader [ Hint: reader = WordListCorpusReader( ... ) ]
2. use word_tokenize from nltk.tokenize to convert the text into words
3. Loop through the text [Hint: Use the for statement]
4. Lemmatize and Stem each word.
5. Look at the difference between the two, notice how the lemmatizer makes mistakes in some cases - can you identify why and propose a solution?
'''
from nltk.corpus.reader import WordListCorpusReader
tokens = []
reader = WordListCorpusReader('./', ['computerscience.txt'])
for count, ele in enumerate(reader.words()):
print(count, "\b:", ele, "\n")
tokens += nltk.word_tokenize(ele)
print(tokens)
from nltk.stem.porter import *
stemmer = PorterStemmer()
from nltk.stem import WordNetLemmatizer
wnl = WordNetLemmatizer()
for token in tokens:
print(token)
print(wnl.lemmatize(token))
print(stemmer.stem(token))