Todo
- Crawl Naver news or Google news based on a query
- Connect to the parsing part to parse and store in database automatically
Todo
Issue
Are all the morphems indeed needed? (ex. 의, 는, 이다, etc.)Multi threading in parsing process doesn't work. It works only for short sentences.doesn't matter. will not use multi threading.
Done
2016.05.12.
- Done with parsing the input sentence for creating rule.
2016.04.09.
- Used dummy text file
- Parsed each line to sentences with multi threading process (2 threads)
Parsed each sentences to morphemes with multi threading process (2 threads)doesn't work.
Todo
| API | Description | CRUD |
|---|---|---|
| (rulesets) PUT /rulesets/{topic_id}/{ruleset_seq}/{new_name} |
|
U
|
Done
Removed
| API | Description | CRUD |
|---|---|---|
/topics/ |
|
R
|
/sources/ |
|
R
|
| (rulesets) POST /rulesets/{topic_id}/ |
|
C
|
/rulesets/{topic_id} |
|
R
|
| (rulesets) DELETE /rulesets/{topic_id}/{category_seq} |
|
D
|
/words/{fulltext} |
|
C
|
| (rules) POST /rules/{topic_id}/{category_seq}/{fulltext}/{word_ids} |
|
C
|
| (rules) GET /rules/{topic_id}/{ruleset_seq} |
|
R
|
/rules/{rule_id} |
|
R
|
/rules/{rule_id}/{word_ids} |
|
U
|
| (rules) DELETE /rules/{rule_id} |
|
D
|
Issue
- See Database - Issue 2.
Done
Todo
Issue
- Some emojis are not properly saved. Some are saved just like '?????'
How about create 'querys' table to store the queries which are used to crawl the posts. (ex. 총선) Then it is possible to categorize the posts and user can analyze only the posts they are interested in. If we only want to analyze just all of the recent posts, it might be redundant data. However, still it is a good option, considering expandability.There is topics table
Done
2015.05.12.
- Crawled posts are stored in MySQL database.
- Rulesets and Rules are sotred in MySQL database.
- Redis hold the result of analysis. There are key-bitarray maps with a rule_id as a key and bitarray with 1 at the position of realted sentece_id as value. If there are no realted sentences for the rule, all the value of bitarray will be 0. The rule_id of Unanalyzed rule is not set in the redis.
2016.04.09.
- Created database shceme and initializing code.
Todo
Issue
Done