##Voyage Voyage currently has 2 components:
- Web Server is capable of editing and displaying all the stored data as well as scopes you will provide to Explorer, through your favorite browser.
- Explorer searches the web using scopes given through to the Web Server and goes for exploring for you. It will automatically store all relevant informations found on the way, so that you can show all the loot through Web Server.
##Requirement
#####Python 2.7.10+
You can check your current version by python --version
If the version available through your package mananger is not 2.7.10 or above, you will need to manually build and install 2.7.10. Luckily, there is a tool for doing that.
curl -L https://raw.githubusercontent.com/yyuu/pyenv-installer/master/bin/pyenv-installer | bash
sudo apt-get build-dep python2.7
pyenv install 2.7.10
pyenv global 2.7.10
pip install -U pip
#####Wget 1.14+
- You can check your current version by
wget --version
##Installation ####Through terminal
- Download, extract the master then execute on terminal
./InstallScript.sh - Set up a mysql server and create a table with the following schema:
CREATE DATABASE crawler CHARACTER SET UTF8 COLLATE utf8_general_ci;
(you can replace crawler with another database name if you want). Then put in the database name and credentials in config.yaml
If any problem occurs, please contact one of us on the email address listed above.
##Configuration You can edit the config.yaml file for personal settings
Frontend/Frontend/settings.py. A new SECRET key can be generate with the following python script:
import random
''.join(random.SystemRandom().choice('abcdefghijklmnopqrstuvwxyz0123456789!@#$%^&*(-_=+)') for _ in range(50))
##Usage: Web Server ####Through terminal
- To start
python server.py run - To stop
python server.py stop
You can now access the server through http://IP:PORT/admin
The default is http://127.0.0.1/admin
Default Login Credentials:
- User: admin
- Password: admin
#Tabs ####Home Here you can view your action history and quick navigations to the database ####Scope Here, you can view and edit 4 requirement to explore:
- Referring Sites: The sites in which explorer will look into. It will automatically get validated when adding.
- Twitter Accounts: The twitter accounts which explorer will look into. It will automatically be validated when adding.
- Source Sites: The sites which explorer looks for in the articles/tweets if they are used as source.
- Keywords: The words which explorer look for in the articles/tweets if they are used.
####Data Here, you can view the collected data by the explorer. Furthermore, you can download the archived entry as Web Archive. For demo, it is filled with pre-explored entries. ####Downloads Here, you can download all the data stored in the database as Json format. ####Statistics Here, you can view the statistics among the collected entries.
For example, you can view how many articles got collected per day as a Annotation Chart ####Visualizations Here, you can view the relations between each of the 4 scopes, based on the exploration.
####Authorization Here, you can manage the users and groups used for log in. Furthermore, users can have different permissions.
Once your scope is ready, you may start exploring by clicking Run on the status bar.
- Article Explorer will explore through the Referring Sites for articles
- Twitter Explorer will explore through Twitter for Twitter Accounts's posts
##UnitTest
Unit test files are located under src/unit_tests