- Summary of set up for first Project: Collaborative Online Judge
-
Install NodeJs:
sudo apt-get update
curl -sL https://deb.nodesource.com/setup_7.x | sudo -E bash -
sudo apt-get install -y nodejs
-
Install Nodemon
npm install -g nodemon
-
Install git
sudo apt-get install git
-
Install Redis
wget http://download.redis.io/releases/redis-3.2.6.tar.gz
tar xzf redis-3.2.6.tar.gz
cd redis-3.2.6
make
sudo make install
cd utils
sudo ./install_server.sh
-
Install python 3
-
Install pip:
sudo apt-get update
sudo apt install python-pip
-
Install Docker:
curl -fsSL https://get.docker.com/ | sh
-
Setup docker permission:
sudo usermod -aG docker $(whoami)
(you need to logout and login again after set permission)
To start docker when the system boots: sudo systemctl enable docker
-
Install MongoDB https://docs.mongodb.com/manual/tutorial/install-mongodb-on-ubuntu/
-
Setup Elasticsearch:
sudo docker pull elasticsearch
mkdir /usr/share/elasticsearch/data/esdata
docker run -d --name elasticsearch -p 9200:9200 -p 9300:9300 -v /esdata:/usr/share/elasticsearch/data elasticsearch
docker start elasticsearch
test elasticsearch container is running: curl -X GET http://localhost:9200
- Setup Logstash:
sudo docker pull logstash
mkdir /logstash
vim /logstash/logstash.conf
```
input {
beats {
port => 5044
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGLINE}" }
}
date {
match => [ "timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
output {
elasticsearch {
hosts => ["elasticsearch:9200"]
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
}
stdout {
codec => rubydebug
}
}
```
docker run -d --name logstash -p 5044:5044 --link elasticsearch:elasticsearch -v "$PWD":/logstash logstash -f /logstash/logstash.conf
- Setup Kibana:
sudo docker pull Kibana
docker run --name kibana --link elasticsearch:elasticsearch -p 5601:5601 -d kibana
- Installing and Configuring Beats:
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.5.0-amd64.deb
sudo dpkg -i filebeat-5.5.0-amd64.deb
sudo vim /etc/filebeat/filebeat.yml
```
filebeat.prospectors:
- input_type: log
paths:
- /var/log/main.log
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["localhost:5044"]
```
sudo systemctl restart filebeat
reference: http://www.itzgeek.com/how-tos/linux/ubuntu-how-tos/how-to-run-elk-stack-on-docker-container.html
- I failed to install
newspaperpackage. It shows errors like 'could not build the egg.' This is because an error when installing nltk dependency. Try following commands:
$ sudo apt-get install python-dev
$ sudo apt-get install libxml2-dev libxslt-dev
$ sudo apt-get install libjpeg-dev zlib1g-dev libpng12-devpip
$ sudo pip install --upgrade setuptools
(if above code does not work: sudo easy_install -U setuptools)
$ sudo pip install newspaper