Skip to content
This repository has been archived by the owner on Jan 30, 2020. It is now read-only.
/ futuritest Public archive

[test task] parse nginx access.log file and store it to noSQL db

Notifications You must be signed in to change notification settings

dir01/futuritest

Repository files navigation

Installation

Dealing with parser requires the following steps:

  1. Add custom log format to the nginx.conf's http section

    log_format my_log_format '$msec|$cookie_id|$uri|$status|$request_time|$request_length';

  2. Specify my_log_format as desired format for your access.log

    access_log /var/log/nginx/futuritest-access.log my_log_format buffer=32k;

  3. Create virtualenv

    virtualenv ~/env/

  4. Install required python dependencies

    pip install -r requirements.txt

  5. Run fake webserver (this webserver responds with random content for any dispatched url)

    python application.py

  6. Run log monitor itself providing path to access.log file, and, optionally, mongo database and collection names

    python logmonitor.py /var/log/nginx/futuritest-access.log --database=test --collection=logs

  7. Generate fake traffic with send_requests.py

    python send_requests.py http://localhost/

  8. In order to force log rotation

    sudo logrotate /etc/logrotate.d/nginx -vf

DB choice justification

Since task is pretty trivial, there is no much to justify, really.

MongoDB was chosen since it has rich query language which can potentially allow any further logs analysis, write speed is high enough, and if not, write sharding is supported out of box for easy horizontal scaling. Long story short, it's good enough for our purposes.

About

[test task] parse nginx access.log file and store it to noSQL db

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages