Daedalus is client/server application. The server (what receives the messages) is implemented using Django. The log messages are stored in Cassandra. The clients send the messages using a POST messages, allowing to send messages from virtually any language.

Please, report any issue here.

• Backend: what receives the log messages

• Frontend: the webapp to list the messages (see screenshots). You can:

  • Filter by application
  • Filter by host
  • Filter by severity
  • Show all messages (default for home page)
  • Simplest form of pagination
  • Show line chart counting messages received

• Python client and logging handler: to send messages from Python or through the Python's logging framework

• Java client and log4j appender: to send messages from Java or log4j.

A log messages includes:

• message: any string with a single or multi-line text
• application: the application that generated the meessage
• host: the host that generated the message
• severity: the serverity of the message: one of 'DEBUG', 'INFO', 'WARN' or 'ERROR'
• timestamp: seconds from EPOCH (in UTC) associated with the message.

How to manually send a message from Python:

    import httplib
    import urllib
    params = urllib.urlencode({
        'application': u'intranet',
        'host': u'appserver3.example.com',
        'severity': u"WARN",
        'message': u"The account 'johndoe' was locked after three login attempts.",
        'timestamp': u"1343607762.837293",
    })
    headers = {"Content-type": "application/x-www-form-urlencoded"}
    conn = httplib.HTTPConnection("localhost", "64364")
    conn.request("POST", "/backend/save/", params, headers)
    response = conn.getresponse()
    print response.read()
    conn.close()

Sending the same message using the Python client:

    msg = u"The account 'johndoe' was locked after three login attempts."
    daedalus_client = DaedalusClient("localhost", 64364)
    daedalus_client.send_message(msg, "ERROR", "appserver3.example.com", "intranet")

To install and run Daedalus server using pip, (assuming a virtualenv directory exists and Cassandra running on localhost) use:

$ ./virtualenv/bin/pip install daedalus
$ export DJANGO_SETTINGS_MODULE=daedalus.settings
$ ./virtualenv/bin/django-admin.py runserver

To install only the Python client (and logging handler), run:

$ pip install daedalus-python-client

See dev-scripts/install-on-ubuntu.sh.

I recommend you to download and run the script in a newly created virtual machine (mainly because it install many packages, and must be run as root). The virtual machine should have at least 1GB of RAM (Cassandra may not work with less memory). The scripts installs JDK, Cassandra, clones the Daedalus repository and launch the Django development server.

You can download this script and run it as root, or use it as a guide, copying-and-pasting each of the commans of the script in a console or ssh session.

JDK: download and install JDK 6 (needed for Cassandra).

Cassandra: download, install and start Cassandra.

Download from GitHub

$ git clone http://github.com/hgdeoro/daedalus
$ cd daedalus

Create virtualenv and install requeriments

$ virtualenv virtualenv
$ ./virtualenv/bin/pip install -r requirements.txt
$ ./virtualenv/bin/pip install -r requirements-dev.txt

Run syncdb and syncdb_cassandra

$ ./dev-scripts/manage.sh syncdb
$ ./dev-scripts/manage.sh syncdb_cassandra

Start memchaed

$ sudo service memcached start

Start development server

$ ./dev-scripts/runserver.sh

By now you'll have the Django development server running. Both the Daedalus backend (the Django app that receives the logs via HTTP) and the frontend (the application used to see the logs) are started. To use it, go to http://127.0.0.1:64364/.

To create some random log messages, you could run:

$ ./dev-scripts/bulk_save_random_messages.sh

(and press Ctrl+C twice to stop it).

The project could be imported from within Eclipse PyDev.

See TODOs.

• A single keyspace holds all the column families.

• Messages are stored using 4 column families:

  • CF: Logs - Cols[]: { uuid1/timestamp: JSON encodded message }
  • CF: Logs_by_app - Cols[]: { uuid1/timestamp: JSON encodded message }
  • CF: Logs_by_host - Cols[]: { uuid1/timestamp: JSON encodded message }
  • CF: Logs_by_severity - Cols[]: { uuid1/timestamp: JSON encodded message }

• Alternative format (implemented by StorageServiceUniqueMessagePlusReferences):

  • CF: Logs - Cols[]: { uuid1/timestamp: JSON encodded message }
  • CF: Logs_by_app - Cols[]: { uuid1/timestamp: '' }
  • CF: Logs_by_host - Cols[]: { uuid1/timestamp: '' }
  • CF: Logs_by_severity - Cols[]: { uuid1/timestamp: '' }

• No SuperColumn nor secondary indexes by now.

• Upgrade to pycassa 1.7
• Backend: new storage: StorageServiceRowPerMinute
• Created multi-messages log entryies to group related messages

• Frontend: added 'host' column to list of messages
• Client: implemented command line arguments for setting the severity
• Client: implemented tests for CLI
• General: automatized steps needed for creating a 'release'

• General: refactored Python package (from hgdeoro.daedalus.web.frontend/backend to daedalus.frontend/backend)
• General: changed default port from 8084 to 64364
• Frontend: implemented reset of memcache

• Client: created a command line to send log events
• Client: created a handler for the Python logging framework
• General: fixed various issues around setup.py and created setup.py for the Python client.
• General: now the Daedalus server and Python client are uploaded to PYPI

#-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
#    daedalus - Centralized log server
#    Copyright (C) 2012 - Horacio Guillermo de Oro <hgdeoro@gmail.com>
#
#    This file is part of daedalus.
#
#    daedalus is free software; you can redistribute it and/or modify
#    it under the terms of the GNU General Public License as published by
#    the Free Software Foundation version 2.
#
#    daedalus is distributed in the hope that it will be useful,
#    but WITHOUT ANY WARRANTY; without even the implied warranty of
#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#    GNU General Public License version 2 for more details.
#
#    You should have received a copy of the GNU General Public License
#    along with daedalus; see the file LICENSE.txt.
#-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-