NCARFlightMonitor is a python package designed to make creating real time data monitoring programs for the EOL/NCAR aircraft simple to write and robust against failures of the real time database and programming syntax errors. NCARFlightMonitor's design is similar to the Processing/Wiring programming languages.

This is a python package. It can be used in the following ways.

To install, simply go to the downloaded directory and type.

python setup.py install

You may need to have root privileges (sudo) to use the above command, if your current user does not have access to the python installation directory.

The install command above will also install the psycopg2 and Twisted python packages. If an error occurs and these packages cannot be installed through the setup.py script then the following command will install the packages manually.

easy_install psycopg2 Twisted

Similar to the first command, this may require root (sudo) privileges.

To install the package temporarily (useful when modifying the package)

python setup.py develop

This carries the same privilege issues as the system wide install; see above method.

Alternatively one can write scripts where the directory NCARFlightMonitor is in the same folder as the script.

The most simple python script using the NCARFlightMonitor package is as follows.

#!/usr/bin/env python
# encoding: utf-8
from NCARFlightMonitor.watch import NWatcher

## default to using eol-rt-data.guest.ucar.edu for SQL server
watch_server = NWatcher(database="GV")
watch_server.startWatching()

This will listen for when the GV is in the air, spit out messages to standard out about missing data values when in flight, and write an ASCII file to the user's temporary directory containing the data from the flight (in the same format as the ASCII files created by Aeros).

Using NWatcher(database="GV", email_fn=some_function will use a user defied function with format some_function(flight_info={}, files=[], body_msg="") to email the file after each flight. More information on how to construct a compatible email function can be found in the examples, release notes, and the source code (which is relatively well documented).

A complete list of initialization parameters and their use can be found in the source code.

I would eventually like to be able to create doxygen like documentation for the package, but have yet to have time. :-(

The NWatcher class is designed to be easily augmented with data processing functions that provide more detailed status messages about the data being collected during flight. To delve into how this paradigm works, let's walk through the following example.

#!/usr/bin/env python
# encoding: utf-8
from NCARFlightMonitor.watch import NWatcher                          # (1)

## Function Definitions
def setup_co(self):                                                   # (2)
    ## All variables starting with `self` are persistant across process
    ## function runs and can be called in the process function. This can
    ## be useful for running averages and the like.
    self.cal = False

def process_co(self, tm, data):                                       # (3)
    coraw_al = data[0]

    if coraw_al <= 8000 and self.cal == False:
        self.log.print_msg("CO cal occuring.", tm)
        self.cal = True
    ## Reset so that the log message does not appear thousands of times.
    elif coraw_al > 8000 and self.cal == True:
        self.cal = False

## Main
watch_server = NWatcher(database="GV")                                # (4)

watch_server.attachAlgo(variables=('coraw_al',),                      # (5)
                        start_fn=setup_co,
                        process_fn=process_co,
                        description="CO raw cal checker")

watch_server.startWatching()                                          # (6)

This program has most of the same structure as the basic example (1, 4, 6). The added sections show how one can add a function to monitor the coraw_al value to determine if it's value drops below 8000, and if so print a message to standard out. The algorithm to do this below 800 check (2,3) is attached through (5), which will set up the attached algorithm (3) to be run against the real time data as it is acquired from the aircraft.

The setup of (5) is relatively simple.

• variables: tuple string list of variable names that are to be analysed in the attached algorithm. In this case, the attached algorithm only analyses the coraw_al variable.

• start_fn (2): a function that is run only once before any data is collected. All variables that are in the form self.variable_name are accessible in the process function each time it is run. These variables can be through of as global variables for the associated process function. The function definition must be in the same format (where the keyword self is the only function argument).

  Additionally there is a variable called self._flight_start_time that provides the start time of the current flight, as a datetime.datetime variable.

• process_fn (3): a function that is run every time new data is acquired, where tm is a datetime.datetime object equal to the time stamp for the data, and data is a list of data values, ordered by order of the variables argument's tuple. For example, if variables=('var1', 'var2'), then data is [var1, var2]. This function will not run when there is no new data (sat com loss).

  All of the variables prefixed with self in the setup function are availible in this function. There is an additional object self.log that can be accessed which provides a log of information about the current flight. This object's main function is self.log.print_msg(message_string, timestamp), which prints the message to standard out and logs the information. This information is then sent, along with a an ascii file of the flight's data, at the end of the flight.

• description: A description of the algorithm, as a string. This is optional, but will assist in debugging the attached algorithm if something goes awry.

Running the example against the HIPPO5_rf05 sample file produces the following output (assuming the program started on 2011-08-19 18:00:00Z). Notice that bad data checks are automatically added by the package.

[2011-08-19 18:00:00Z] Waiting for flight.
[2011-08-19 18:05:00Z] In Flight.
[2011-08-19 18:05:03Z] co2_qlive MISSING DATA
[2011-08-19 18:19:23Z] ch4_qlive MISSING DATA
[2011-08-19 18:19:23Z] co_qlive MISSING DATA
[2011-08-19 19:03:23Z] CO cal occuring.
[2011-08-19 19:47:03Z] ch4_qlive no longer has missing data
[2011-08-19 19:47:03Z] co_qlive no longer has missing data
[2011-08-19 20:06:13Z] CO cal occuring.
[2011-08-19 21:09:03Z] CO cal occuring.
[2011-08-20 00:17:33Z] CO cal occuring.
[2011-08-20 01:20:23Z] CO cal occuring.
[2011-08-20 02:23:13Z] CO cal occuring.
[2011-08-20 02:37:30Z] Flight ending.
[2011-08-20 02:39:35Z] Outputting file to /var/folders/dw/v6_mfm_j175f4l7n11rn44nm0000gn/T/HIPPO-5-rf05-2011_09_20-05_40_26.asc
[2011-08-20 02:39:35Z] Waiting for flight.

The program will continue until it receives a kill command (CTRL-C on *nix).

More information can be found in the documentation.

All sample flight ASCII files are in the samples/ directory.

• HIPPO-5-rf02-2011_08_11-21_49_04.asc: HIPPO 5 Research Flight 02 on the GV. Provides a sample where nothing has occurred abnormally (such as GPS going out, losing satcom, etc).

• HIPPO-5-rf05-2011_08_20-03_34_52.asc: HIPPO 5 Research Flight 05 on the GV. This flight was over the arctic, and hence satcom was lost for the magority of the flight (in one chunk).

• HIPPO-5-rf05-rf06-combined.asc: HIPPO 5 Research Flights 05 and 06 on the GV, with the times on 06 reduced by two days to make wait times inbetween flights when in simulation mode shorter. This provides an opportunity to test the entire package in a continuous monitoring setting ('daemon' mode) across multiple flights.

• HIPPO-5-rf06-2011_08_24-01_09_09.asc: HIPPO 5 Research Flight 06 on the GV. In this case, the TASX flight speed variable is unavailable for a period of time. This provides an example of calculating flight speed from GPS.

• ICE-T-rf12-2011_07_30-19_38_00.asc: ICE-T Research Flight 12 on the C130. This has a different instrumentation set.

All example programs are in the examples/ directory.

• bot.py: A chatbot that is identical to cli_monitor.py but outputs to a IRC server as well as the command line.
• cli_monitor.py: Watches for when a plane takes off, records data from the flight, and watches for data integrity.
• load_file.py: A program to load the sample files into a PSQL repository
• simulate.py: A version of the program in README.mkd that runs against a database from a simulated start time, and for one flight only.

Copyright (C) 2011 Ryan Orendorff, NCAR

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.