ParaText is a C++ library to read text files in parallel on multi-core machines. The alpha release includes a CSV reader and Python bindings.
ParaText has the following dependencies for its Python bindings:
- a C++ compiler that is C++11 compliant (gcc 4.8 or above, clang 3.4 or above)
- SWIG 2.0.11 or above
- Python (2.7 or above)
- setuptools
Though Pandas is optional, it must be installed to use ParaText to read CSV files into Pandas.
The ParaText library can be built with two commands:
cd python/
python setup.py build install
Use the --prefix option if you prefer to install ParaText to a
different location:
cd python/
python setup.py build install --prefix=/my/prefix/dir
First, import the paratext Python package.
import paratext
A CSV file can be loaded into Pandas in just one line of code using
the load_csv_to_pandas function.
df = paratext.load_csv_to_pandas("hepatitis.csv")
Its output looks something like this:
In [1]: print df.head()
AGE SEX STEROID ANTIVIRALS FATIGUE MALAISE ANOREXIA LIVER_BIG \
0 30 male no no no no no no
1 50 female no no yes no no no
2 78 female yes no yes no no yes
3 31 female nan yes no no no yes
4 34 female yes no no no no yes
LIVER_FIRM SPLEEN_PALPABLE SPIDERS ASCITES VARICES BILIRUBIN \
0 no no no no no 1.0
1 no no no no no 0.9
2 no no no no no 0.7
3 no no no no no 0.7
4 no no no no no 1.0
ALK_PHOSPHATE SGOT ALBUMIN PROTIME HISTOLOGY Class
0 85 18 4.0 NaN no LIVE
1 135 42 3.5 NaN no LIVE
2 96 32 4.0 NaN no LIVE
3 46 52 4.0 80 no LIVE
4 NaN 200 4.0 NaN no LIVE
A Python dictionary of arrays is preferable over a DataFrame
if the memory budget is very tight. The load_csv_to_dict
loads a CSV file, storing the columns as a dictionary of
arrays.
dict_frame, levels = paratext.load_csv_to_dict(filename)
It returns a two element tuple. The first dict_frame is a Python
dictionary that maps column names to column data. The second levels
is also a Python dictionary keyed by column name. It contains a list
of level strings for each categorical column.
The following code visits the columns. For each column, it
prints its name, the first 5 values of its data, and the categorical
levels (None if not categorical).
for key in dict_frame.keys():
print key, repr(dict_frame[key][0:5]), levels.get(key, None)
The output looks something like this:
PROTIME array([ nan, nan, nan, 80., nan], dtype=float32) None
LIVER_BIG array([0, 0, 1, 1, 1], dtype=uint8) ['no' 'yes' 'nan']
ALBUMIN array([ 4. , 3.5, 4. , 4. , 4. ], dtype=float32) None
ALK_PHOSPHATE array([ 85., 135., 96., 46., nan], dtype=float32) None
ANTIVIRALS array([0, 0, 0, 1, 0], dtype=uint8) ['no' 'yes']
HISTOLOGY array([0, 0, 0, 0, 0], dtype=uint8) ['no' 'yes']
BILIRUBIN array([ 1., 0.89999998, 0.69999999, 0.69999999, 1. ], dtype=float32) None
AGE array([30, 50, 78, 31, 34], dtype=uint8) None
SEX array([0, 1, 1, 1, 1], dtype=uint8) ['male' 'female']
STEROID array([0, 0, 1, 2, 1], dtype=uint8) ['no' 'yes' 'nan']
SGOT array([ 18., 42., 32., 52., 200.], dtype=float32) None
MALAISE array([0, 0, 0, 0, 0], dtype=uint8) ['no' 'yes' 'nan']
FATIGUE array([0, 1, 1, 0, 0], dtype=uint8) ['no' 'yes' 'nan']
SPIDERS array([0, 0, 0, 0, 0], dtype=uint8) ['no' 'yes' 'nan']
VARICES array([0, 0, 0, 0, 0], dtype=uint8) ['no' 'nan' 'yes']
LIVER_FIRM array([0, 0, 0, 0, 0], dtype=uint8) ['no' 'yes' 'nan']
SPLEEN_PALPABLE array([0, 0, 0, 0, 0], dtype=uint8) ['no' 'yes' 'nan']
ASCITES array([0, 0, 0, 0, 0], dtype=uint8) ['no' 'yes' 'nan']
Class array([0, 0, 0, 0, 0], dtype=uint8) ['LIVE' 'DIE']
ANOREXIA array([0, 0, 0, 0, 0], dtype=uint8) ['no' 'yes' 'nan']
All categorical columns in this data set have 3 or fewer levels so
they are all uint8_t. A string representation uses at least 8 times
as much space, but it can also be less computationally efficient. An
integer representation is ideal for learning on categorical columns.
Integer comparisons over contiguous integer buffers are pretty cheap
compared to exhaustive string comparisons on (potentially)
discontiguous string values. This makes a big difference for
combinatorial learning algorithms.
ParaText supports reading CSV files with multi-line fields in parallel. This feature must be explicitly activated as it requires extra overhead to adjust the boundaries of the chunks processed by the workers.
df = paratext.load_csv_to_pandas("messy.csv", allow_quoted_newlines=True)
ParaText detects the presence of a header. This can be turned off with
no_header=True.
Wise ParaText supports three kinds of columns:
- numeric: for numeric data.
- categorical: for categorical data.
- text: for large strings like e-mails and text documents.
In the library, we distinguish between semantics and data type. The
semantics defines how to interpret a column. The data type (uint8,
int64, float, etc.) defines how its encoded.
Most CSV loading functions in ParaText have the following
- `cat_names`: A list of column names to force as categorical regardless
of the inferred type.
- `text_names`: A list of column names that should be treated as rich text
regardless of its inferred type.
- `num_names`: A list of column names that should be treated as
numeric regardless of its inferred type.
- `num_threads`: The number of parser threads to spawn. The default
is the number of cores.
- `allow_quoted_newlines`: Allows multi-line text fields. This
is turned off by default.
- `no_header`: Do not auto-detect the presence of a header. Assume
the first line is data. This is turned off by default.
- `max_level_name_length`: If a field's length exceeds this value,
the entire column is treated as text rather than
categorical. The default is unlimited.
- `max_levels`: The maximum number of levels of a categorical column.
(default=max integer)
- `number_only`: Whether it can be safely assumed the columns only
contain numbers. This is turned off by default.
- `block_size`: The number of bytes to read at a time in each worker.
(default=32768)
ParaText is a work-in-progress. There are a few unimplemented features that may prevent it from working on all CSV files. We note them below.
-
ParaText does not yet support escape characters or comments.
-
There is no way to supply type hints (e.g.
uint64orfloat) of a column. Only the interpretation of a column (numeric, categorical, or text) can be forced. -
DateTime support will be added in a future release.