− a cross-validation simulator for use with dbacl.
automates the task of cross-validating email filtering and
classification programs such as dbacl(1). Given a set
of categorized documents, mailcross initiates simulation
runs to estimate the classification errors and thereby
permits fine tuning of the parameters of the classifier.
is a method which is widely used to compare the quality of
classification and learning algorithms, and as such permits
rudimentary comparisons between those classifiers which make
use of dbacl(1) and bayesol(1), and other
of cross-validation are as follows: A set of pre-classified
email messages is first split into a number of roughly
equal-sized subsets. For each subset, the filter (by
default, dbacl(1)) is used to classify each message
within this subset, based upon having learned the categories
from the remaining subsets. The resulting classification
errors are then averaged over all subsets.
obtained by cross validation essentially do not depend upon
the ordering of the sample emails. Other methods (see
mailtoe(1),mailfoot(1)) attempt to capture the
behaviour of classification errors over time.
uses the environment variables MAILCROSS_LEARNER and
MAILCROSS_FILTER when executing, which permits the
cross-validation of arbitrary filters, provided these
satisfy the compatibility conditions stated in the
ENVIRONMENT section below.
convenience, mailcross implements a testsuite
framework with predefined wrappers for several open source
classifiers. This permits the direct comparison of
dbacl(1) with competing classifiers on the same set
of email samples. See the USAGE section below.
preparation, mailcross builds a subdirectory named
mailcross.d in the current working directory. All needed
calculations are performed inside this subdirectory.
returns 0 on success, 1 if a problem occurred.
Prepares a subdirectory named
mailcross.d in the current working directory, and populates
it with empty subdirectories for exactly size
Takes a set of emails from
either FILE if specified, or STDIN, and associates them with
category. All emails are distributed randomly into
the subdirectories of mailcross.d for later use. For each
category, this command can be repeated several times,
but should be executed at least once.
Deletes the directory mailcross.d and all its
For every previously built subset of email messages,
pre-learns all the categories based on the contents of all
the subsets except this one. The command_arguments
are passed to MAILCROSS_LEARNER.
For every previously built subset of email messages,
performs the classification based upon the pre-learned
categories associated with all but this subset. The
command_arguments are passed to MAILCROSS_FILTER.
Prints statistics for the
latest cross-validation run.
Scans the last run statistics
and extracts all the messages which belong to category
truecat but have been classified into category
predcat. The extracted messages are copied to the
directory mailcross.d/review for perusal.
Shows a list of available
filters/wrapper scripts which can be selected.
Prepares the filter(s) named
FILTER to be used for simulation. The filter name is
the name of a wrapper script located in the directory
@PKGDATADIR@/testsuite. Each filter has a rigid
interface documented below, and the act of selecting it
copies it to the mailcross.d/filters directory. Only
filters located there are used in the simulations.
Removes the named filter(s)
from the directory mailcross.d/filters so that they
are not used in the simulation.
Invokes every selected filter
on the datasets added previously, and calculates
Describes the scheduled
Shows the cross validation
results for all filters. Only makes sense after the
usage pattern is the following: first, you should separate
your email collection into several categories (manually or
otherwise). Each category should be associated with one or
more folders, but each folder should not contain more than
one category. Next, you should decide how many subsets to
use, say 10. Note that too many subsets will slow down the
calculations rapidly. Now you can type
Next, for every
category, you must add every folder associated with this
category. Suppose you have three categories named
spam, work, and play, which are
associated with the mbox files spam.mbox,
work.mbox, and play.mbox respectively. You
% mailcross add
% mailcross add work work.mbox
% mailcross add play play.mbox
You can now
perform as many simulations as desired. Every cross
validation consists of a learning, a running and a
summarizing stage. These operations are performed on the
classifier specified in the MAILCROSS_FILTER and
MAILCROSS_LEARNER variables. By setting these variables
appropriately, you can compare classification performance as
you vary the command line options of your classifier(s).
% mailcross run
% mailcross summarize
The testsuite commands are designed to simplify the
above steps and allow comparison of a wide range of email
classifiers, including but not limited to dbacl.
Classifiers are supported through wrapper scripts, which are
located in the @PKGDATADIR@/testsuite
The first stage
when using the testsuite is deciding which classifiers to
compare. You can view a list of available wrappers by
Note that the
wrapper scripts are NOT the actual email classifiers, which
must be installed separately by your system administrator or
otherwise. Once this is done, you can select one or more
wrappers for the simulation by typing, for example:
testsuite select dbaclA ifile
If some of the
selected classifiers cannot be found on the system, they are
not selected. Note also that some wrappers can have
hard-coded category names, e.g. if the classifier only
supports binary classification. Heed the warning
It remains only
to run the simulation. Beware, this can take a long time
(several hours depending on the classifier).
% mailcross testsuite summarize
Once you are
all done with simulations, you can delete the working files,
log files etc. by typing
The progress of
the cross validation is written silently in various log
files which are located in the mailcross.d/log
directory. Check these in case of problems.
testsuite takes care of learning and classifying your
prepared email corpora for each selected classifier. Since
classifiers have widely varying interfaces, this is only
possible by wrapping those interfaces individually into a
standard form which can be used by mailcross
script is a command line tool which accepts a single command
followed by zero or more optional arguments, in the standard
script also makes use of STDIN and STDOUT in a well defined
way. If no behaviour is described, then no output or input
should be used. The possible commands are described
In this case, a single email is expected on STDIN, and a
list of category filenames is expected in $2, $3, etc. The
script writes the category name corresponding to the input
email on STDOUT. No trailing newline is required or
In this case, a standard mbox stream is expected on
STDIN, while a suitable category file name is expected in
$2. No output is written to STDOUT.
In this case, a directory is expected in $2, which is
examined for old database information. If any old databases
are found, they are purged or reset. No output is written to
IN this case, a single line of
text is written to STDOUT, describing the filter’s
functionality. The line should be kept short to prevent line
wrapping on a terminal.
In this case, a directory is
expected in $2. The wrapper script first checks for the
existence of its associated classifier, and other
prerequisites. If the check is successful, then the wrapper
is cloned into the supplied directory. A courtesy
notification should be given on STDOUT to express success or
failure. It is also permissible to give longer descriptions
Used by mailtoe(1).
Used by mailfoot(1).
loading, mailcross reads the hidden file .mailcrossrc
in the $HOME directory, if it exists, so this would be a
good place to define custom values for environment
This variable contains a shell
command to be executed repeatedly during the running stage.
The command should accept an email message on STDIN and
output a resulting category name. It should also accept a
list of category file names on the command line. If
undefined, mailcross uses the default value
MAILCROSS_FILTER="dbacl -T email -T xml -v" (and
also magically adds the -c option before each category).
This variable contains a shell
command to be executed repeatedly during the learning stage.
The command should accept a mbox type stream of emails on
STDIN for learning, and the file name of the category on the
command line. If undefined, mailcross uses the
default value MAILCROSS_LEARNER="dbacl -H 19 -T email
-T xml -l".
This directory is exported for
the benefit of wrapper scripts. Scripts which need to create
temporary files should place them a the location given in
subdirectory mailcross.d can grow quite large. It contains a
full copy of the training corpora, as well as learning files
for size times all the added categories, and various
is a widely used, but ad-hoc statistical procedure,
completely unrelated to Bayesian theory, and subject to
controversy. Use this at your own risk.
The source code
for the latest version of this program is available at the
Laird A. Breyer
dbacl(1), mailinspect(1), mailtoe(1),