a train-on-error simulator for use with dbacl.
automates the task of testing email filtering and
classification programs such as dbacl(1). Given a set
of categorized documents, mailtoe initiates test runs to
estimate the classification errors and thereby permit fine
tuning of the parameters of the classifier.
(TOE) is a learning method which is sometimes advocated for
email classifiers. Given an incoming email stream, the
method consists in reusing a fixed set of category databases
until the first misclassification occurs. At that point, the
offending email is used to relearn the relevant category,
until the next misclassification. In this way, categories
are only updated when errors occur. This directly models the
way that some email classifiers are used in practice.
error rates depend directly on the order in which emails are
seen. A small change in ordering, as might happen due to
networking delays, can have a large impact on the number of
misclassifications. Consequently, mailtoe does not
give meaningful results, unless the sample emails are chosen
carefully. However, as this method is commonly used by spam
filters, it is still worth computing to foster comparisons.
Other methods (see mailcross(1),mailfoot(1))
attempt to capture the behaviour of classification errors in
To improve and
stabilize the error rate calculation, mailtoe
performs the TOE simulations several times on slightly
reordered email streams, and averages the results. The
reorderings occur by multiplexing the emails from each
category mailbox in random order. Thus if there are three
categories, the first email classified is chosen randomly
from the front of the sample email streams of each type. The
second email is also chosen randomly among the three types,
from the front of the
streams after the first email was removed. Simulation stops
when all sample streams are exhausted.
uses the environment variable MAILTOE_FILTER when executing,
which permits the simulation of arbitrary filters, provided
these satisfy the compatibility conditions stated in the
ENVIRONMENT section below.
convenience, mailtoe implements a testsuite
framework with predefined wrappers for several open source
classifiers. This permits the direct comparison of
dbacl(1) with competing classifiers on the same set
of email samples. See the USAGE section below.
preparation, mailtoe builds a subdirectory named
mailtoe.d in the current working directory. All needed
calculations are performed inside this subdirectory.
returns 0 on success, 1 if a problem occurred.
Prepares a subdirectory named
mailtoe.d in the current working directory, and populates it
with empty subdirectories for exactly size
add category [
Takes a set of emails from
either FILE if specified, or STDIN, and associates
them with category. The ordering of emails within
FILE is preserved, and subsequent FILEs are
appended to the first in each category. This command can be
repeated several times, but should be executed at least
Deletes the directory mailtoe.d and all its
Multiplexes randomly from the email streams added
earlier, and relearns categories only when a
misclassification occurs. The simulation is repeated
Prints average error rates for
plot [ ps |
Plots the number of errors over
simulation time. The "ps" option, if present,
writes the plot to a postscript file in the directory
mailtoe/plots, instead of being shown on-screen. The
"logscale" option, if present, causes the plot to
be on the log scale for both ordinates.
Scans the last run statistics
and extracts all the messages which belong to category
truecat but have been classified into category
predcat. The extracted messages are copied to the
directory mailtoe.d/review for perusal.
Shows a list of available
filters/wrapper scripts which can be selected.
testsuite select [
Prepares the filter(s) named
FILTER to be used for simulation. The filter name is
the name of a wrapper script located in the directory
@PKGDATADIR@/testsuite. Each filter has a rigid
interface documented below, and the act of selecting it
copies it to the mailtoe.d/filters directory. Only
filters located there are used in the simulations.
testsuite deselect [
Removes the named filter(s)
from the directory mailtoe.d/filters so that they are
not used in the simulation.
testsuite run [
Invokes every selected filter
on the datasets added previously, and calculates
misclassification rates. If the "plots" option is
present, each filter simulation is plotted as a postscript
file in the directory mailtoe.d/plots.
Describes the scheduled
Shows the cross validation
results for all filters. Only makes sense after the
usage pattern is the following: first, you should separate
your email collection into several categories (manually or
otherwise). Each category should be associated with one or
more folders, but each folder should not contain more than
one category. Next, you should decide how many runs to use,
say 10. The more runs you use, the better the predicted
error rates. However, more runs take more time. Now you can
Next, for every
category, you must add every folder associated with this
category. Suppose you have three categories named
spam, work, and play, which are
associated with the mbox files spam.mbox,
work.mbox, and play.mbox respectively. You
% mailtoe add
% mailtoe add work work.mbox
% mailtoe add play play.mbox
You should aim for a similar number of emails in each
category, as the random multiplexing will be unbalanced
otherwise. The ordering of the email messages in each
file is important, and is
preserved during each simulation. If you repeatedly add to
the same category, the later mailboxes will be appended to
the first, preserving the implied ordering.
You can now
perform as many TOE simulations as desired. The multiplexed
emails are classified and learned one at a time, by
executing the command given in the environment variable
MAILTOE_FILTER. If not set, a default value is used.
% mailtoe run
% mailtoe summarize
commands are designed to simplify the above steps and allow
comparison of a wide range of email classifiers, including
but not limited to dbacl. Classifiers are supported
through wrapper scripts, which are located in the
The first stage
when using the testsuite is deciding which classifiers to
compare. You can view a list of available wrappers by
Note that the
wrapper scripts are NOT the actual email classifiers, which
must be installed separately by your system administrator or
otherwise. Once this is done, you can select one or more
wrappers for the simulation by typing, for example:
testsuite select dbaclA ifile
If some of the
selected classifiers cannot be found on the system, they are
not selected. Note also that some wrappers can have
hard-coded category names, e.g. if the classifier only
supports binary classification. Heed the warning
It remains only
to run the simulation. Beware, this can take a long time
(several hours depending on the classifier).
% mailtoe testsuite summarize
Once you are
all done, you can delete the working files, log files etc.
testsuite takes care of learning and classifying your
prepared email corpora for each selected classifier. Since
classifiers have widely varying interfaces, this is only
possible by wrapping those interfaces individually into a
standard form which can be used by mailtoe
script is a command line tool which accepts a single command
followed by zero or more optional arguments, in the standard
script also makes use of STDIN and STDOUT in a well defined
way. If no behaviour is described, then no output or input
should be used. The possible commands are described
In this case, a single email is expected on STDIN, and a
list of category filenames is expected in $2, $3, etc. The
script writes the category name corresponding to the input
email on STDOUT. No trailing newline is required or
In this case, a standard mbox stream is expected on
STDIN, while a suitable category file name is expected in
$2. No output is written to STDOUT.
In this case, a directory is expected in $2, which is
examined for old database information. If any old databases
are found, they are purged or reset. No output is written to
IN this case, a single line of
text is written to STDOUT, describing the filter’s
functionality. The line should be kept short to prevent line
wrapping on a terminal.
In this case, a directory is
expected in $2. The wrapper script first checks for the
existence of its associated classifier, and other
prerequisites. If the check is successful, then the wrapper
is cloned into the supplied directory. A courtesy
notification should be given on STDOUT to express success or
failure. It is also permissible to give longer descriptions
In this case, a list of categories is expected in $3,
$4, etc. Every possible category must be listed. Preceding
this list, the true category is given in $2.
Used by mailfoot(1).
loading, mailtoe reads the hidden file .mailtoerc in
the $HOME directory, if it exists, so this would be a good
place to define custom values for environment variables.
This variable contains a shell
command to be executed repeatedly during the running stage.
The command should accept an email message on STDIN and
output a resulting category name. On the command line, it
should also accept first the true category name, then a list
of all possible category file names. If the output category
does not match the true category, then the relevant
categories are assumed to have been silently
updated/relearned. If MAILTOE_FILTER is undefined,
mailtoe uses a default value.
This directory is exported for
the benefit of wrapper scripts. Scripts which need to create
temporary files should place them a the location given in
subdirectory mailtoe.d can grow quite large. It contains a
full copy of the training corpora, as well as learning files
for size times all the added categories, and various
simulations for dbacl(1) can be used to compare with
other classifiers, TOE should not be used for real world
classifications. This is because, unlike many other filters,
dbacl(1) learns evidence weights in a nonlinear way,
and does not preserve relative weights between tokens, even
if those tokens aren’t seen in new emails.
ordering of emails within the added mailboxes matters, the
estimated error rates are not well defined or even
meaningful in an objective sense. However, if the sample
emails represent an actual snapshot of a user’s
incoming email, then the error rates are somewhat
meaningful. The simulations can then be interpreted as
alternate realities where a given classifier would have
intercepted the incoming mail.
The source code
for the latest version of this program is available at the
Laird A. Breyer
dbacl(1), mailinspect(1), mailcross(1),