INTRODUCTION:
JMLRExperiment is a cross platform program specifically written to perform Breiman's random split experiments for Forest-RI (one valid random feature per split) and three variants of Artificial Prediction Market.

Building:
1. Download and install CMake.
2. Run cmake
- Windows: cmake-gui
- UNIX-like: ccmake
3. Point the source directory to this directory.
4. Choose a different directory as a build directory.
5. Configure and choose a suitable compiler.
- Windows: Press 'Configure' button
- UNIX-like: Press 'c'
6. Configure again and then generate.
- Windows: Press 'Generate' button
- UNIX-like: Press 'g'
7. Go to the build directory you specified and build.
- Windows: For Visual Studio, open the solution and change 'Debug' to 'Release' then select 'Build'->'Build Solution' in the file bar. The resulting executable (JMLRExperiment.exe) is placed in the 'Release' directory of the build directory.
- UNIX-like: Run make(1).

USAGE:
usage: JMLRExperiment -d dataRoot [-l dataSetList] -o outputRoot -n numSplits -r trainingRatio -e numEpochs [-s seed] [dataset1 dataset2 ...]

-d dataRoot:
The data root is the root directory where all data sets are stored. Each data set is stored in its own directory.
For example:
/path/to/dataRoot/dataSet1/
C:\path\to\dataRoot\dataSet1\

-l dataSetList:
This is an optional flag. It specifies a list of data set directories to process in batch.

NOTE: This flag may be used to supplement data sets manually specified on the command line.

-o outputRoot:
Specify the root output directory. Output is stored in data set-dedicated directories in the output root directory.
For example:
/path/to/outputRoot/dataSet1/Forest-train.txt
C:\path\to\outputRoot\dataSet1\Forest-train.txt

-n numSplits:
The number of random splits to perform. Breiman uses 100.

-r trainingRatio:
The ratio of the training set to the test set. Breiman uses 0.9 (90%).

NOTE: Random splits are not used on data sets that provide test sets.

-e numEpochs:
The number of epochs to run each Market. We use 50.

-s seed:
The random seed. This is optional (defaults to 1). The seed is set for each data set processed in batch.

OUTPUT:
Output are organized in data set-dedicated directories in the specified output root directory. The results stored are:
Forest-train.txt - Random Forest training errors.
Forest-test.txt - Random Forest test errors:
ConstantMarket-train.txt - Constant Market training errors.
ConstantMarket-test.txt - Constant Market testing errors.
LinearMarket-train.txt - Linear Market training errors.
LinearMarket-test.txt - Linear Market testing errors.
AggressiveMarket-train.txt - Aggressive Market training errors.
AggressiveMarket-test.txt - Aggressive Market testing errors.

Forest Error File Format:
Errors for each random split are specified on one line delimited by spaces.

Market Error File Format:
Errors for each epoch for one split are stored on one line delimited by spaces. There are numSplit such lines.

Data Set File Format:
Data sets are processed into two files X.dat and Y.dat. X.dat stores instances space-delimited and line oriented (one instance per line). All attributes are assumed numerical (map categorical features to integers). Y.dat stores the corresponding label line oriented. Labels are assumed to be integers beginning at 0 and ending at numLabels-1.

These are stored in
/path/to/dataRoot/dataSet1/X.dat
/path/to/dataRoot/dataSet1/Y.dat
C:\path\to\dataRoot\dataSet1\X.dat
C:\path\to\dataRoot\dataSet1\Y.dat

JMLRExperiment will also look for test sets Xtest.dat and Ytest.dat.

I recommend using AWK (the language of awk(1)) to convert data sets to this format.

ADVANCED:
The source code itself can be tuned. In particular are self-explanatory options specified in main.cpp. The functions of interest include:
main()
Experiment::operator()(const std::string &dataSet)
Experiment::GenerateFeatures()

Features may also be changed by changing the MyFeature typedef. For example, RandomCombinationFeature<> gives an approximation of CART (the difference being that these random linear combinations are generated once in advance). Other features and feature data are possible.

COMPILERS KNOWN TO WORK:
- Visual C++ 2010
- gcc 4.2
- clang 3.0

OTHER NOTES:
This code assumes the presence of stdint.h.

