3.3 (26/04/2019)

 * Fix buggy test that breaks CRAN.                   (Manuel Lpez-Ibez)
                                                        
 * Do not print "23:59:59" when wall-clock time is actually close to zero.
                                                        (Manuel Lpez-Ibez)

3.2 (24/04/2019)

 * Fix `irace --check --parallel 2` on Windows.         (Manuel Lpez-Ibez)
                                                         
 * Values of real-valued parameter are now printed with sufficient precision to
   satisfy `digits` (up to `digits=15`).
                                                        (Manuel Lpez-Ibez)
                                                        
 * It is possible to specify boundMax without capping.
                                    (Leslie Prez Cceres, Manuel Lpez-Ibez) 

 * `irace --check` will exit with code 1 if the check is unsuccessful
                                                        (Manuel Lpez-Ibez)

 * Print where irace is installed with `--help`.          (Manuel Lpez-Ibez)
                                                        
 * irace will now complain if the output of target-runner or target-evaluator
   contains extra lines even if the first line of output is correct. This is to
   avoid parsing the wrong output. Unfortunately, this may break setups that
   relied on this behavior. The solution is to only print the output that irace
   expects. 
                                                        (Manuel Lpez-Ibez)
   
 * Completely re-implement `log` parameters to fix several bugs. Domains that
   contain zero or negative values are now rejected.
                                    (Leslie Prez Cceres, Manuel Lpez-Ibez)
                                    
 * New option `aclib=` (`--aclib 1`) enables compatibility with the
   GenericWrapper4AC (https://github.com/automl/GenericWrapper4AC/) used by
   AClib (http://aclib.net/). This is EXPERIMENTAL.
   `--aclib 1` also sets digits to 15 for compatibility with AClib defaults.
                                                         (Manuel Lpez-Ibez)
                                                         
 * Fix printing of output when capping is enabled.
                                                         (Manuel Lpez-Ibez)
 
 * `checkTargetFiles()` (`--check`) samples an instance unless
   `sampleInstances` is FALSE.                           (Manuel Lpez-Ibez)
   
 * Fix symbol printed in elimination test.               (Manuel Lpez-Ibez)

 * Use `dynGet()` to find `targetRunner` and `targetEvaluator`.
   As a result, we now require R >= 3.2.
                                                         (Manuel Lpez-Ibez)
 
 * All tests now use `testthat`.                         (Manuel Lpez-Ibez)
 
 * New function `scenario.update.paths()`                (Manuel Lpez-Ibez)
 
 * Fix assert failure that may happen when `elitistNewInstances` is larger than
   `firstTest`. Reported by Jose Riveaux.                (Manuel Lpez-Ibez)
   
 * Fix bug in `checkTargetFiles()` (`--check`) with capping.
                                                        (Leslie Prez Cceres)

 * Clarify a few errors/warnings when `maxTime` > 0.
                     (Manuel Lpez-Ibez, suggested by Haroldo Gambini Santos) 
 
 
3.1  (12/07/2018)

 * Use testthat for unit testing.                        (Manuel Lpez-Ibez)

 * Allow instances to be a list of arbitrary R objects (`mlr` bugfix).
                                                         (Manuel Lpez-Ibez)
 
3.0  (05/07/2018)

 * irace now supports adaptive capping for computation time minimization.
   The default value of the testType option is t-test when adaptive capping
   is enabled. Please see the user-guide for details.
                                    (Leslie Prez Cceres, Manuel Lpez-Ibez)

 * The package contains an `ablation()` function implementing the ablation
   method for parameter importance analysis by Fawcett and Hoos (2016).
                                     (Leslie Prez Cceres, Manuel Lpez-Ibez)

 * New option `postselection` executes a post-selection race.
                                                        (Leslie Prez Cceres)

 * At the end of each race, if the race stops before evaluating all instances
   seen in previous races, then the best overall may be different than the best
   of the race. We now print the best overall (best-so-far). Elites evaluated
   on more instances are considered better than those evaluated on fewer.
                                    (Manuel Lpez-Ibez, Leslie Prez Cceres)

 * Last active parameter values of numerical parameters (`i` and `r`) are carried
   by the sampling model. When a value must be assigned and the parameter was
   previously not active, the sampling is performed around the last value.
                                   (Leslie Prez Cceres, Manuel Lpez-Ibez)

 * R help pages are now generated with Roxygen2.
                                   (Leslie Prez Cceres, Manuel Lpez-Ibez)

 * The user guide documents --version, --help, and --check.
                                                        (Manuel Lpez-Ibez)
 
 * A return value of `Inf` from `targetRunner`/`targetEvaluation` results in
   the immediate rejection of the configuration without any further evaluation.
   This is useful for handling unreliable or broken configurations that should
   not stop irace.                                        (Manuel Lpez-Ibez)

 * Numerical parameters may be sampled on a logarithmic scale using `i,log`
   or `r,log`.                                               (Alberto Franzin)
 
 * New `target-runner.bat` for Windows contributed by Andr de Souza Andrade.
  
 * Fixed all shell scripts calling functions before defining them, which is not
   portable.
                                                         (Manuel Lpez-Ibez)

 * Fixed `--parallel` bug in Windows that resulted in
   `Error in checkForRemoteErrors(val)`.
                                                          (Manuel Lpez-Ibez)

 * Improve error message when no training instances are given.
                                                          (Manuel Lpez-Ibez)


2.4 (03/08/2017)

 * The output of irace now specifies in which order, if any, configurations are
   printed.
                              (Manuel Lpez-Ibez, suggested by Markus Wagner)
 
 * Several fixes for handling paths in Windows.
                                                     (Manuel Lpez-Ibez)
                                                     
 * readConfigurationsFile() now has a text= argument, which allows reading
   configurations from a string.
                                                     (Manuel Lpez-Ibez)

 * User-provided functions (targetRunner, targetEvaluator and
   repairConfiguration) and user-provided conditions for forbidden
   configurations are now byte-compiled when read, which should make their
   evaluation noticeably faster.
                                                      (Manuel Lpez-Ibez)

 * The argument 'experiment' passed to the R function targetRunner does not
   contain anymore an element 'extra.params'. Similarly, the 'scenario'
   structure does not contain anymore the elements 'instances.extra.params' and
   'testInstances.extra.params'. Any instance-specific parameters values now
   form part of the character string that defines an instance and it is up to
   the user-defined targetRunner to parse them appropriately. These changes
   make no difference when targetRunner is an external script, or when
   instances and instance-specific parameter values are read from a file.
                                                        (Manuel Lpez-Ibez)

2.3

 * Fix bug that will cause iraceResults$experimentLog to count calls to
   targetEvaluator as experiments, even if no call to targetRunner was
   performed. This does not affect the computation of the budget consumed and,
   thus, it does not affect the termination criteria of irace. The bug triggers
   an assertion that terminates irace, thus no run that was successful with
   version 2.2 is affected.
                                                     (Manuel Lpez-Ibez)

2.2

 * Command-line parameters are printed to stdout (useful for future
   replications). (Manuel Lpez-Ibez, suggested by Markus Wagner)

 * Users may provide a function to repair configurations before being
   evaluated. See the scenario variable repairConfiguration.
                                                     (Manuel Lpez-Ibez)
 
 * The option --sge-cluster (sgeCluster) was removed and replaced by
   --batchmode (batchmode). It is now the responsibility of the target-runner
   to parse the output of the batch job submission command (e.g., qsub or
   squeue), and return just the job ID. Values supported are: "sge", "torque",
   "pbs" and "slurm".                                     (Manuel Lpez-Ibez)

 * The option --parallel can now be combined with --batchmode to limit the
   number of jobs submitted by irace at once. This may be useful in batch
   clusters that have a small queue of jobs.
                                                          (Manuel Lpez-Ibez)

 * New examples under inst/examples/batchmode-cluster/.
                                                       (Manuel Lpez-Ibez)

 * It is now possible to include scenario definition files from other scenario
   files by using:
   
      eval.parent(source("scenario-common.txt", chdir = TRUE, local = TRUE))

   This feature is VERY experimental and the syntax is likely to change in the
   future.                                             (Manuel Lpez-Ibez)
 
 * Fix a bug that re-executed elite results under some circumstances.
   (Leslie Prez Cceres)

 * Restrict the number of maximum configurations per race to 1024.
   (Leslie Prez Cceres)
   
 * Do not warn if the last line in the instance file does not terminate with a
   newline. (Manuel Lpez-Ibez)
 
 * Fix bug when deterministic == 1.
   (Manuel Lpez-Ibez, Leslie Prez Cceres)

 * Update manual and vignette with details about the expected arguments and
   return value of targetRunner and targetEvaluator. (Manuel Lpez-Ibez)

 * Many updates to the User Guide vignette. (Manuel Lpez-Ibez)

 * Fix \dontrun example in irace-package.Rd (Manuel Lpez-Ibez)

 * Fix bug: If testInstances contains duplicates, results of testing are not
   correctly saved in iraceResults$testing$experiments nor reported correctly
   at the end of a run. Now unique IDs of the form 1t, 2t, ... are used for
   each testing instance. These IDs are used for the rownames of
   iraceResults$testing$experiments and the names of the scenario$testInstances
   and iraceResults$testing$seeds vectors.  (Manuel Lpez-Ibez)

 * Fix bug where irace keeps retrying the target-runner call even if it
   succeeds. (Manuel Lpez-Ibez)

 * New command-line parameter

       --only-test FILE

   which just evaluates the configurations given in FILE on the testing
   instances defined by the scenario. Useful if you decide on the testing
   instances only after running irace.    (Manuel Lpez-Ibez)

 * Bugfix: When using maxTime != 0, the number of experiments performed may be
   miscounted in some cases.              (Manuel Lpez-Ibez)


2.1

 * Fix CRAN errors in tests. (Manuel Lpez-Ibez)
 
 * Avoid generating too many configurations at once if the initial time
   estimation is too small. (Manuel Lpez-Ibez)
 
2.0

 * Minimum R version is 2.15. 

 * Elitist irace by default, it can be disabled with parameter --elitist 0. 
   (Leslie Prez Cceres, Manuel Lpez-Ibez)

 * The parameter --test-type gains two additional values:

     t-test-bonferroni (t-test with Bonferroni's correction for multiple
                        comparisons), 
     t-test-holm (t-test with Holm's correction for multiple comparisons)

   (Manuel Lpez-Ibez)

 * MPI does not create log files with --debug-level 0.
   (Manuel Lpez-Ibez)

 * For simplicity, the parallel-irace-* scripts do not use an auxiliary
   `tune-main` script.  For customizing them, make a copy and edit them
   directly.
   (Manuel Lpez-Ibez)

 * New parameters:
   
   --target-runner-retries : Retry target-runner this many times in case
                             of error. 
                                                   (Manuel Lpez-Ibez)

 * We print diversity measures after evaluating on each instance:
   (Leslie Prez Cceres)

   - Kendall's W (also known as Kendall's coefficient of concordance) If 1,
     all candidates have ranked in the same order in all instances.  If 0, the
     ranking of each candidate on each instance is essentially random.  

             W = Friedman / (m * (k-1))
   
   - Spearman's rho: average (Spearman) correlation coefficient computed on the
     ranks of all pairs of raters. If there are no repeated data values, a
     perfect Spearman correlation of +1 or -1 occurs when each of the variables
     is a perfect monotone function of the other.

 * Many internal and external interfaces have changed. For example, now we
   consistently use 'scenario' to denote the settings passed to irace and
   'configuration' instead of 'candidate' to denote the parameter settings
   passed to the target algorithm. Other changes are:

   parameters$boundary -> parameters$domain
   hookRun             -> targetRunner
   hookEvaluate        -> targetEvaluator
   tune-conf           -> scenario.txt
   instanceDir         -> trainInstancesDir
   instanceFile        -> trainInstancesFile
   testInstanceDir     -> testInstancesDir
   testInstanceFile    -> testInstancesFile

 * Minimal example of configuring a MATLAB program
   (thanks to Esteban Diaz Leiva)

 * Paths to files or directories given in the scenario file are relative to the
   scenario file (except for --log-file, which is an output file and it is
   relative to --exec-dir). Paths given in the command-line are relative to the
   current working directory. Given

      $ cat scenario/scenario.txt
      targetRunner <- "./target-runner"
      $ irace -s scenario/scenario.txt

   irace will search for "./scenario/target-runner", but given

      $ irace -s scenario/scenario.txt --target-runner ./target-runner
   
   irace will search for "./target-runner".      (Manuel Lpez-Ibez)

 * New command-line wrapper for Windows installed at
   'system.file("bin/irace.bat", package="irace")'
   (thanks to Anthony Antoun)                                        

 * Budget can be specified as maximum time (maxTime, --max-time) consumed by
   the target algorithm. See the documentation for the details about how this
   is handled.
   (Leslie Prez Cceres, Manuel Lpez-Ibez)


1.07

 * The best configurations found, either at the end or at each iteration of an
   irace run, can now be applied to a set of test instances different from the
   training instances. See options testInstanceDir, testInstanceFile,
   testNbElites, and testIterationElites. (Leslie Prez Cceres, Manuel Lpez-Ibez)

 * The R interfaces of hookRun, hookEvaluate and hookRunParallel have changed.
   See help(hook.run.default) and help(hook.evaluate.default) for examples of
   the new interfaces.

 * Printing of race progress now reports the actual configuration and instance
   IDs, and numbers are printed in a more human-readable format.
   (Leslie Prez Cceres, Manuel Lpez-Ibez)

 * Reduce memory use for very large values of maxExperiments.  
   (Manuel Lpez-Ibez, thanks to Federico Caselli for identifying the issue)

 * New option --load-balancing (loadBalancing) for disabling load-balancing
   when executing jobs in parallel. Load-balancing makes better use of
   computing resources, but increases communication overhead. If this overhead
   is large, disabling load-balancing may be faster.
   (Manuel Lpez-Ibez, thanks to Federico Caselli for identifying the issue)

 * The option --parallel in Windows now uses load-balancing by default.
   (Manuel Lpez-Ibez)

 * The wall-clock time after finishing each task is printed in the output.
   (Manuel Lpez-Ibez, thanks to Federico Caselli for providing an initial
   patch)



1.06

 * Fix bug that could introduce spurious whitespace when printing the
   final configurations. (Manuel Lpez-Ibez)

 * Fix bug if there are more initial candidates than needed for the
   first race. (Leslie Prez Cceres, Manuel Lpez-Ibez)

 * New configuration options, mainly for R users:
                    
   - hookRunParallel: Optional R function to provide custom
     parallelization of hook.run.

   - hookRunData: Optional data passed to hookRun. This is ignored by
     the default hookRun function, but it may be used by custom hookRun R
     functions to pass persistent data around.
   (Manuel Lpez-Ibez)

1.05

 * New option --version. (Manuel Lpez-Ibez)

 * Terminate early if there is no sufficient budget to run irace with
   the given settings. (Manuel Lpez-Ibez)

 * The option --parallel (without --mpi) now works under Windows.
   (Manuel Lpez-Ibez, thanks to Pablo Valledor Pellicer for testing
   it)

 * Improved error handling when running under Rmpi. Now irace will
   terminate as soon as the master node detects at least one failed
   slave node. This avoids irace reporting two times the same error.
   Also, irace will print all the unique errors returned by all slaves
   and not just the first one.
   (Manuel Lpez-Ibez)

 * Forbidden configurations may be specified in terms of constraints
   on their values. Forbidden configurations will never be evaluated by irace. 
   See --forbidden-file and inst/templates/forbidden.tmpl.
   (Manuel Lpez-Ibez)

 * New option --recovery-file (recoveryFile) allows resuming a
   previous irace run. (Leslie Prez Cceres)

 * The confidence level for the elimination test is now
   configurable with parameter --confidence. (Leslie Prez Cceres)

 * Much more robust handling of relative/absolute paths. Improved support
   for Windows. (Leslie Prez Cceres, Manuel Lpez-Ibez)

 * Provide better error messages for incorrect parameter
   descriptions. (Manuel Lpez-Ibez)
    Examples:

    x "" i (0, 0)       # lower and upper bounds are the same
    x "" r (1e-4, 5e-4) # given digits=2, ditto
    x "" i (-1, -2)     # lower bound must be smaller than upper bound
    x "" c ("a", "a")   # duplicated values
 
 * Print elapsed time for calls to hook-run if debugLevel >=1.
   (Manuel Lpez-Ibez)

 * examples/hook-run-python/hook-run: A multi-purpose hook-run written
   in Python. (Franco Mascia)

 * Parallel mode in an SGE cluster (--sge-cluster) is more
   robust. (Manuel Lpez-Ibez)

1.04

 * Replace obsolete package multicore by package parallel
   (requires R >= 2.14.0)

 * Use load-balancing (mc.preschedule = FALSE) in mclapply.

1.03

 * Use reg.finalizer to finish Rmpi properly without clobbering
   .Last().

 * Remove uses of deprecated as.real().

 * Nicer error handling in readParameters.

 * Add hypervolume (multi-objective) example.

 * Fix several bugs in the computation of similar candidates.

1.02

 * More concise output.

 * The parameters expName and expDescription are now useless and they
   were removed.

 * Faster computation of similar candidates (Jeremie Dubois-Lacoste
   and Leslie Prez Cceres).

 * Fix bug when saving instances in tunerResults$experiments.

 * irace.cmdline ("--help") does not try to quit R anymore.

1.01

 * Fix bug caused by file.exists (and possibly other functions)
   not handling directory names with a trailing backslash or slash on
   Windows.
 
 * Fix bug using per-instance parameters (Leslie Prez Cceres).

 * Fix bug when reading initial candidates from a file.
