testtoys
--------

texttoys is a test tool for text-based behavioral testing. 
It can be combined with JUnit for proper IDE integration. 



usage example:
--------------

Let's examine following unittest example:

    import org.junit.*;
    import static org.junit.Assert.*;
    import java.io.IOException;

    public class ExampleTest {	
        @Test
        public void testPlus() throws IOException  {
            TestTool t = new TestTool("testio/example/plus"); 
		
            t.tln("testing plus operation"); 
            t.tln("  1+1=" + (1+1));
  
            assertTrue(t.done());
        }
    }

Running the test first time will create following output:

    running testio/example/plus...
  
      testing plus operation
        1+1=2

    2 ms. [d]iff, [c]ontinue or [f]reeze?f
    frozen.

After freezing the behavior, the tool will create expectation file
with following content

    testing plus operation 
      1+1=2

If for some reason the result changes (e.g. 1+1=3), the 
test will fail. The tool will show on the margin the expected
result:

    running testio/example/plus...

      testing plus operation
    !   1+1=3                                         |  1+1=2

    1 ms. 1 errors! [d]iff, [c]ontinue or [f]reeze?

If the test is run in interactive mode, the developer can 
mark the new result as ok by [f]reezing it. [d]iff lets
user to see the difference between new and old results in 
Meld (if installed). 

To run non-interactive by default (can be overridden in code),
define the environment variable TESTTOYS_NEVER_FREEZE.

the benefits:
-------------

   * Enables rapid behavioral testing. The effort level
     of creating a unittest case is similar to printing 
     debug info with System.out.println();  

   * The benefit compared to traditional unit testing are:

       1. Greatly reduced effort and increased speed of creating tests

          * No precreated vectors containing right orders & results

          * No checking of each function result with custom comparison

          -> You get better test coverage faster and with less effort

       2. Fast comparison to old results: 

          * If you are measuring things without 'one right answer' 
            like performance, error/quality rates or orders, 
            you will immediately see, whether the new results are better
            than the old ones. 

          -> you can test things for which conventional unit tests
             would be clumsy and ineffective.

       3. Greatly reduced effort in maintaining the tests

          * No updating of result vectors or result objects
            Just select [f]reeze and you have new expected result.

          * This is HUGE deal, when developing software, which 
            doesn't create single correct results, but where results
            improve with every iteration. When improving results
            little by little it would be major pair to update
            parts of the unittest code, whenever results change. 

          -> you can maintain larger test sets cheaper

       4. The tests are self-documenting and the tool create in essense
          'formal documentation of the expected behavior'

          * The generated file can document itself by simply printing
            comments on the results. 

          * The expectation files form a natural documentation of the system's
            expected behavior. If you have performance tests, the expected 
            performance will be stored in the expectation files. If you have 
            error rates or e.g. results of stemmers: the expected error rates
            and tokens are there too. these results are immediately available 
            in human readable form without a need to run potentially slow tests
            (e.g. performance, scaling and stress tests may take a while). 

          -> it will improve your awareness of the system's behavior and
             performance in the most convenient way. 

   * While the tool is not perfect and the approach has its own inconveniences 
     (e.g. links to relevant code lines are missing), my own experience with
     this kind of testing has been hugely positive. It is extremely 
     convenient and productive way to do testing. It allows better coverage, 
     and it helps building better quality and better behaving software faster.
     It is also very convenient and easy way to do testing.



setup:
------

To install testtoys in your local repository, use: 

    sbt publishLocal

To use testtoys in your project, add following dependencies into
your sbt project:

    libraryDependencies ++= Seq(
      "com.futurice" %% "testtoys" % "0.1-SNAPSHOT" % "test",
      "junit" % "junit" % "4.8" % "test",
      "com.novocode" % "junit-interface" % "0.11" % Test
     )

First line adds the testtoy. Second adds the needed junit library 
and the third line adds sbt junit integration so that you 
can run the tests with:

    sbt test

To be able to see diff between old and new result, install meld: 

    sudo apt install meld



more on usage:
--------------

The first time the ExampleTest is run, it will generate
output in 
   
    testio/example/plus_out.txt

file. After you freeze the file it will be renamed as...

    testio/example/plus_exp.txt
 
If test fails, both plus_out.txt and plus_exp.txt will remain so 
you can later compare the differences and try identify the 
cause of failure.

If you create new files with the testtool, the files will 
be created in 

    testio/example/plus_out/file1.txt

kind of directory. This directory will be cleaned, when
the test is ran again.

After freezing the expectation files, you likely want to 
add testio/*/*_exp.txt files into your repository. 

Also you likely want make your clean script remove all
testio/*/*_out.txt files and testio/*/*_out directories. 

Also you likely want to add *_out.txt and *_out files into
your gitignore file.



more information:
-----------------

See the unit tests in src/test/java folder to see more examples
of the the tool usage.



acknowledgements:
-----------------

  Thanks for our customer for letting us (Futurice) publish 
  this tool as it was originally developed for their project
  and legally their IPR. 

  Thanks for Ferenc Dosa-Racz for showing the way and the 
  benefits of this approach with his ITK (integrated testkit). 

  This kind of testing is also available in texttest tool  
  developed by Geoff Bache and Daniel Galdames in: 

    http://texttest.sourceforge.net/

  So this kind of testing approach is not exactly a new idea. 
  This tool was created merely to provide similar kind of
  functionality in Java / JVM environment and to suit
  my individual tastes.



the way forward:
----------------

testtoys is not 100% production complete. It is simple and it works, 
but e.g. shipping the expectation text files within a test jar file 
is not supported. Without doubt, it also missing useful functionality
and it may have bugs.

One nice thing would be better annotations / integrations with IDE 
so that you could entirely skip TestTool creation and assertTrue(t.done()) 
check so that you would have. 

   @TestToy 
   public void testPlus(TestTool t) throws IOException  {
       t.tln("testing plus operation"); 
       t.tln("  1+1=" + (1+1));
   }

This would reduce typing and syntatic noise.

It would be great to have dynamically sized test areas. This could be 
especially useful if you printing in test files things like log, which 
content can be partly ignored:

  We are going to do X. It needs to happen without errors.

  Starting test...

  log: {
     10:20 Random thing happened
     10:21 Location of this may change
     10:23 X occurred 
     10:25 ERROR!!!
     10:26 This line may appear in log, or not 
  }

  found: {
    X occurred - ok 
    ERROR!!! - failed 
  }

In this case the area between '{' and '}' would form a scope, which
can have varying amount of lines.