Scala

ScaTDD: Casting an eye over three major Test frameworks in Scala

Testing has traditionally been the gateway process for Scala development at a number of Java shops looking to evolve their tech stack with minimal disruption or committment. This posts hopes to cover three of the main testing frameworks in the Scala landscape, ( Specs2, Scalatest and Scalacheck) with an example of the classic FizzBuzz test, for how they can be used for fun and profit. So, the brief outline of the FizzBuzz requirement are that given a sequence of numbers:

  • Any number divisible by 3 should return the String “Fizz”
  • Any number divisible by 5 should return the String “Buzz”
  • Any number divisible by both 3 and 5 should return the String “FizzBuzz”
  • Otherwise the number should be returned as a String

As a comparison, I’ve also included a sample Java JUnit test for this. The actual implementation code (included here) for this is trivial, (partly by design) as it is just to show illustrate the libraries being used.             

My setup for this was using Eclipse v3.7.2 on Windows 7 with the  Scala IDE v2.0.1.v-2_09 plugin and Scala version 2.9.0.1.

SPECS2

In a nutshell

  • A meta framework for writing unit and acceptance tests by specification. Supports concurrent execution, mocking, reporting, scalatest and scalacheck feature support, datatable style testing and form based specification tests. The propject emerged from the original, (and now deprecated) Scala Specs BDD testing framework.

Link

http://etorreborre.github.com/specs2/

Version

v2.9.2_v1.10

Ohloh

http://www.ohloh.net/p/specs2

Active community

Yes.A project initiated as Specs (v1) in mid 2007 and later evolving to a more general framework as Specs2 in 2010. The project has a number of comitters (though primarily driven by Eric Torreborre) and supporting resources such as a google group, blog and corporate backing for the project.

Example

 
package org.scalabound.scatdd.specs2

import org.scalabound.scatdd.FizzBuzz
import org.junit.runner.RunWith
import org.specs2.runner.JUnitRunner

// package unitTest { |> Scala IDE will not pick the class up as runnable if I sub-package it like this :-(
@RunWith(classOf[JUnitRunner])
class FizzBuzzJUnitSpec extends org.specs2.mutable.Specification {
    
  "Multiples of both three and five" should {
    "print 'FizzBuzz'" in { FizzBuzz.eval(15) must_== "FizzBuzz" }
  }
  "Multiples of three only" should {
    "print 'Fizz'" in { FizzBuzz.eval(12) must_== "Fizz" }
  }
  "Multiples of five only" should {
    "print 'Buzz'" in { FizzBuzz.eval(10) must_== "Buzz" }
  }
  "Non multiples of five or three" should {
    "print the number back" in { FizzBuzz.eval(11) must_== "11" }
  }
}

// package acceptanceTest { |> Scala IDE will not pick the class up as runnable if I sub-package it like this :-(
@RunWith(classOf[JUnitRunner])
class FizzBuzzUATSpec extends org.specs2.Specification { def is = 
  "This specification is to check the FizzBuzz evaluator"          ^
               p^
  "The FizzBuzz evaluator should"                                  ^
    "process a multiple of both three and five to return FizzBuzz" ! e1^
    "process a multiple of three only to return Fizz"              ! e2^
    "process a multiple of five only to return Buzz"               ! e3^
    "process a non multiple of three or five to return the input"  ! e4^
                                                                   end                                            
                               
  def e1 = FizzBuzz.eval(15) must_== "FizzBuzz"
  def e2 = FizzBuzz.eval(12) must_== "Fizz"
  def e3 = FizzBuzz.eval(10) must_== "Buzz"
  def e4 = FizzBuzz.eval(11) must_== "11"
}

@RunWith(classOf[JUnitRunner])
class FizzBuzzDataTableSpec extends org.specs2.Specification with org.specs2.matcher.DataTables { def is =  
  "Fizz Buzz testing with DataTables" ! e1   

  // note: when the first column of a DataTable is a String, '!!' needs to be used instead of '!'
  // see: http://etorreborre.github.com/specs2/guide/org.specs2.guide.Matchers.html#DataTables
  def e1 =
   "spec name"                       || "input val" | "expected output" |
   "Multiple of both three and five" !! 15          ! "FizzBuzz"        |
   "Multiple of three only"          !! 12          ! "Fizz"            | 
   "Multiple of five only"           !! 10          ! "Buzz"            |
   "Non multiple of five or three"   !! 11          ! "11"              |> { 
     (a, b, c) => FizzBuzz.eval(b) must_== c // (a, b, c) is a structural match on the table
   }
}

Feedback

Generally the pain points I encountered were with Scala IDE rather than Specs2. Specifically, I had originally wanted to include the different types of tests in the same source file, but differentiated by different subpackages. Unfortunately, Scala IDE didn’t pickup the sub packages so this was a non starter. One minor weirdness was the inability to reorder my arguments for the DataTable spec test.

Update: Thanks to Eric Torreborre this was solved using !! instead of ! when a String type is used for the first column in the DataTable !! It was a minor annoyance though.

Otherwise, Specs 2 looks mature, with good tooling support and would likely be me go to choice as a ‘wrapper’ framework for testing. I found the default API for specifications clear and easy to use, and liked the seperation of tests to the expressions evaluated to support or refute them. Nice.

Comparisons

RSpec, JBehave, easyb, Instinct, JUnit
<
SCALATEST

In a nutshell

  • A generalised framework for Java and Scala which uses a variety of traits to mixin a number of different test styles and strategies. Out of the box, TDD, BDD, functional, integration TestNG and JUnit tests are supported.

Link

http://www.scalatest.org/

Version v1.7.2

Ohloh

http://www.ohloh.net/p/scalatest

Active community

Yes. This is a second generation framework spawned from a project called SuiteRunner by Bill Venners in 2001. There are actively maintained forum groups accessible via: http://www.scalatest.org/community

Example

 package org.scalabound.scatdd.scalatest
 
 import org.junit.runner.RunWith
 
 import org.scalatest.FunSpec
 import org.scalatest.junit.JUnitRunner
 import org.scalabound.scatdd.FizzBuzz
 
 @RunWith(classOf[JUnitRunner])
 class FizzBuzzScalaTest extends FunSpec {
   
   describe('A FizzBuzz processor') {
     
     it('should return 'FizzBuzz' from a mulitple of three and five') { assert(FizzBuzz.eval(15) == 'FizzBuzz') }
     
     it('should return 'Fizz' from a multiple of three only') { assert(FizzBuzz.eval(12) == 'Fizz') }
     
     it('should return 'Buzz' from a multiple of five only') { assert(FizzBuzz.eval(10) == 'Buzz') }
     
     it('should return the stringified input from a non multiple of three or five') { assert(FizzBuzz.eval(11) == '11') }
     
   }
 
 }

Feedback
I only used one spec type and it seemed both clear and concise. The language dictated by this API ( describe.. it..) was straightforward it not slightly awkward at first site. I also prefer a clearer distinction between my tests and assertions, (which is a matter of taste). Still, it’d be hard to find a more straightforward and ‘friendly’ framework to get a Java team up and running with first time around.

Comparisons

JUnit, TestNG

SCALACHECK

In a nutshell

  • A specification based test generation framework for Scala and Java. The library was originally inspired by QuickCheck in Haskell.

Link

https://github.com/rickynils/scalacheck#readme

Version

v2.9.0-1-1.9

Ohloh

http://www.ohloh.net/p/scalacheck

Active community

Yes, but primarily stemming from Rickard Nilsson. There doesn’t appear to be any forum support for this project.

Update: Thanks to @Daniel Sobral there is (in fact) a google group in support of ScalaCheck here !               

Example

 package org.scalabound.scatdd.scalacheck
 
 import org.scalabound.scatdd.FizzBuzz
 import org.scalacheck.ConsoleReporter.testReport
 import org.scalacheck.Prop.forAll
 import org.scalacheck.Prop.propBoolean
 import org.scalacheck.ConsoleReporter
 import org.scalacheck.Test
 
 object FizzBuzzScalaCheck {
   
   val propFizzBuzzCheck = forAll { n: Int =>{
      if(n % 15 == 0) FizzBuzz.eval(n) == 'FizzBuzz' 
      else if(n % 3 ==0) FizzBuzz.eval(n) == 'Fizz'
      else if(n % 5 ==0) FizzBuzz.eval(n) == 'Buzz'
      else '' + n == FizzBuzz.eval(n) 
    }
   }
   
    def main(args : Array[String] ) = {
     ConsoleReporter.testStatsEx('blah', testReport(Test.check(propFizzBuzzCheck)))
   }
   
 }

Feedback Compared to the other frameworks, this one took me a while to get running with, (bear in mind, I was running with the others in minutes though !). The main sources of my pain were:

  • No obvious out of the box JUnit support
  • I originally ran my tests as a Scala App (i.e. something that extended App), though kept getting
    GenException(java.lang.NullPointerException)from the ConsoleReporter whenever I tried to run my tests.
  • Whenever I tried to add too many conditions to my implication operator, I found not enough tests were generated to validate my property, (hence the somewhat cludgy example I present).

Having said all that, I still find this framework unique and really valuable. On a few test runs I logged the inputs generated and Scalacheck really is great, (a fair span of inputs generated and passed through). This was probably the most challenging framework of the three, but also offered the sort of features I’d certainly want to use again and again. Likely next time I’d run this via Specs though !

Comparisons

Java QuickCheck, JUnit quickcheck

Conclusion

In conclusion all three frameworks are much clearer, (or expressive) than comparative frameworks that I have personally used in Java previously. If I was trying to get a team to ‘evovle’ to Scala, I’d choose ScalaTest. If I was working with a team who were slightly more comfortable with Scala I’d go the Specs2 route. In any event I’d try to use ScalaCheck for test generation, (ideally abstracted away through one of the other frameworks though). Either way I can see why using Scala test frameworks helps to sway opinion towards teams progressing towards Scala in the enterprise. I hope this feedback was useful and not too dry. As ever, the best ‘validation’ is in walking the path yourself (unless ScalaCheck will do it for you !). Good luck with your TDD and Happy hacking !

Reference: ScaTDD: Casting an eye over three major Test frameworks in Scala from our JCG partner Kingsley Davies at the Scalabound blog.

Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Back to top button