No, you didn't want that stack trace

Tail of a test run.




What am I doing today? End to end performance testing of the new S3A implementation. Or not.

2016-06-16 18:17:32,339 INFO  s3.S3aIOSuite (Logging.scala:logInfo(58)) -
 Loading configuration from ../cloud.xml
*** RUN ABORTED ***
  java.lang.RuntimeException: Unable to load a Suite class that was discovered in the
   runpath: org.apache.spark.cloud.s3.S3aIOSuite
  at org.scalatest.tools.DiscoverySuite$.getSuiteInstance(DiscoverySuite.scala:84)
  at org.scalatest.tools.DiscoverySuite$$anonfun$1.apply(DiscoverySuite.scala:38)
  at org.scalatest.tools.DiscoverySuite$$anonfun$1.apply(DiscoverySuite.scala:37)
  at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
  at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
  at scala.collection.Iterator$class.foreach(Iterator.scala:727)
  at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
  at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
  at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
  at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)

So, a test case wouldn't load. No details. Well, let's paste in the stack trace and see if the IDE will tell me where things went wrong.

catch {
  case e: Exception => {
    // TODO: Maybe include the e.getClass.getName
    // and the message for e in the message
    // cannotLoadDiscoveredSuite,
    // because Jess had the problem
    // That gradle cut off the stack trace so she
    // couldn't see the cause.
    val msg = Resources("cannotLoadDiscoveredSuite", suiteClassName)
    throw new RuntimeException(msg, e)   ***HERE ***
  }
}

Ah. The inner cause has been lost, whoever wrote the test runner knew this as someone called "Jess" complained. And the reaction was not to address it, but just add a TODO todo note.

I generally automatically -1 any patch whose test cases lose exception data. This test runner isn't quite doing that, but it is cutting the message from the top of the list, relying on the test reporter to display the whole list. Which, clearly, the scalatest reporter is not doing.

Having written a distributed JUnit runner in the distant path, I understand a key issue with test runners: they're hard to test. You essentially have to bootstrap your own testing infrastructure, then generate the failure conditions you see from tests. Which does mean: you get to stare at the stack traces. I think what we are seeing here is evidence that whoever is developing scalatest doesn't use maven.

This also shows what's wrong with TODO notes: they're technical debt you forget about.

At least if you keep adding more and more things to your issue tracker, you've got a nice shared graph of filed vs completed. Whose project tracks the number of TODO line items in a source tree?

(Update: found the problem. The test in question had a base class with an after{} cleanup clause, as did a mixed in trait. Whatever error message scalatest raises at that point, well, it doesn't make it through the testrunner unscathed)