Our testing tools

We care about testing our software and we create tools and libraries that help us.

•

November 3, 2016

Introduction

TL;DR: Choose an approach to software testing that helps your organization create the best possible end results. That might mean using and/or creating a few tools and/or libraries along the way. Or, maybe not. What follows is a description of what we do at SmartyStreets, couched as a response to Dan Mullineux's equally valid way of doing things.

The cost

A favourite test helper library, with some simple test assertion functions clearly has some value... They [testing libraries] are not so bad, but they come at a cost, defer to avoid them.

It's difficult at first, given the title and the introductory paragraphs, to realize the nature of Dan's article. Is he against testing libraries altogether or just not necessarily for them? In the end I get the feeling that he's now very much against them because of several costs which he enumerates. In our experience many of these costs are minimal because they are amoritized across several consistently-implemented projects.

The effect of being able to quickly create and then steadily maintain these projects as they run in production, serving the needs of thousands of customers means that the upfront cost associated with creating and/or adopting helpful test libraries (for us) has turned out to be more like a worthwhile investment with perpetual high-interest returns.

The cost of extracting a dependency later

But then actually once you have assert.Equals dotted throughout the tests the cost of removing it becomes unaffordable.

Here Dan raises the concern that once you've chosen a library to help with test assertions/comparisons, it would be a huge chore to eventually rip it out. We feel that such calls to assertions helper functions aid readability by elegently expressing the essence of the test. We've settled on this approach for several years so (luckily) this isn't a cost we've needed to deal with because we really like the approach. The broader lesson to learn here is: adopt dependencies judiciously...

The cost of typing

Dan, like many other gophers before him, brings up the issue of efficiency in typing when selecting an approach. The difference here is that in this case, using a library actually lessens the number of keystrokes which is usually deemed a worthy goal in the go community (see Brvty, an alternate take on go naming conventions):

Clearly typing effort is not the final arbiter (by any means) in assessing the best approach, but it is an important factor.

There are way more important criteria to consider than whether using a particular library will increase or decrease keystrokes:

Will it help me write high-quality code?
Will it help me write code that is readable?
Will the tests I write with it serve as a readable, reliable source of documentation?
Does this library have lots of open issues? What is the nature of those issues? Are there lots of scary bug reports?

The cost of learning an API

Some of the examples cited in the article use the GoConvey and assertions projects (both written by SmartyStreets). As the primary contributor to both of those projects I am uniquely qualified to comment on this question: Dan asks a question shared by many who are just beginning to use GoConvey or the assertions package:

What will happen when you write:

So(x, ShouldAlmostEqual, 2)

or

So(y, ShouldNotResemble, 2)

At its foundation this question is expressing a frustration:

There's too many of these functions! How will I know which one to use?

Try them and see what they do! Then read the godocs or the unit tests for the assertion functions themselves and you'll know exactly how they were inteded to be used.

You're right, we've provided an awful lot of assertions functions. We hope you'll try them out. You might just find a few of them really that are useful on a regular basis for your situation. I'm surprised by how many times ShouldBeChronological (a rather specific assertion function which I created on a whim) has come in handy. You can also define your own custom assertions! I just did this the other day to great effect.

BTW, having an editor with autocomplete and godoc integration makes it easier to discover all of the assertion functions and learn what they do and when to use them.

The cost of bugs

The one thing these libs have in common is bugs. It is annoying enough to have to debug test code, let alone 3rd party test support libraries.

Now the hammer drops in the form of a broad, yet serious, accusation. Game on!

Admittedly, we don't use GoConvey anymore. That's not because it was buggy or didn't do the job. There are still a lot of gophers that use it. More power to them!

We have found that we are even happier with a more classic xUnit-style tool, which is why we created gunit. Here are some interesting statistics and facts about the testing tools we currently use (including LOC, since it was a big deal in the article I've been quoting):

gunit:

0 open issues at the time of this writing
3 closed issues (minor issues related to display and optimzation)
Production Code LOC: 464 (For comparison, GoConvey clocks in at 3940)
Test Code LOC: 474
Plugs directly into the standard library testing package
Uses reflection to gather test methods on a struct you feed it
Uses Go 1.7 sub-tests under the hood (so you can still use the -run flag to select specific tests)
Displays correct line numbers in failure output
Allows usage of familiar t.Error*() functions
Allows marking tests as skipped or long-running (in conjunction with the -short flag)
We've trusted this library to help us compose tests for our customer-facing projects at SmartyStreets as well as our open-source initiatives
Basically, anything written in Go has been tested using gunit.

assertions:

The article points out that the assertions project (with all of its subordinate components) clocks in at 26,000 LOC. Much of this number is derived from the vendored test dependencies (see the 'internal' folder from a few commits ago). Because of our approach to package management this has never been an issue for us at SmartyStreets. But Dan makes a fair point that this is, certainly, a barrier to entry. So I've trimmed the fat, cleaning out the test dependencies and only leaving the skeleton of what was actually being used to make the assertions package work. I've included updated LOC statistics below.

0 open issues
6 closed issues (one was a legitimate bug in comparison logic)
Production Code LOC: 2889
Test Code LOC: 904
Provides assertion functions for GoConvey
Can provide assertion helper functions for gunit (used by all SmartyStreets projects listed above)
Can provide assertion helper functions in standard testing functions
Can provide assertion/comparison functions in any other go program context

oglematchers:

A part of this project has been vendered with the assertions package above.
Built by Aaron Jacobs of Google
3 open issues (none are bug reports)
22 closed issues (almost all related to adding new features, not bugs)
Purpose is much like that of the assertions library
Part of a larger suite of testing tools, all sharing the curious 'ogle' prefix.
Does a lot of tedious heavy lifting with reflection, allowing comparisons of any numeric type (e.g. float64 < int and uint64 >= float32)

go-render:

vendored, internal dependency of assertions
Built by Googlers
0 issues (open or closed)
Even more detailed output than fmt.Sprintf("%#v", x) which also includes nested structures.

If I do say so myself, this is a lightweight collection of useful, well tested, stable code. Our complete testing tools weigh in at 4,731 lines, and that number includes the test code for the test tools themselves. This is code we have built and currated over several years and we're very comfortable, a word which here means effective and efficient, with it.

At the bottom of the article you'll find an example of a test fixture implemented with gunit and the assertions package. The test cases are declarative and don't need any if statements or hand-crafted failure messages. The assertions package handles all of that. Some argue against this approach, opting to write their own comparisons and failure messages--to each his own. We find the tests below easier to read and reason about than the alternative. See one of my previous articles for a comparison of what these tests might look like without gunit.

After having read the following short snippet you will, most likely, be able to clearly imagine the structure and details of production code that would satisfy these test cases, which is a testament to the clarity offered by this approach.

Conclusion

It's wonderful that there are people that are passionate enough about writing good software that they write tests as a way to prove and document functionality. At the end of the day, every developer and every team has to decide on an effective approach for their circumstance. There are scenarios in which I find myself using the standard library testing package on its own to great effect. Sometimes I add the assertions package to a standard test function and it makes life nicer. Most of the time I'm writing a gunit fixture and leveraging the assertions library and I couldn't be happier with the result. Good luck honing your own approach to software testing.

Fixture example:

package examples

import (
	"testing"

	"github.com/smartystreets/assertions/should"
	"github.com/smartystreets/gunit"
)

func TestBowlingGameScoringFixture(t *testing.T) {
	gunit.Run(new(BowlingGameScoringFixture), t)
}

type BowlingGameScoringFixture struct {
	*gunit.Fixture
	game *Game
}

func (this *BowlingGameScoringFixture) Setup() {
	this.game = NewGame()
}

func (this *BowlingGameScoringFixture) TestAfterAllGutterBallsTheScoreShouldBeZero() {
	this.rollMany(20, 0)
	this.So(this.game.Score(), should.Equal, 0)
}

func (this *BowlingGameScoringFixture) TestAfterAllOnesTheScoreShouldBeTwenty() {
	this.rollMany(20, 1)
	this.So(this.game.Score(), should.Equal, 20)
}

func (this *BowlingGameScoringFixture) TestSpareReceivesSingleRollBonus() {
	this.rollSpare()
	this.game.Roll(4)
	this.game.Roll(3)
	this.rollMany(16, 0)
	this.So(this.game.Score(), should.Equal, 21)
}

func (this *BowlingGameScoringFixture) TestStrikeReceivesDoubleRollBonus() {
	this.rollStrike()
	this.game.Roll(4)
	this.game.Roll(3)
	this.rollMany(16, 0)
	this.So(this.game.Score(), should.Equal, 24)
}

func (this *BowlingGameScoringFixture) TestPerfectGame() {
	this.rollMany(12, 10)
	this.So(this.game.Score(), should.Equal, 300)
}

func (this *BowlingGameScoringFixture) rollMany(times, pins int) {
	for x := 0; x < times; x++ {
		this.game.Roll(pins)
	}
}
func (this *BowlingGameScoringFixture) rollSpare() {
	this.game.Roll(5)
	this.game.Roll(5)
}
func (this *BowlingGameScoringFixture) rollStrike() {
	this.game.Roll(10)
}

Console output:

$ go test -v -run TestBowling
=== RUN   TestBowlingGameScoringFixture
=== RUN   TestBowlingGameScoringFixture/TestAfterAllGutterBallsTheScoreShouldBeZero
=== RUN   TestBowlingGameScoringFixture/TestAfterAllOnesTheScoreShouldBeTwenty
=== RUN   TestBowlingGameScoringFixture/TestPerfectGame
=== RUN   TestBowlingGameScoringFixture/TestSpareReceivesSingleRollBonus
=== RUN   TestBowlingGameScoringFixture/TestStrikeReceivesDoubleRollBonus
--- PASS: TestBowlingGameScoringFixture (0.00s)
    --- PASS: TestBowlingGameScoringFixture/TestAfterAllGutterBallsTheScoreShouldBeZero (0.00s)
    --- PASS: TestBowlingGameScoringFixture/TestAfterAllOnesTheScoreShouldBeTwenty (0.00s)
    --- PASS: TestBowlingGameScoringFixture/TestPerfectGame (0.00s)
    --- PASS: TestBowlingGameScoringFixture/TestSpareReceivesSingleRollBonus (0.00s)
    --- PASS: TestBowlingGameScoringFixture/TestStrikeReceivesDoubleRollBonus (0.00s)
PASS
ok  	github.com/smartystreets/gunit/examples	0.005s

Subscribe to our blog!

Learn more about RSS feeds here.

Subscribe Now