Why did PhantomJS get two seconds slower for each Jasmine test?

Sometimes it’s the small things.

At Brigade we use Jasmine along with the Jasmine gem to unit test our React.js application. This gem uses PhantomJS under the hood to run the specs in CI (in our case, Jenkins CI). We recently started using webpack’s css-loader (and cousins sass-loader and style-loader) which caused our specs to not run at all under PhantomJS 1. After adding a little bit of configuration to our spec/javascripts/support/jasmine_helper.rb file, we were able to get our specs running again with PhantomJS 2.

Jasmine.configure do |config|
  # Use whatever version of PhantomJS is already installed. 
  # Our Jasmine tests require PhantomJS 2.0+, which the
  # phantomjs gem does not currently install.
  config.prevent_phantom_js_auto_install = true
end

Although this allowed our specs to run in PhantomJS again, the Jenkins job’s running time jumped from ~4 minutes to ~20 minutes. Oddly, the specs seemed to run at a normal clip in a regular browser, such as Chrome or Firefox. Fast CI feedback is important to our developer experience, and we run our tests in CI hundreds of times every day — so we felt the pain pretty quickly.

Knowledge is power

Jasmine doesn’t give us any running time information so it wasn’t clear exactly what was slow. To give us some visibility I created a slow spec reporter.

spec/javascripts/support/slow_reporter.js

This, coupled with a bit more configuration in our jasmine_helper.rb file allowed us to see which specs were slow when running rake jasmine:ci.

config.show_console_log = true

This information showed me that many simple specs took longer than 2 seconds to run. With a little more digging, I discovered a couple of very similar tests with extraordinarily different performance characteristics.

// This test took ~15ms
it('has a <p> with the first line of content', () => {
  expect(this.subject()).toHaveSelector('p', { text: '...' });
});
// This test took ~2000ms
it('has two <p>s', () => {
  expect(this.subject()).toHaveSelector('p', { count: 2 });
});

Note: we are using the subject pattern here.

A simple solution surfaces

This was repeatable in isolation, which told me that the problem was likely somewhere in our toHaveSelector custom matcher. Since the only difference between these examples was the type of options passed to the matcher, I looked for code differences in the matcher when using these different options and found this block:

if (options.count === undefined) {
 // Check for element presence
 result.pass = Utils.findElement(node, selector, options);
} else {
 // Check the count of elements
 result.pass =
    Utils.countElement(node, selector, options) === options.count;
}

I knew that the fast example used the count and the slow example didn’t. The Jasmine documentation for custom matchers tells us:

The compare function must return a result object with a pass property that is a boolean result of the matcher. The pass property tells the expectation whether the matcher was successful (true) or unsuccessful (false).

Since our Utils.findElement helper function returns a DOM node, for the without-count branch we were relying on the truthiness of this object somewhere in Jasmine internals. This caused the browser to hold on to and pass around potentially large DOM tree references instead of a lightweight boolean. We are not entirely sure why this causes so much slowness in PhantomJS 2, but the solution here was simple: adding a !! to the result.pass assignment to cast it to a boolean allowed the browser to free up the DOM nodes more quickly and easily.

result.pass = !!Utils.findElement(node, selector, options);

This tiny change brought our slow spec down from the 2-second zone back into the 10-millisecond zone in PhantomJS. A quick run of all our specs showed that this resolved the issue globally, bringing our CI run back down from ~20 minutes to ~4 minutes. The effect was immediately noticed and celebrated by the team, making this simple solution all the more satisfying.

In this treasure hunt we learned how to get more visibility into our spec runs by adding custom reporters, which led us to discover that small changes can have a big impact — especially in “weird” environments such as PhantomJS. This knowledge allowed us to shave off ~16 minutes from our CI job that we run hundreds of times each day, which made our team more productive.