Apptio Eliminates Testing Bottleneck to Scale with Rapid Customer Growth

Spending a week setting up a browser version

Apptio’s enterprise apps help CIOs manage, plan and optimize their tech costs. Founded in 2007, Apptio today has more than 1,000 customers—including half of the Fortune 100 companies. It’s the top-ranking and most reviewed IT financial management solution on Gartner PeerInsights.

Apptio uses machine learning to translate technology costs across the entire IT portfolio (including on-premises systems, vendors, projects, and cloud systems) into a holistic, business-centric view. This helps IT leaders set future targets, measure business results and drive investment decisions.

In such a complex, layered system, a minor error could easily cascade into incorrect forecasts and millions in misallocated resources for the clients. To minimize the odds of such an outcome, Apptio ran UI, functional and end-to-end test suites on an in-house server farm, outfitted with Linux and Firefox. Over time, as usage trends shifted and Apptio grew to international markets, it was time to expand browser coverage.

“As part of a good test suite, we needed to have cross browser testing as well,” says Paule Grow, Tester-turned-Software Design Engineer for Tooling at Apptio.

The initiative began with adding the now-dominant Chrome to their test servers. But during the process, Paule saw the inefficiencies of maintaining an internal infrastructure. “TechOps helped us install Chrome on the server farm,” Paule says. “It worked—sort of—but we needed to keep the Chrome versions up to date.”

However, due to server storage restrictions, updating browser versions became a bottleneck in Apptio’s continuous testing workflow. “The request for more servers takes time to be approved. Then you need more time to flesh it out and even more time before the server is online. It added up to at least a week’s worth of effort,” says Paule. “We needed to keep pace with our customers, and for that, we needed an infrastructure which would let us scale our testing.”

Cross browser testing for every stage of Agile development

As a former tester, Paule understood flakiness. He wanted to ensure that the infrastructure he picked wouldn’t add to it.

“Stability was key. As we tried different solutions, that’s one thing we were watching out for. We also had occasional problems with our network, which meant testing would have to wait until it was back up. To make up for the lost time, we needed uptime and good availability,” he says. “We chose BrowserStack because it was so stable, it just sold itself.”

Paule began by scaling QA testing across browsers. “We have always used JUnit for our tests, so being able to specify browsers and versions programmatically in Java was a great help,” he says. “In very little time, we were able to scale automation to cover IE, Edge, Firefox, and Chrome—and even some of the beta browsers like Edge Insider preview.”

But Paule’s vision extended beyond scaling cross browser testing. ”My goal was to provide a stable framework for continuous integration—one that’d make people want to write tests,” he says. ”I wanted to make automation accessible to everyone.”

To scale automation testing on CI, Paule plugged BrowserStack into Apptio’s TeamCity implementation—through Local. “This is a huge plus for us,” he says. “We don’t have to set up URLs on a public server to receive test results.”

But Paule and his team went further still. “You want the bugs to be found as early as possible in the life cycle—and the earliest point is developers,” he says. By reconfiguring the IDEs, Paule enabled developers to run unit and smaller functional tests across browsers.

Paule says, “Instead of sending the request to localhost for testing, a developer can test directly on BrowserStack by flipping a config. So, if our developers expect some problems with, say IE, based on something they’re doing, this enables them to test on IE without switching context.”

“This way, bugs are cheaper to fix, you get more time to fix them and it’s safer all around. It’s really good for an Agile lifecycle like ours,” he says.

Reduced build times and faster deployment

With CI, dev and QA fully integrated with BrowserStack, Apptio was able to significantly reduce build times.

“Since our internal data center isn’t throttling the test execution anymore, we get builds done faster and see results faster. We can test each build and know exactly what’s in each of them,” Paule says. “It speeds up deployment and improves quality,” he adds.

Without the overhead of manual browser setup, Apptio overhauled their approach to testing. “There’s a logical separation in our testing now,” Paule says. “We copy all our test suites—UI, functional, end-to-end—and run them on 5-10 different browsers. It gives us visibility—if a test fails once, we know it’s a browser compatibility issue. If it fails on all of them, there’s an error in application logic,” he says.

The effect reverberates through the troubleshooting workflows. “Now when we have a bug bash, we can have people on 10 different browsers. That wasn’t possible before because we just didn’t have that much hardware. If we did, we were installing all of those browsers manually beforehand,” he says.

Besides automation, Apptio uses BrowserStack Live for manual, exploratory testing. “We’re not using any VMs or services at this point,” Paule says. “We go to BrowserStack Live and just page through the application on different browsers and resolutions. Basically, we’re exercising the UI, looking for specific areas that have changed and comparing them with older builds.”

Paule looks forward to testing in new ways with BrowserStack. “With more customers across the ocean, localization features are a big use case for us. I’ve already implemented it with BrowserStack and now we are rolling it out to our entire platform,” he says.

Within a year since integrating BrowserStack, Paule had succeeded in making automation accessible for everyone at Apptio. “The dev team is implementing their own tests based on the same framework as the QA team uses—and more teams are picking it up. Our cloud and login security teams started automating their tests. Our Benchmarking team prototypes new automation using Selenide with BrowserStack,” he says.

“With more teams automating their tests, our test suites are expanding rapidly. But with parallel sessions and queuing, we know we can continue to scale with our customers,” he says.