Build vs Buy: How to decide between in-house lab or cloud solution

One of the most common questions asked in software development is “build or buy”? To expand, it means “Do we build a solution in-house or do we buy it from a third party?” Given how important certain tools can be for building a solid, high-functioning software product, this question is not easily answered. 

This article will try to make the question easier to answer by breaking down the various factors that should be considered when deciding “build vs buy”. 

Build vs Buy: What to consider before making the decision

The build vs buy decision matrix entails a longer, not-as-catchy question that has deeper implications for the quality of any software development pipeline. Should you surrender control of certain stages in your project for a third-party solution? Or should you spend the man-hours, human effort, money, and learning resources required to create a solution tailored to the project/team/company’s needs?

To make the decision, start with asking the following questions:

What is my budget?

The most important question to answer is how much money you can spend. To build a solution (in this case, a testing solution) from the ground up will require extensive planning, money, and time. Additionally, every tool must be maintained and updated regularly for it to stay relevant and aligned with the breakneck pace of technological development. So, how much will the solution cost you to build, maintain and update well enough that it will continue to be an asset in five years?

In the case of a testing tool, the most integral element is – the real device. Software tests can only elicit dependable results if they are run on real browsers, devices, and operating systems. Simulators and emulators, especially the free ones, cannot replicate the real user conditions that websites and apps must operate in when made publicly available. 

To build a device lab worth the investment, you’d have to buy multiple devices from multiple manufacturers installed with different browsers, browser versions of each browser, as well as operating systems. This is no easy task, considering the fact that new devices are being released every year, as are new browser and OS versions. So, you’d have to keep buying new devices, and installing them with new browsers & OSes, while also maintaining and updating the existing devices. 

Needless to say, this process can be tedious and never-ending. Not to mention, the expenses of system maintenance, hardware/software updates, fixing system bugs, and other tasks that will require dedicated personnel. Don’t forget the electricity bills required to keep the servers, mobile devices, and cooling equipment running. When added up, the costs involved can be prohibitive, especially for medium to small companies. 

Here’s a comparison of the sample costs for Build vs Buy to understand it in terms of Annual Costs incurred:

With a cloud-based testing platform like BrowserStack, the only thing you are spending on is the plan of your choosing. You don’t have to bother about purchasing, maintaining, managing, or updating anything. All you do is choose from a real device cloud of 3000+ browsers and devices (including thousands of desk and mobile devices), run manual and automated tests as required, and get 100% accurate results every time.

Test on Real Browsers & Devices Free

Bonus Tip

With regard to budget, here are a few questions that, in our experience, developers and project managers often ask:

How do I justify the cost of manpower for Build vs. Buy? Should manpower cost not be included on the Buy side?

When building a testing solution, the manpower cost for VMs and Real Devices includes the annual salary expense for Site Reliability Engineer, QA Engineer, DC Engineer, and Software Engineer. When you buy a cloud-based subscription, you don’t need an SRE and DC Engineer since nobody has to manually maintain the grid. 

As for having a QA and Software Engineer, in most cases, the already available resources in your organization for testing and development can handle the work when you move to a new tool. Hence there is no need to hire an extra resource for this purpose. Even if one more person is needed for testing, it will still be only 1/3rd to 1/4th of the total manpower cost while building your own grid.

Typically, the ratio of manpower needed for infrastructure maintenance and testing is 1:1. For example, if two people (hardware engineers) are needed for initial setup and maintenance, two more are deployed for testing and debugging (writing the code). If you build your own lab, you will have to spend on resources needed for infrastructure maintenance and continuous monitoring. If you buy a cloud platform like BrowserStack, you’ll need, at most, two people for testing (usually it will be the engineers already in your team). 

My current resources are enough to handle all testing and development tasks. Therefore, why have you added such a huge manpower cost in Build vs. Buy Framework?

When you are building your own grid, there are a lot of new operational tasks which require dedicated manpower. Not spending on resources could lead to an added opportunity cost, with current resources dealing with the overhead of additional tasks outside of their core function.

What does the team/business need?

Obviously, when deciding whether to build vs buy software, you and your team will document, in detail, everything required from a technical and business perspective. Now, if a product already available in the market actually meets and fulfills these requirements, why would you build instead of buy?

An in-house device lab would require, at a minimum, the following:

1. A dev team dedicated to implementing CI/CD, including building, maintaining, and supporting the tool.
2. Legal personnel to ensure necessary compliance.
3. The money and time to manage the software that allows the device lab to be accessed by teams.
4. A 24×7 support team meant to fix bugs, glitches, and errors. 

Of course, a major reason for building a solution in-house would be corporate security policies. Many companies have security standards that prevent development versions of their in-progress software from being deployed out of the company firewall. In this case, you would think that building an in-lab is the only option. 

However, testers can use a feature such as BrowserStack’s Local Testing, which allows remote browsers and devices from the BrowserStack cloud to access websites hosted on your internal networks that are otherwise not accessible over the public Internet. In other words, test private websites and apps behind firewalls on 3000+ real browsers and devices without having to purchase or maintain any devices or frameworks. 

Additionally, BrowserStack is SOC2 Type 2 compliant and is audited regularly to check if customer data is managed securely with complete privacy protection. 

Can an in-house lab keep up with the industry?

Among all the domains in the professional world, none evolves, adapts, and advances as fast as technology. Software changes with increasing frequency and tools must be developed to add new features and abilities to test these newer software models. 

If you build an in-house lab, can you keep up with industry developments? For example, not only will you have to keep buying new devices, but also ensure that the testing infra supports new testing frameworks and testing types/techniques. Additionally, you have to keep training the team on skills needed to support and expand the infrastructure.

Remember that all changes to a robust testing framework will inevitably put reliability at risk. Your team will run tests during, say, their lunch break or overnight. Can your expanding test architecture incorporate new features without compromising on the quality/consistency/reliability of existing and new tests?

Needless to say, none of these pressures exist if you find the right real device cloud for testing. BrowserStack customers don’t worry about upgrading existing frameworks or adding new ones; they just sit back and let the cloud take care of that. All they need to do is access the devices they want and run the requisite tests. 

How fast do you need a tool?

Building a testing solution from scratch takes a significant amount of time, no matter how much money you may have at hand. You can rush high quality, which means that any tests depending on this solution will be delayed or run in inadequate circumstances. Implementing, integrating, and testing a tool, as well as training staff takes time. Can you or your project afford to wait that long?

Of course, if there is literally no tool in the market that matches your requirements (or at least, the most important ones), you have no choice but to build it. However, if a third-party tool is able to align with most of what you are looking for, consider giving it a trial run, even while the team is building the tool. It is wisest not to pause day-to-day operations completely.

Building In-House Lab: Pros & Cons

Pros

• More control: Obviously, if you own the tool, you have full control over numerous technical aspects – system updates, options for users, and features that match your internal workflow exactly. Sensitive data is also far more secure, as third-party services simply cannot access your code, config files, and the like.
• More customization: An in-house lab can be customized to suit your particular, unique needs and obstacles. You, your team, and the company can prioritize, develop and implement specific features to ensure an uninterrupted and seamlessly functioning pipeline. You can also build the tool to work with processes, hardware, and software that runs your IT stack. With an in-house lab, you are only limited by your ideas and the availability of resources and talent.

Cons

• Prohibitively Expensive: As explained so far, building a customizable, scalable, reliable and full-featured testing lab entails enormous costs – including research and development costs, legal and compliance fees, maintenance, upgrade and support costs, electricity bills, costs entailed in the buildout period, data storage assets, and much. Building any tool will bring up unexpected issues, so expect costs to be unpredictable and increase beyond initial estimates. Costs will also be ongoing to account for updates, maintenance, and error/glitch/bug resolution.
• Technical Issues: Building an in-house solution requires devs and testers with considerable expertise in relevant fields. If you hire someone externally, they may move on since their skills make them highly employable. If you shift personnel internally, you might lose the manpower needed to accomplish other tasks. But without the right people involved, the tool will simply not be built, or at least not built to deal with the many errors and anomalies that will inevitably show up in day-to-day functions.
• Late time-to-market: Depending on the specifications, building an in-house lab can take months, even years – especially if you don’t have a team big enough to devote to this undertaking. The size and complexity of the tool, as well as the technical abilities of the team, must also be taken into account. Any tests dependent on this tool will be delayed, and negatively impact time-to-market.
  Additionally, the time it takes for an in-house lab to be developed into a perfected solution matching third-party options is time that software teams cannot afford to spend without releasing products. Users want their website and apps released fast and updated faster, which means that you can’t wait a few months (or even weeks) to test the MVP (minimum viable product).

Buying A Cloud-Based Solution: Pros & Cons

Pros

• Lower costs: As explained earlier, the costs involved in building an in-house solution usually tend to be massive. When you buy a third-party solution, you don’t have to spend on maintenance, operational, or R&D costs.
  
• Faster time-to-market: With nothing to build, upgrade or maintain, your effort in testing software is significantly reduced. Naturally, time to market is increased, which caters positively to customer expectations.
  
• Access Mobile Devices with No Maintenance: Maintaining thousands of mobile devices in “Build” mode is not a simple task. Keep in mind that, to stay effective, your in-house lab will have to keep buying and making new mobile devices available for testing 24×7.
  It is much simpler to use a platform like BrowserStack, whose real device cloud is automatically updated with new mobile devices, browsers, and operating systems frequently. You simply have to sign up for free, pick a mobile device and start testing.
  
• Access to better support: A third-party solution worth the money will give users access to dedicated industry experts who are accustomed to solving complicated problems and figuring out how to cater to customers’ expectations. It is their job to find out how you can get the results you want out of the solution you have purchased.
  That means your team will no longer have to spend time figuring out why their testing platform is glitching or flaking. They will have support personnel fixing issues for them while they can focus on designing smarter, more effective tests, code and environments.  
  
• More Features: If you are spending money on a third-party solution, you can expect it to have the majority of features you need. Third-party vendors will regularly implement new features to stay competitive and accommodate newer developments in the domain. With that handled, your team no longer has to think “But can the platform support this?” when trying out a new framework or testing technique.  

Cons

• Limited Customization: A third-party platform has not been built with your team’s needs in mind, which means you might hit some dead-ends in terms of what you can do. A good solution will have support teams that try to accommodate your requirements, but that too may have its limits.
  Additionally, you can’t make any decisions about including new features, so your competitive advantage may be lower.
• Limited Device Coverage: Unless you are buying a solution known for its wide device/browser coverage, you will be limited by the real devices, browsers, and OS(s) they are able to provide.
• Long-term Expenses: If your team needs to significantly expand their test pipeline within a short period, costs in the form of annual subscription fees will shoot up. Additionally, you might end up with a plan that includes features you don’t need to use at the time, which results in money spent without acquiring value. 

Read More: How do you ensure maximum test coverage?

Conclusion

The build vs buy framework has no clear winner. Obviously, one may suit you more than the others, depending on what you can afford to spend and what your short-term goals are. Generally, however, buying a cloud-based testing platform like BrowserStack does tend to provide more advantages, while requiring minimal effort. 

Try BrowserStack’s Cloud Solution for Free

For small and mid-sized teams or companies without a financial war chest to back them up, building an in-house lab can be draining, and delay essential development activities required to give customers what they want. By using a cloud platform, a few clicks will get them the device access and excellent tech stack they need to conduct comprehensive software testing.