About This Episode:

Cloud device farms make it easy to get started with real device testing. But what happens when you need to scale? Costs climb fast, you wait in a queue for the devices you need, and you lose visibility into what’s actually happening when a test fails.

In this episode, Nick Metcalfe, Head of Technical Sales Engineering at Cambrionix, joins Joe to explain why more testing teams are bringing their device labs in house.

Nick shares how one team was quoted roughly a million dollars a month to outsource their real device testing, then cut that cost to around $300K by building their own lab.

And cost is only part of the story.

You’ll discover why an in house device lab can outperform a cloud lab:

Nick also covers the honest tradeoffs, including the setup and resources an in house lab requires, and why a hybrid approach of emulators first and real devices second gives you the most confidence before release. Plus, his number one tip for building a lab: do it once, do it right.

If your team is scaling mobile test automation, fighting flaky tests, or questioning your cloud device farm bill, this episode is for you.

[fusebox_track_player url=”https://traffic.libsyn.com/testtalks/ai_agents_passed_the_tests._real_devices_didnt._draft_v1_1080p.mp3″ social_linkedin=”true” social_email=”true” ]

About This Episode’s Sponsor: Cambrionix

Have you ever chased a failing mobile test for an hour, only to find the real problem was a device that dropped off its USB hub? In large device labs, flaky infrastructure often looks like a code defect. That wastes your team’s time and erodes trust in your automation.

Cambrionix builds intelligent USB hubs and device management software for software QA teams that test on real devices at scale. Whether you’re validating an OS release, regression testing an app update, or flashing firmware across hundreds of devices, Cambrionix helps keep every device charged, connected, and ready for testing.

Here’s what Cambrionix brings to your device lab:

Real world proof: Snap moved from an outsourced device lab costing about $1M a month to an in house lab supporting 12,000+ physical devices and 700,000 daily test executions, saving over $2M a year. Teams at Apple, Google, Uber, SpaceX, and Snap all rely on Cambrionix.

👉 See how Cambrionix can help your QA team test faster on real devices:  https://testgld.link/realdevice

Episode Summary

When a test fails, stop blaming your code first

Nick opens with a habit almost every engineer shares. When a test goes red, we assume our latest change broke something and dig deep into our own scripts. But in device labs, the failure often comes from the setup itself: how devices are connected, the cables, the power, or the host system. Asking “how are these devices connected, and where did this failure really come from?” can save hours of misdirected debugging.

AI is tripling your code. Can your lab keep up?

Teams Nick works with report AI is increasing their output by roughly three times, meaning three times the pull requests and updates that need validation. Nick also warns about AI “checking its own homework.” A test that passes only proves the outcome matched what the AI expected, not that it’s correct. More code means you need more testing capacity, and scaling emulators alone means scaling your compute costs right along with it.

Emulators vs real devices: use both

Nick recommends a hybrid model. Emulators and simulators are a fast, cheap first pass for pull requests. But once code passes there, run it on real devices, because your users live in the real world. Things like battery swelling in a four or five year old phone, device temperature, and how long since a device was rebooted can all cause failures that users will blame on your app, and emulators won’t catch them.

Don’t cheap out on cables

Nick sees teams buy thousands of dollars of Mac Studios and flagship phones, then plug them in with whatever cable is lying on the floor. His recommendation: cables rated for up to 100W and USB 3.2, which typically cost between $5 and $10 each. Cambrionix hubs can even flag a faulty cable with a purple LED and report the exact error through the API.

Cloud device farm vs in house lab

Nick is fair about cloud providers like Sauce Labs and BrowserStack. They’re a solid option for small scale testing. But at scale the math changes. One customer was looking at about a million dollars a month to outsource their real device testing. Bringing it in house brought that down to around $300K, even after factoring in hardware engineers, space, and setup time. Beyond cost, in house means no queue, full control, and full visibility.

Telling infrastructure failures from code failures

Because Cambrionix sits at the hardware layer, it can capture low level USB communication that you’d never see plugging devices straight into a computer. If the OS reboots the USB tree or shuffles ports, the system records exactly when it happened so you can line it up against your test failure. Nick shares a story of a device maker whose new product stopped behaving correctly. Cambrionix tools revealed the device was negotiating power in a nonstandard way outside the USB spec, something the team never would have suspected.

Automating the painful stuff through an API

Most test teams don’t want a pretty UI, they want an API. Cambrionix offers a lightweight local API with Swagger docs that works with Python, JSON, and your CI pipeline. Standout use cases include putting up to 16 Apple devices into DFU mode at once (no more holding button combinations on each device), scripting devices out of restore mode after a crash, and ADB integration for Android with custom power profiles for devices like the Pixel 6 that behave differently.

Scaling without running out of power

Nick shares a California customer that simply couldn’t get more power into their building. They were running four or five phones per Mac, filling rooms with host systems. Switching to ThunderSync hubs took them to 16 devices per Mac instantly, freeing up power and space to grow their lab. In his own test, Nick wiped and updated 60 iPhones in about 13.4 minutes using hubs connected to a single Mac mini.

Why Snap tests on legacy devices

Snapchat is heavily used by kids on hand me down phones, so the app has to work on older devices that most cloud labs don’t keep. That’s why Snap built an in house lab with over 12,000 devices running 700,000 tests a day using Cambrionix hubs.

Nick’s number one tip

Set it up once and set it up right. Nick regularly sees teams buy the cheapest option available, then email him two years later with problems, having spent more money than planned because they had to build the setup twice.

Key Takeaways

Frequently Asked Questions

Is a cloud device farm or an in house device lab better for mobile testing?
Cloud device farms are a good fit for small scale testing and quick access to devices. At larger scale, an in house lab often costs significantly less and gives you full control, no queue times, hardware level visibility, and the ability to test legacy or custom devices cloud vendors don’t carry.

Are emulators good enough for mobile testing?
Emulators and simulators are a fast, low cost first pass. But they can’t fully reproduce real world conditions like battery degradation, heat, and power negotiation behavior. A hybrid approach that runs on emulators first and real devices before release gives the most confidence.

How do you tell if a failing test is caused by hardware or code?
You need visibility into the hardware layer. Tools that capture USB level events, like port resets or controller reboots, let you match infrastructure events to the exact moment a test failed.

What do you need to set up an in house device lab?
According to Nick, you can start with a dedicated host such as a Mac mini or small Windows or Linux PC, a multi port hub like the ThunderSync5-C16, quality cables rated for up to 100W and USB 3.2, and the Cambrionix API installed on the host.

Can you automate DFU mode on multiple Apple devices?
Yes. With Cambrionix hubs and a Connect Premium licence, you can put up to 16 supported Apple devices into DFU mode at once through the API instead of pressing button combinations on each one.

About Nicholas Metcalfe

Nick podcast guest headshot

Nicholas “Nick” Metcalfe, Head of Technical Sales Engineering, Cambrionix

Nick leads Cambrionix’s global Technical Sales Engineering team, bringing ten years of experience in technical sales engineering and hands-on customer support.

Working closely with SQA and engineering teams, Nick helps customers design, build and optimise real-device testing environments. His work spans lab architecture, hardware integration, automation scripts and APIs, as well as integrating device infrastructure into existing CI/CD pipelines. He also provides hands-on troubleshooting at the physical layer, helping teams identify when hardware, connectivity or device behaviour is affecting the reliability of their test results.

Supporting customers across a wide range of lab sizes and testing requirements gives Nick a broad understanding of how real-device testing environments behave at scale, the common challenges SQA teams encounter, and the practices that help create reliable, repeatable and maintainable test infrastructure.

Nick combines this technical experience with a strong ability to simplify complex information, delivering technical guidance, product demonstrations, training and on-site support to help customers successfully integrate Cambrionix solutions into their existing test environments.

Connect with Nicholas Metcalfe

Resources Mentioned

Rate and Review TestGuild

Thanks again for listening to the show. If it has helped you in any way, shape, or form, please share it using the social media buttons you see on the page. Additionally, reviews for the podcast on iTunes are extremely helpful and greatly appreciated! They do matter in the rankings of the show and I read each and every one of them.