Smart homes run on hundreds of add-ons.
Who tests them?
Devices. Hubs. Clouds. Apps.
And every protocol between them. IoT platforms are hard to test by design.
ApplicationsnotificationsaccessUI
Cloudstorageweb servicesdatabase
EdgeMQTTCoAPHTTPZigBee hub
IoT devicessensorsbulbscamerasfirmware
The first look at testing in open-source IoT platforms
OpenHAB & Home Assistant
≈37K test methods mined. 80 developers surveyed.
How tested is each add-on?
Pair every method with its tests.
add-on source · illustrative
public void handleCommand(Command cmd)
…
@Test
void handleCommandTurnsOn()
FOCAL METHOD · fm
FOCAL TEST · ftm
Most add-ons fall short.
OpenHAB
0 of 406 add-ons above 0.5
Home Assistant
0 of 937 add-ons above 0.5
Average test ratio 0.04: ≈4K test methods for ≈76K functional methods.
Average 0.42, yet 65% of add-ons sit below the threshold.
80 IoT developers, in their words
Automated. Unit-level. Rarely manual.
PRACTICE
Most prefer automated testing, to catch defects early.
GAP
No integration testing found in either platform.
TOP CHALLENGE
Compatibility across countless devices and platforms.
SECURITY
Harder to test with diverse devices and few IoT tools.
Testing Practices, Challenges, and Developer Perspectives in Open-Source IoT Platforms
Daniel Rodriguez-Cardenas · Safwat Ali Khan · Prianka Mandal · Adwait Nadkarni · Kevin Moran · Denys Poshyvanyk
ICST 2025 · IEEE International Conference on Software Testing, Verification and Validation