JultikaUniversity of Oulu repository

Executing test automation for large-scale software-intensive embedded systems requires a lot of hardware because tests must be executed for different hardware configurations. When there is not enough hardware for all kind of configurations, other solutions are developed to fill high test coverage with less hardware. Placing simulated configurations to the cloud makes the hardware usage more effective. The case company has developed a cloud-based testing service for embedded systems which are in this case 4G base stations. This study investigated how well the service fulfils the target team’s needs for testing. Testers have a different kind of requirements for cloud-based testing. Requirements were split into four categories from the qualitative data which was collected by interviewing testers of the target team. Requirement categories are test environments, test automation development, test execution and partly-simulated system under tests, more commonly known as SUTs. Four tests were implemented to the cloud with Robot Framework which is a test automation tool for developing automated tests. An empirical data showed that executing cloud-based tests is not always so fast due to long waiting times of getting a test environment from the cloud. However, when test environments were received they have initialised automatically with default settings and required testing tools. That reduces testers’ workload because locally built test environments require a lot of manual work like maintaining test environments. Beside other research questions, this study investigated fault detection capability of partly-simulated SUTs. 34 Cloud-based tests with partly-simulated SUTs were executed. 26 Tests passed and 8 tests failed. The simulator’s software caused only one failure and tested software caused other seven failures. From these statistics, we can at least say that the partly-simulated SUT can find some faults. The study also investigated a specific fault which was not found in the simulator even though it was found in the real SUT, which was a clear disadvantage.