To ensure the delivery of a reliable and stable robot system, an end-to-end quality control process is needed. The robot software should be tested as much as possible using automated software tests. However, only software testing would not guarantee the reliability of the product. Ultimately, the software is interfaced with additional software and/or hardware systems. Hence, system testing can be used as a solution that evaluates a fully integrated product. This blog explains how the system testing approach can be used to develop a reliable robot system and discusses different types of system tests that can be used.
Why system testing is important
To develop a stable and reliable robot, an end-to-end quality control process is important. Tests are needed to check the performance and reliability of the robot. System testing is the highest level of testing which tests the fully integrated system; software and hardware. System tests are performed after the automated integration tests, which were discussed in a previous blog post regarding continuous integration. The goal of system tests is to evaluate the compliance of the system with the requirements
System testing can be considered as a black-box test technique that evaluates the behavior of a robot system without peering with the internal structure of the application. With system testing, an end-to-end test is performed to prevent any unexpected system failure in a live customer environment. System testing validates the fully integrated system, including external peripherals such as buffering systems, in order to check how components interact with one another and with the robot system as a whole.
In addition, system testing is the perfect opportunity to experience and verify the behavior of the robot from an operator’s point of view. During development, engineers use different kinds of tooling for controlling and monitoring the implemented behaviors to make inspection easier, meaning they do not use the system as it will be used at a customer. For example, during the development phase, engineers do not use the user interface as it would be at the customer. Hence, if a button in the user interface is too small or dysfunctional, it is not noticed in the development phase. That is why system testing is so important. As one uses the user interface from an operator’s point of view, you immediately notice that the button is too small and unclickable and should be changed to improve the reliability of the system.
The process of system testing
To perform system testing, one should:
- Create a test plan
- Define complete test scenarios
- Perform the test based on a clear test description.
The test plan should be derived from the system requirements and acceptance criteria. In addition, the test cases need to be linked to the corresponding system requirements so that they can verify the coverage of system requirements. System tests can be performed for every software release to ensure the robot system meets the quality standards for each release. There are various types of system tests that can be used to ensure a reliable robot system:
Happy-flow system tests
Happy-flow tests can be used to check default scenarios and functions of the system. For example, when considering an item picking robot, happy-flow consists of correctly picking and placing items in pre-defined positions.
Error-handling tests can be used to test all situations that lead to a known error. For example, if an item picking robot loses an item during pick and place action, operator intervention is needed and an error is raised. The error can be related to any external components, lost connection, application errors and so on. The error-handling test makes sure the possible causes for system errors are tested.
Recovery tests are performed to demonstrate that a reliable solution is available to successfully recover the robot system from possible errors and exceptions.
Configuration tests check the robot system using different possible configurations of software and hardware to ensure the system can work without any flaws in all needed states. For example, customers may need different robot speeds in their production line. Hence, configuration tests check if the application is guaranteed to work for all possible robot speeds.
It is also necessary to know that a robot system will work properly under extreme conditions. Considering the item picker example again, extreme tests can include testing various sizes and masses of items and totes.
Finally, during software testing, a system may behave exactly as expected during 1-hour tests. However, if the same system is tested for 30 hours straight, unforeseen problems may occur and cause the system to fail or behave randomly. That is why it is important to use endurance tests to test the happy-flow of a robot over a significant period of time.
A stable and reliable robot
As has been described above, using system testing to test a robot system as a whole is an important part of any successful quality control process. After all, one needs to ensure the robot will be fully functional at the customer and will perform its tasks as required. That is why system testing has been implemented within Smart Robotics; to ensure we deliver stable and reliable robot systems to our customers.