Experiments on quality evaluation of embedded software in Japan robot software design contest

As a practical opportunity for educating Japanese young developers in the field of embedded software development, a software design contest involving the design of software to automatically control a line-trace robot, and conduct running performance tests was held. In this paper, we give the results of the contest from the viewpoint of software quality evaluation. We create a framework for evaluating the software quality which integrated design model quality and the final system performance, and conduct analysis using the framework. As a result of analysis, it is found that the quantitative measurement of the structural complexity of the design models bears a strong relationship to qualitative evaluation of the design conducted by judges. It is also found that there is no strong correlation between design model quality evaluated by the judges and the final system performance. For embedded software development, it is particularly important to estimate and verify reliability and performance in the early stages, using the model. Based on the analysis result, we consider possible remedies with respect to the models submitted, the evaluation methods used and the contest specifications. In order to adequately measure several non-functional quality characteristics including performance on the model, it is necessary to improve the way of developing robot software (such as applying model driven development) and reexamine the evaluation methods.