ChriSmA2: A Structured Framework for IoT Platform Evaluation Inspired by Systematic Review Principles

The rapid growth of the Internet of Things (IoT), including both consumer oriented and Industrial Internet of Things (IIoT) <br> platforms, has given rise to a highly heterogeneous IoT platform landscape with significant differences in scalability, replicability, interoperability, deployment flexibility, and operation complexity, and hence selecting the most appropriate platform is a crucial issue in industrial and sustainability oriented engineering domains. This paper proposes a comprehensive and evidence based framework called ChriSmA2 to facilitate the systematic identification and evaluation of IoT platforms concerning their real world applicability, scalability, and replicability. The proposed framework is based on the application of PRISMA principles to a platform oriented evaluation process and can be used to perform a transparent and replicable screening and evaluation of both open source and proprietary IoT platforms. A comprehensive and unified set of evaluation criteria is defined for the assessment of IoT platforms from a multi dimensional perspective, such as community support, documentation quality, security features, deployment flexibility, platform maturity level, scalability, replicability, application domains, and operation limitations. The effectiveness of the proposed framework is validated through a large scale case study involving the systematic screening of a total of 255 candidate platforms and the subsequent evaluation of eighteen high quality platforms from different application domains such as smart cities, industrial automation, health care, and energy management. The evaluation results show a clear trade off between different types of platforms: while open source platforms show promising scalability and replicability due to the presence of a community driven and open nature of the platforms, they may face problems such as integration complexity and long term stability, whereas proprietary platforms may show better integration flexibility and operation robustness but may suffer from a lack of openness and flexibility. By providing a comprehensive and evidence based decision support framework through the synthesis of both technical and economic evidence, the proposed framework can guide engineers and decision makers to select sustainable and scalable IoT and IIoT platforms.