Cloud-Based Big Data Observability Frameworks for Healthcare Analytics Platforms

Cloud computing platforms for big data analytics, such as Amazon Web Services, Google Cloud Platform, or Microsoft Azure, have entered a mainstream growth phase, yet despite the enormous market demand, their effective operation remains a challenging task. Monitoring and observability of big data workloads needing support for dynamic and complex infrastructure represent an important area of research. Healthcare analytics platforms are particularly demanding in this respect, as workloads are commonly performing computations over patient data, which introduces additional requirements. During the last few years, several frameworks have been proposed to address various observability needs for cloud-based platforms, yet information on those approaches remains scattered. This survey work provides a structured overview of definitions, architectural patterns, frameworks, and tools, along with special focus on healthcare observability requirements. Furthermore, the discussion identifies potential areas for future investigation. The cloud observability area is still evolving, making it necessary to review the state-of-the-art, identify gaps, and propose a research agenda. Foundation work covers an analysis of core concepts and architectural patterns for big data observability in the cloud, followed by the examination of requirements driven by healthcare workloads. These aspects have laid the groundwork for a survey of existing observability frameworks and tools tailored to cloud platforms, with special emphasis placed on monitoring telemetry collection, distributed tracing, and performance management. Finally, additional data management considerations are discussed before the identification of architectural patterns that address the specific observability demands from healthcare data processing workloads.