Evaluating the ARF diagnosis calculator: A survey and content analysis. (Preprint)

BACKGROUND Acute Rheumatic Fever (ARF) is a critically important condition for which there is no diagnostic test. Diagnosis requires the use of a set of criteria comprising clinical, laboratory, electrocardiographic and echocardiographic findings. The complexity of the algorithm and the fact that clinicians lack familiarity with ARF, make ARF diagnosis ideally suited to an electronic decision support tool. We developed an ARF Diagnosis Calculator to assist clinicians in diagnosing ARF and correctly assigning categories of ‘possible, ‘probable’ or ‘definite’ ARF. OBJECTIVE To evaluate the acceptability and accuracy of the ARF Diagnosis Calculator as perceived by clinicians in Northern Australia where ARF rates are high, and test performance against a ‘gold standard’. METHODS Three strategies were used to provide triangulation of data. Users of the calculator employed at Top End Health Service, Northern Territory, Australia were invited to participate in an online survey about the calculator, and clinicians with ARF expertise were invited to participate in semi-structured interviews. Qualitative data were analysed using inductive analysis. Performance of the calculator in correctly assigning a diagnosis of possible, probable or definite ARF, or not ARF, was assessed using clinical data from 35 patients presenting with suspected ARF. Diagnoses obtained from the calculator were compared using the Kappa statistic with those obtained from a panel of expert clinicians, considered the ‘gold standard’. Findings were shared with developers of the calculator and changes were incorporated. RESULTS Survey responses were available from 23 Top End Health Service medical practitioners, and interview data were available from five expert clinicians. Using a 6-point Likert scale, participants highly recommended the ARF Diagnosis Calculator (median score 6, IQR 1) and found it easy to use (median 5, IQR 1). Participants believed the calculator helped them diagnose ARF (median 5, IQR 1). Valued features included educational content and laboratory test reference ranges. Criticisms included: too many pop-up messages to be clicked through; that it is less helpful in remote areas which lack access to investigation results; and the need for more clarity about actively excluding alternative diagnoses to avoid false-positive ARF diagnoses. Importantly, clinicians with ARF expertise noted that electronic decision making is not a substitute for clinical experience. There was high agreement between the ARF Diagnosis Calculator and the ‘gold standard’ ARF diagnostic process (κ=0.767, 95% CI: 0.568-0.967). However, incorrect assignment of diagnosis occurred in 4/35 (11%) patients highlighting the greater accuracy of expert clinical input for ambiguous presentations. Sixteen changes were incorporated into a revised version of the calculator. CONCLUSIONS The ARF Diagnosis Calculator is an easy-to-use, accessible tool, but it does not replace clinical expertise. Effective resources to support clinicians in diagnosing and managing ARF are critically important for improving the quality of care of ARF.