LLM-Based Exploratory Testing Charter Generation:A Framework and Empirical Evaluation

Abstract Exploratory testing is a widely adopted practice in software quality as-surance, yet the authoring of structured testing charters — the artefacts that guideexploratory sessions — remains an entirely manual activity. No automated approachto charter generation has been proposed or empirically evaluated in the literature. Thispaper presents the Exploratory Testing Charter Generator (ETCG), a framework thatapplies GPT-4o, a transformer-based large language model, to generate structuredexploratory testing charters from software requirement specifications. ETCG employsa four-component structured prompt architecture — role instruction, context injec-tion, output format constraint, and generation rules — to produce five charters perspecification in a validated JSON schema encoding target area, risk focus, exploratoryapproach, priority, and estimated session duration. We evaluate ETCG against two comparison conditions — a role-instructed inter-mediate baseline (same guidance, no schema) and an unstructured prompt baseline —across 25 specifications, scoring 375 charters (125 per condition) using a purpose-builtfive-dimension rubric (Specificity, Testability, Risk Coverage, Clarity, Actionability)with inter-rater reliability validated against an independent reviewer (observed agree-ment 97.2% within one scale point across 250 dimension ratings; n = 50 charters). Thetwo-baseline design reveals that role framing and explicit guidance (Intermediate con-dition) account for the primary quality gain over the unstructured baseline (+3.20pp,SD reduced from 14.54% to 8.23%), while the structured JSON output schema (ETCGversus Intermediate) contributes negligible mean difference (−0.26pp). No pairwiseoverall comparison reaches statistical significance due to a ceiling effect (72–80%of charters at maximum score). The schema’s contribution is dimension-specific:Risk Coverage improvement is driven by role framing and guidance (Intermediateversus Baseline, p = 0.013), while the schema slightly constrains Clarity relativeto a free-form guided prompt (ETCG versus Intermediate, p = 0.018). Preliminaryevidence suggests that input specification richness moderates framework performance:on structured specifications (n = 7 ), the framework achieves near-perfect quality and consistency (99.81% ± 1.13%); this finding warrants replication with a largerstructured-specification set.