Workflows for continuous protein structure prediction benchmarking

CAMEO - Continuous Automated Model EvaluatiOn (https://www.cameo3d.org) – continuously evaluates the performance of public servers for 3D Protein Structure Prediction (3D), Quality Estimation (QE) and Residue-Residue Contact Prediction (CP). CAMEO[1] is hinged on the weekly PDB pre-release of amino acid sequences as part of the PDB release of experimentally determined macromolecular structures. Typically, 20 protein sequences are selected as the target set and submitted to the participating servers for CAMEO 3D and CP categories. For CAMEO QE model coordinates of the CAMEO 3D servers are subsequently submitted. All servers return predictions to CAMEO until the following Wednesday. For the evaluation, the coordinates released by the PDB are used as reference, and the assessments are then published on cameo3d.org for interactive performance analyses. For CAMEO 3D 6736 targets were evaluated over 375 weeks, rapidly accumulating data for testing new developments on unseen and independent data. The CAMEO team contributed to the development of the community-agnostic data model within OpenEBench[2], allowing unified access to benchmarking data. The CAMEO workflows have recently been ported to NextFlow, resulting in a more robust and parallel setup to run on e.g. OpenEBench resources. CAMEO evaluates various aspects of protein structure modeling with different scores. The developers of prediction servers benefit of rapid assessment of new developments and continuous monitoring of their public servers, thereby complementing the bi-annual CASP effort[3]. CAMEO allows life scientists to better understand which public modeling server is the most suited for their specific use case.