Harmonizing QSAR Machine Learning‐Based Models and Docking Approaches for Identifying Novel Histone Deacetylase 2 Inhibitors

AbstractMachine learning (ML) algorithms have gained widespread application in constructing computational models for predicting the bioactivity and physicochemical properties of numerous compounds, notably HDAC inhibitors. In this work, 2801 unique compounds with confirmed bioassays on HDAC2 were collected and employed to train ML models to virtually screening and possibly design potential inhibitors for HDAC2. A meticulous 3‐step procedure was first proposed to ensure the hits are within the application domain of the screening models. Through virtual screening of 91 million compounds, 59 structures were suggested by the four highest predictability models as potential HDAC2 inhibitors. The bioactivity of these compounds is further confirmed through a validated docking protocol, wherein 57 out of the 59 active hits proposed by the four models exhibited superior affinity with HDAC2 compared to vorinostat, with docking scores ranging from −10.74 to −7.06 kcal/mol. Lead optimization strategies were then implemented on the top‐performing compound from the molecular docking scores, CID 122648337, enhancing its binding affinity and interactions within the HDAC2 active site. This optimization led to the creation of two novel lead compounds, demonstrating higher affinity to HDAC2 than the initial hits. The stability of lead compounds in the active site was confirmed via MD simulations.