Over-Sampling Effect in Pre-Training for Bidirectional Encoder Representations from Transformers (BERT) to Localize Medical BERT and Enhance Biomedical BERT (Preprint)

BACKGROUND Pre-training large-scale neural language models on raw texts has made a significant contribution to improving transfer learning in natural language processing. With the introduction of transformer-based language models, such as bidirectional encoder representations from transformers (BERT), the performance of information extraction from free text has significantly improved for both the general and medical domains; however, it is difficult to train specific BERT models that perform well for domains in which there are few publicly available databases of high quality and large size. OBJECTIVE We hypothesize that this problem can be addressed by over-sampling a domain-specific corpus and using it for pre-training with a larger corpus in a balanced manner. In this study, we verify our hypothesis by developing pre-training models using our method and evaluating their performance. METHODS Our proposed method was based on simultaneous pre-training after over-sampling. We conducted three experiments in which we generated (1) English biomedical BERT from a small biomedical corpus, (2) Japanese medical BERT from a small medical corpus, and (3) enhanced biomedical BERT pre-trained from complete PubMed abstracts in a balanced manner and compared their performance with the conventional models. RESULTS We first confirmed that our English BERT pre-trained using both general and small medical-domain corpora performed sufficiently well for practical use in the biomedical language understanding evaluation (BLUE) benchmark. Moreover, our proposed method was more effective than conventional methods for each different biomedical corpus size with the same corpus size for the general domain. Next, our Japanese medical BERT outperformed the other BERT models built using a conventional method concerning the medical document classification task. It demonstrated the same trend as in the first experiment in English. Lastly, our enhanced biomedical BERT model, in which clinical notes were not used during pre-training, achieved both clinical and biomedical scores on the BLUE benchmark that were 0.3 points above those of the model trained without our proposed method. CONCLUSIONS Well-balanced pre-training by over-sampling instances derived from a corpus appropriate for the target task allowed us to construct a high-performance BERT model.