A Messy Handwriting Dataset with Student Crossouts and Corrections

Most existing studies and public datasets for Handwritten English Text Recognition (HTR) are based on organized and clean documents. There has been little work on messy documents. This paper addresses the recognition of messy handwritten documents that contain multiple novel challenges: strikethrough text, text line addition, noisy background, unbalanced text length and inserted text. The paper presents a process for collecting and annotating messy handwritten documents when no ground truth is available. We evaluated two current state-of-the-art text recognition methods for the messy handwritten dataset, convolutional recurrent neural network (CRNN) and Sequence to Sequence (Seq2Seq) with an attention mechanism. The results showed that both techniques exhibit poor performance on the new dataset, establishing that the new dataset provides significant new challenges for the HTR field. Through visualization and error evaluation, we observed that humans can easily avoid the majority of the error predictions, which indicates the limitations of the current methods for English HTR systems.Furthermore, considering the difficulties of annotating handwritten documents, we investigated the use of an existing clean dataset in combination with the messy handwritten dataset. The question is how best to train a robust model when limited messy data is available — transfer learning with messy data or train from scratch with a combination of messy and existing data. We concluded from the experimental work that the best way to get a robust model is to initialize with weights from a pre-trained model and use the same number of samples from both datasets during training.