End-to-End Multimodal 16-Day Hatching Eggs Classification

Sixteen-day hatching eggs are divided into fertile eggs, waste eggs, and recovered eggs. Because different categories may have the same characteristics, they are difficult to classify. Few existing algorithms can successfully solve this problem. To this end, we propose an end-to-end deep learning network structure that uses multiple forms of signals. First, we collect the photoplethysmography (PPG) signal of the hatching eggs to obtain heartbeat information and photograph hatching eggs with a camera to obtain blood vessel pictures. Second, we use two different network structures to process the two kinds of signals: Temporal convolutional networks are used to process heartbeat information, and convolutional neural networks (CNNs) are used to process blood vessel pictures. Then, we combine the two feature maps and use the long short-term memory (LSTM) network to model the context and recognize the type of hatching eggs. The system is then trained with our dataset. The experimental results demonstrate that the proposed end-to-end multimodal deep learning network structure is significantly more accurate than using a single modal network. Additionally, the method successfully solves the 16-day hatching egg classification problem.