Modeling Temporal Dynamics with Function Approximation in Deep Spatio-Temporal Inference Network

Biologically inspired deep machine learning is an emerging framework for dealing with complex high-dimensional data. An unsupervised feature extraction deep learning architecture called Deep Spatio-Temporal Inference Network (DeSTIN) utilizes a hierarchy of computational nodes, where each node features a common algorithm for inference of temporal patterns. The nodes all are geared to online learning and offer a generalization component which uses clustering and mixture models, as well as a temporal dynamics module. The latter is designed for tabular representation but such techniques are notoriously ill-suited for scaling as they impose an O(N3) memory complexity. Instead, function approximation methods such as neural networks can serve as a more concise representation. In this work we present the results of DeSTIN on a popular problem, the MNIST data set of handwritten digits, using mixture models and function approximation to create a temporally evolving feature representation. We compare the results of the extracted features from DeSTIN under the tabular method and the function approximation method and contrast these results with our past work in this area.