[Paper Review] k-FFNN: A priori knowledge infused Feed-forward Neural Networks

Sri Harsha Dumpala, Rupayan Chakraborty+1 more|arXiv (Cornell University)|Apr 24, 2017

Music and Audio Processing11 references3 citations

TL;DR

This paper proposes k-FFNN, a feed-forward neural network architecture that integrates a priori knowledge of temporal relationships in sequential data, enabling it to outperform standard RNNs in low-resource scenarios. By infusing known temporal correlations through learned functions, k-FFNN achieves comparable or superior performance to RNNs—especially with limited training data—demonstrating that explicit knowledge infusion can reduce reliance on large-scale RNN training.

ABSTRACT

Recurrent neural network (RNN) are being extensively used over feed-forward neural networks (FFNN) because of their inherent capability to capture temporal relationships that exist in the sequential data such as speech. This aspect of RNN is advantageous especially when there is no a priori knowledge about the temporal correlations within the data. However, RNNs require large amount of data to learn these temporal correlations, limiting their advantage in low resource scenarios. It is not immediately clear (a) how a priori temporal knowledge can be used in a FFNN architecture (b) how a FFNN performs when provided with this knowledge about temporal correlations (assuming available) during training. The objective of this paper is to explore k-FFNN, namely a FFNN architecture that can incorporate the a priori knowledge of the temporal relationships within the data sequence during training and compare k-FFNN performance with RNN in a low resource scenario. We evaluate the performance of k-FFNN and RNN by extensive experimentation on MediaEval 2016 audio data ("Emotional Impact of Movies" task). Experimental results show that the performance of k-FFNN is comparable to RNN, and in some scenarios k-FFNN performs better than RNN when temporal knowledge is injected into FFNN architecture. The main contributions of this paper are (a) fusing a priori knowledge into FFNN architecture to construct a k-FFNN and (b) analyzing the performance of k-FFNN with respect to RNN for different size of training data.

Motivation & Objective

• Address the challenge of poor performance in RNNs when training data is scarce.
• Investigate whether a priori knowledge of temporal relationships can enhance FFNN performance in sequential data tasks.
• Develop a novel FFNN architecture that incorporates known temporal dependencies without relying on recurrent feedback loops.
• Compare k-FFNN performance against RNNs and standard FFNNs across varying training data sizes.
• Demonstrate that explicit knowledge infusion can compensate for the lack of large-scale data in sequence modeling.

Proposed method

• Design k-FFNN by modifying the standard FFNN to include a priori temporal knowledge as input features.
• Represent temporal relationships using three distinct knowledge-infused functions (Fn1, Fn2, Fn3), each encoding different assumptions about sequential dependencies.
• Train k-FFNN using the same loss functions (MSE, PCC) as RNN and FFNN for direct comparison.
• Use the MediaEval 2016 audio dataset for emotional impact prediction, with arousal and valence as regression targets.
• Evaluate performance across multiple training set sizes (from 200 to 6814 samples) to assess data efficiency.
• Compare k-FFNN with RNN, LSTM, BLSTM, and standard FFNN using MSE and Pearson correlation coefficient (PCC).

Experimental results

Research questions

• RQ1 Can a priori knowledge of temporal relationships in sequential data be effectively integrated into a feed-forward neural network?
• RQ2 How does the performance of k-FFNN compare to RNNs when training data is limited?
• RQ3 Does the choice of temporal knowledge representation (e.g., Fn1 vs. Fn2 vs. Fn3) significantly affect k-FFNN performance?
• RQ4 Can k-FFNN outperform RNNs in low-resource settings where RNNs struggle due to insufficient data?
• RQ5 Is there a performance saturation point where k-FFNN and RNN converge as training data increases?

Key findings

• k-FFNN with Fn1 achieved a PCC of 0.274 on arousal prediction using only 200 training samples, outperforming RNN (PCC = 0.079) in the same low-data regime.
• For valence prediction with 6814 training samples, k-FFNN (Fn1) achieved a PCC of 0.128, slightly better than RNN (PCC = 0.126), and significantly better than FFNN (PCC = 0.106).
• The MSE for k-FFNN (Fn1) on valence prediction was 0.319, lower than RNN (0.331), LSTM (0.327), and BLSTM (0.329), indicating superior regression performance.
• Performance degradation occurred when using Fn2 and Fn3, with PCC values dropping to 0.185 and 0.059 respectively, showing that improper knowledge representation harms performance.
• As training data increased, the performance gap between k-FFNN and RNN narrowed, with both systems converging in performance at larger data scales.
• k-FFNN with Fn1 consistently outperformed standard FFNN across all training set sizes, confirming the benefit of knowledge infusion.