[Paper Review] Position-aware Self-attention with Relative Positional Encodings for Slot Filling

Ivan Bilan, Benjamin Roth|arXiv (Cornell University)|Jul 9, 2018

Tactile and Sensory InteractionsNeuroscience11 references21 citations

TL;DR

This paper proposes a self-attention-based model with relative positional encodings and a position-aware attention layer for slot filling in relation extraction. By replacing LSTMs with a self-attention encoder and incorporating relative positional encodings, the model achieves state-of-the-art performance on the TACRED dataset, outperforming prior methods while relying solely on attention mechanisms.

ABSTRACT

This paper describes how to apply self-attention with relative positional encodings to the task of relation extraction. We propose to use the self-attention encoder layer together with an additional position-aware attention layer that takes into account positions of the query and the object in the sentence. The self-attention encoder also uses a custom implementation of relative positional encodings which allow each word in the sentence to take into account its left and right context. The evaluation of the model is done on the TACRED dataset. The proposed model relies only on attention (no recurrent or convolutional layers are used), while improving performance w.r.t. the previous state of the art.

Motivation & Objective

• To develop a purely attention-based model for relation extraction that avoids recurrent or convolutional components.
• To improve performance on slot filling by integrating position-aware attention that considers subject and object positions in a sentence.
• To enhance self-attention mechanisms by replacing absolute positional encodings with relative positional encodings for better modeling of token relationships.
• To evaluate the effectiveness of architectural modifications such as activation functions, weight initialization, and layer configurations on relation classification performance.

Proposed method

• The model uses a self-attention encoder with a custom implementation of relative positional encodings to model left and right context for each token.
• A position-aware attention layer is added on top of the self-attention encoder to explicitly model interactions between query and object tokens and their relative positions.
• Relative positional encodings are represented using a binning strategy that groups distant tokens into larger bins to improve generalization.
• The model employs multi-head self-attention with query, key, and value transformations, followed by a feed-forward network in each encoder layer.
• The attention mechanism computes compatibility scores via softmax-normalized dot products between queries and keys, producing weighted outputs over values.
• The final classification head uses the context-aware representation to predict one of 42 predefined relations or 'no_relation'.

Experimental results

Research questions

• RQ1 Can a self-attention-only architecture outperform sequence-based models like LSTMs in relation extraction for slot filling?
• RQ2 How does using relative positional encodings improve performance compared to absolute positional encodings in self-attention for this task?
• RQ3 To what extent does adding a position-aware attention layer enhance performance when combined with a self-attention encoder?
• RQ4 What is the impact of architectural choices such as activation functions, weight initialization, and layer depth on model performance?

Key findings

• The proposed model achieves state-of-the-art performance on the TACRED dataset, outperforming previous SOTA methods that used LSTMs or hybrid architectures.
• Using relative positional encodings instead of absolute encodings improves the F₁-score by 0.2%, demonstrating their effectiveness in modeling long-range dependencies.
• The self-attention encoder alone achieves a high recall of 85.4% but only 26.7% precision, indicating a need for additional supervision to balance performance.
• Replacing RReLU with ReLU reduces the F₁-score by 0.3%, showing that RReLU provides a slight but consistent performance advantage.
• Using Kaiming weight initialization instead of Xavier results in a marginal increase in precision with no change in F₁, suggesting minimal impact on overall performance.
• Combining the self-attention encoder with an LSTM for hidden state computation does not yield a significant performance gain, indicating the self-attention component is sufficient.