[Paper Review] Time2Vec: Learning a Vector Representation of Time

Seyed Mehran Kazemi, Rishab Goel+8 more|arXiv (Cornell University)|Jul 11, 2019

Time Series Analysis and Forecasting59 references51 citations

TL;DR

Time2Vec introduces a learnable vector representation for time that can be plugged into various architectures, improving performance on both synthetic and real-world time-aware tasks.

ABSTRACT

Time is an important feature in many applications involving events that occur synchronously and/or asynchronously. To effectively consume time information, recent studies have focused on designing new architectures. In this paper, we take an orthogonal but complementary approach by providing a model-agnostic vector representation for time, called Time2Vec, that can be easily imported into many existing and future architectures and improve their performances. We show on a range of models and problems that replacing the notion of time with its Time2Vec representation improves the performance of the final model.

Motivation & Objective

• Motivate the need for a general-purpose time representation that captures both periodic and non-periodic patterns.
• Propose Time2Vec, a learnable time embedding that integrates with diverse models.
• Show that Time2Vec improves performance across multiple architectures and datasets.
• Analyze what Time2Vec learns about time and compare periodic versus non-periodic activations and fixed versus learned frequencies.

Proposed method

• Define Time2Vec as a vector of size k+1 with a linear term and k sine-based periodic terms, where both frequencies and phases are learnable.
• Use a periodic activation function (sine) for i = 1..k and a linear term for i = 0 to capture non-periodic progression.
• Replace the model’s time input tau with t2v(tau) and adjust architecture-specific time handling accordingly.
• Experiment with Time2Vec across multiple datasets and architectures (e.g., LSTM+T, LSTM+Time2Vec, TLSTM1, TLSTM3) to assess performance gains.
• Evaluate periodic versus non-periodic activations and the impact of learned versus fixed frequencies/phases.
• Provide ablation studies on the necessity of the linear term and the effect of learned frequencies.

Experimental results

Research questions

• RQ1 Is Time2Vec an effective representation for time across diverse tasks and datasets?
• RQ2 Can Time2Vec be integrated with different architectures to improve performance?
• RQ3 What patterns do the learned sine components capture about time (periodicity, frequencies)?
• RQ4 Do periodic activations outperform non-periodic activations in Time2Vec-based models?
• RQ5 Is learning sine frequencies/phases beneficial compared to fixing them?

Key findings

• Time2Vec improves performance over time as a raw feature across multiple datasets and architectures in the majority of cases.
• Replacing time with Time2Vec in TLSTM1 and TLSTM3 also yields performance gains on Last.FM and CiteULike.
• On synthetic data, Time2Vec learns underlying periodicity (e.g., a 7-day cycle) and aligns phase shifts to separate periodic events.
• Learning sine frequencies and phases outperforms fixed-frequency encodings in Event-MNIST experiments.
• Periodic activations (sine, mod, triangle) generally outperform non-periodic activations in Time2Vec.
• Including a linear term alongside sine components helps capture non-periodic temporal patterns and can improve extrapolation.