[論文レビュー] FOCAL: Contrastive Learning for Multimodal Time-Series Sensing Signals in Factorized Orthogonal Latent Space

Shengzhong Liu, Tomoyoshi Kimura+6 more|arXiv (Cornell University)|Oct 30, 2023

Music and Audio Processing被引用数 8

ひとこと要約

FOCALは自己教師付き対照フレームワークを導入し、マルチモーダル時系列データの表現を共有特徴とモダリティ専有特徴の双方を学習する因子分解された直交潜在空間で獲得し、表現と下流タスクを改善する時間局所性制約を導入する。

ABSTRACT

This paper proposes a novel contrastive learning framework, called FOCAL, for extracting comprehensive features from multimodal time-series sensing signals through self-supervised training. Existing multimodal contrastive frameworks mostly rely on the shared information between sensory modalities, but do not explicitly consider the exclusive modality information that could be critical to understanding the underlying sensing physics. Besides, contrastive frameworks for time series have not handled the temporal information locality appropriately. FOCAL solves these challenges by making the following contributions: First, given multimodal time series, it encodes each modality into a factorized latent space consisting of shared features and private features that are orthogonal to each other. The shared space emphasizes feature patterns consistent across sensory modalities through a modal-matching objective. In contrast, the private space extracts modality-exclusive information through a transformation-invariant objective. Second, we propose a temporal structural constraint for modality features, such that the average distance between temporally neighboring samples is no larger than that of temporally distant samples. Extensive evaluations are performed on four multimodal sensing datasets with two backbone encoders and two classifiers to demonstrate the superiority of FOCAL. It consistently outperforms the state-of-the-art baselines in downstream tasks with a clear margin, under different ratios of available labels. The code and self-collected dataset are available at https://github.com/tomoyoshki/focal.

研究の動機と目的

• 共有情報とモダリティ限定情報の両方を活用して、マルチモーダル時系列表現の理解を促進する。
• 共有特徴と私有特徴を分離する因子分解直交潜在空間を開発する。
• 事前訓練時の時系列構造を尊重する時間局所性制約を導入する。
• 共有、私有、直交性、時間的制約の損失項を concrete に設計する。
• 複数のデータセットとバックボーンにわたり下流性能の向上を示す。

提案手法

• 各モダリティを共有部と私有部を含む因子分解潜在空間にエンコードする。
• 共有空間と私有空間のモダリティ別・モダリティ間対照損失を適用する（共有空間は InfoNCE、私有空間は NT-Xent）。
• 共有部と私有部の直交性、さらには異なるモダリティの私有空間間の直交性を強制する。
• 事前訓練中の平均 intra-sequence 距離が平均 inter-sequence 距離以下となるよう時間局所性制約を課す。
• 記憶バンクを用いず、モダリティごとのランダムデータ拡張と同バッチ内対照を用いて学習する。

Figure 1: Overview of the FOCAL framework. Best viewed in color.

実験結果

リサーチクエスチョン

• RQ1 モーダル共有情報とモーダル個別情報を明示的にモデル化することで、マルチモーダル時系列表現はどのように豊かになるか？
• RQ2 共有空間と私有空間の直交性を課すと、下流の識別性能は改善されるか？
• RQ3 自己教師付き事前学習中に時間的局所性制約は時系列構造をより適切に反映できるか？
• RQ4 FOCAL の各成分は多様なデータセットで下流の分類・クラスタリングタスクにどのような影響を与えるか？
• RQ5 FOCAL はラベル希少性に頑健で、バックボーンエンコーダの移植性があるか？

主な発見

Dataset	Encoder	Framework	Acc	F1	Acc	F1	Acc	F1	Acc	F1
MOD	DeepSense	Supervised	0.9404	0.9399	0. -	0. -	0. -	0. -	0. -	0. -
MOD	DeepSense	SimCLR	0.8855	0.8855	0.7438	0.6101	0.7138	0.6841	0.6802	0.6583
MOD	DeepSense	MoCo	0.8808	0.8812	0.7717	0.6205	0.7859	0.7708	0.7559	0.7387
MOD	DeepSense	CMC	0.9196	0.9186	0.8443	0.7244	0.7975	0.8116	0.7906	0.7706
MOD	DeepSense	MAE	0.5981	0.5993	0.6644	0.5618	0.7565	0.7515	0.7114	0.6158
MOD	DeepSense	Cosmo	0.8989	0.8998	0.8511	0.6929	0.8956	0.8888	0.8356	0.8135
MOD	DeepSense	Cocoa	0.8774	0.8764	0.6644	0.5359	0.8465	0.8488	0.7603	0.7187
MOD	DeepSense	MTSS	0.4153	0.3582	0.4352	0.2441	0.2989	0.1405	0.3541	0.1795
MOD	DeepSense	TS2Vec	0.7669	0.7648	0.5224	0.3587	0.6595	0.5984	0.5729	0.4715
MOD	DeepSense	GMC	0.9257	0.9267	0.9096	0.7929	0.8869	0.8948	0.8119	0.7860
MOD	DeepSense	TNC	0.9518	0.9528	0.8237	0.6936	0.8892	0.8971	0.8387	0.8143
MOD	DeepSense	TS-TCC	0.8707	0.8735	0.7667	0.6164	0.8073	0.8010	0.7776	0.7250
MOD	DeepSense	FOCAL	0.9732	0.9729	0.9516	0.8580	0.9382	0.9290	0.8588	0.8463
MOD	SW-T	Supervised	0.8948	0.8931	0.9137	0.7770	0.9313	0.9278	0.8612	0.8384
MOD	SW-T	SimCLR	0.9250	0.9247	0.9128	0.8144	0.7046	0.7220	0.7705	0.7424
MOD	SW-T	MoCo	0.9390	0.9384	0.9174	0.8100	0.7813	0.8024	0.7717	0.7313
MOD	SW-T	CMC	0.9129	0.9105	0.8128	0.6857	0.8840	0.8955	0.8080	0.7901
MOD	SW-T	MAE	0.7803	0.7772	0.8516	0.7023	0.8829	0.8813	0.7910	0.7606
MOD	SW-T	Cosmo	0.3429	0.3378	0.7110	0.6086	0.8604	0.8169	0.7741	0.7366
MOD	SW-T	Cocoa	0.7040	0.7038	0.7096	0.5794	0.8892	0.8861	0.7689	0.7317
MOD	SW-T	MTSS	0.4206	0.4163	0.3429	0.2250	0.5136	0.4370	0.2847	0.1714
MOD	SW-T	TS2Vec	0.7254	0.7174	0.7183	0.5748	0.6151	0.5955	0.6195	0.5426
MOD	SW-T	GMC	0.8640	0.8611	0.9402	0.7766	0.9319	0.9379	0.8312	0.8083
MOD	SW-T	TNC	0.8533	0.8539	0.8352	0.7372	0.8817	0.8784	0.8013	0.7506
MOD	SW-T	TS-TCC	0.8734	0.8735	0.9041	0.7547	0.8731	0.8454	0.7997	0.7260
MOD	SW-T	FOCAL	0.9805	0.9800	0.9489	0.8262	0.9451	0.9503	0.8580	0.8401
ACIDS	DeepSense	Supervised	0.9566	0.8407	0. -	0. -	0. -	0. -	0. -	0. -
ACIDS	DeepSense	SimCLR	0.7438	0.6101	0. -	0. -	0. -	0. -	0. -	0. -
ACIDS	DeepSense	MoCo	0.7717	0.6205	0. -	0. -	0. -	0. -	0. -	0. -
ACIDS	DeepSense	CMC	0.8443	0.7244	0. -	0. -	0. -	0. -	0. -	0. -
ACIDS	DeepSense	MAE	0.6644	0.5618	0. -	0. -	0. -	0. -	0. -	0. -
ACIDS	DeepSense	Cosmo	0.8511	0.6929	0. -	0. -	0. -	0. -	0. -	0. -
ACIDS	DeepSense	Cocoa	0.6644	0.5359	0. -	0. -	0. -	0. -	0. -	0. -
ACIDS	DeepSense	MTSS	0.4352	0.2441	0. -	0. -	0. -	0. -	0. -	0. -
ACIDS	DeepSense	TS2Vec	0.5224	0.3587	0. -	0. -	0. -	0. -	0. -	0. -
ACIDS	DeepSense	GMC	0.9096	0.7929	0. -	0. -	0. -	0. -	0. -	0. -
ACIDS	DeepSense	TNC	0.8237	0.6936	0. -	0. -	0. -	0. -	0. -	0. -
ACIDS	DeepSense	TS-TCC	0.7667	0.6164	0. -	0. -	0. -	0. -	0. -	0. -
ACIDS	DeepSense	FOCAL	0.9516	0.8580	0. -	0. -	0. -	0. -	0. -	0. -
ACIDS	SW-T	Supervised	0.9137	0.7770	0. -	0. -	0. -	0. -	0. -	0. -
ACIDS	SW-T	SimCLR	0.9128	0.8144	0. -	0. -	0. -	0. -	0. -	0. -
ACIDS	SW-T	MoCo	0.9174	0.8100	0. -	0. -	0. -	0. -	0. -	0. -
ACIDS	SW-T	CMC	0.8128	0.6857	0. -	0. -	0. -	0. -	0. -	0. -
ACIDS	SW-T	MAE	0.8516	0.7023	0. -	0. -	0. -	0. -	0. -	0. -
ACIDS	SW-T	Cosmo	0.7110	0.6086	0. -	0. -	0. -	0. -	0. -	0. -
ACIDS	SW-T	Cocoa	0.7096	0.5794	0. -	0. -	0. -	0. -	0. -	0. -
ACIDS	SW-T	MTSS	0.3429	0.2250	0. -	0. -	0. -	0. -	0. -	0. -
ACIDS	SW-T	TS2Vec	0.7183	0.5748	0. -	0. -	0. -	0. -	0. -	0. -
ACIDS	SW-T	GMC	0.9402	0.7766	0. -	0. -	0. -	0. -	0. -	0. -
ACIDS	SW-T	TNC	0.8352	0.7372	0. -	0. -	0. -	0. -	0. -	0. -
ACIDS	SW-T	TS-TCC	0.9041	0.7547	0. -	0. -	0. -	0. -	0. -	0. -
ACIDS	SW-T	FOCAL	0.9489	0.8262	0. -	0. -	0. -	0. -	0. -	0. -
RealWorld-HAR	DeepSense	Supervised	0.9348	0.9388	0. -	0. -	0. -	0. -	0. -	0. -
RealWorld-HAR	DeepSense	SimCLR	0.7138	0.6841	0. -	0. -	0. -	0. -	0. -	0. -
RealWorld-HAR	DeepSense	MoCo	0.7859	0.7708	0. -	0. -	0. -	0. -	0. -	0. -
RealWorld-HAR	DeepSense	CMC	0.7975	0.8116	0. -	0. -	0. -	0. -	0. -	0. -
RealWorld-HAR	DeepSense	MAE	0.7565	0.7515	0. -	0. -	0. -	0. -	0. -	0. -
RealWorld-HAR	DeepSense	Cosmo	0.8956	0.8888	0. -	0. -	0. -	0. -	0. -	0. -
RealWorld-HAR	DeepSense	Cocoa	0.8465	0.8488	0. -	0. -	0. -	0. -	0. -	0. -
RealWorld-HAR	DeepSense	MTSS	0.2989	0.1405	0. -	0. -	0. -	0. -	0. -	0. -
RealWorld-HAR	DeepSense	TS2Vec	0.6595	0.5984	0. -	0. -	0. -	0. -	0. -	0. -
RealWorld-HAR	DeepSense	GMC	0.8869	0.8948	0. -	0. -	0. -	0. -	0. -	0. -
RealWorld-HAR	DeepSense	TNC	0.8892	0.8971	0. -	0. -	0. -	0. -	0. -	0. -
RealWorld-HAR	DeepSense	TS-TCC	0.8073	0.8010	0. -	0. -	0. -	0. -	0. -	0. -
RealWorld-HAR	DeepSense	FOCAL	0.9382	0.9290	0. -	0. -	0. -	0. -	0. -	0. -
RealWorld-HAR	SW-T	Supervised	0.9313	0.9278	0. -	0. -	0. -	0. -	0. -	0. -
RealWorld-HAR	SW-T	SimCLR	0.7046	0.7220	0. -	0. -	0. -	0. -	0. -	0. -
RealWorld-HAR	SW-T	MoCo	0.7813	0.8024	0. -	0. -	0. -	0. -	0. -	0. -
RealWorld-HAR	SW-T	CMC	0.8840	0.8955	0. -	0. -	0. -	0. -	0. -	0. -
RealWorld-HAR	SW-T	MAE	0.8829	0.8813	0. -	0. -	0. -	0. -	0. -	0. -
RealWorld-HAR	SW-T	Cosmo	0.8604	0.8169	0. -	0. -	0. -	0. -	0. -	0. -
RealWorld-HAR	SW-T	Cocoa	0.774?	0. -	0. -	0. -	0. -	0. -	0. -	0. -
PAMAP2	DeepSense	Supervised	0.8849	0.8761	0. -	0. -	0. -	0. -	0. -	0. -
PAMAP2	DeepSense	SimCLR	0.6802	0.6583	0. -	0. -	0. -	0. -	0. -	0. -
PAMAP2	DeepSense	MoCo	0.7559	0.7387	0. -	0. -	0. -	0. -	0. -	0. -
PAMAP2	DeepSense	CMC	0.7906	0.7706	0. -	0. -	0. -	0. -	0. -	0. -
PAMAP2	DeepSense	MAE	0.7114	0.6158	0. -	0. -	0. -	0. -	0. -	0. -
PAMAP2	DeepSense	Cosmo	0.8356	0.8135	0. -	0. -	0. -	0. -	0. -	0. -
PAMAP2	DeepSense	Cocoa	0.7603	0.7187	0. -	0. -	0. -	0. -	0. -	0. -
PAMAP2	DeepSense	MTSS	0.3541	0.1795	0. -	0. -	0. -	0. -	0. -	0. -
PAMAP2	DeepSense	TS2Vec	0.5729	0.4715	0. -	0. -	0. -	0. -	0. -	0. -
PAMAP2	DeepSense	GMC	0.8119	0.8083	0. -	0. -	0. -	0. -	0. -	0. -
PAMAP2	DeepSense	TNC	0.8387	0.8143	0. -	0. -	0. -	0. -	0. -	0. -
PAMAP2	DeepSense	TS-TCC	0.7776	0.7250	0. -	0. -	0. -	0. -	0. -	0. -
PAMAP2	DeepSense	FOCAL	0.8588	0.8463	0. -	0. -	0. -	0. -	0. -	0. -
PAMAP2	SW-T	Supervised	0.8612	0.8384	0. -	0. -	0. -	0. -	0. -	0. -
PAMAP2	SW-T	SimCLR	0.7705	0.7424	0. -	0. -	0. -	0. -	0. -	0. -
PAMAP2	SW-T	MoCo	0.7717	0.7313	0. -	0. -	0. -	0. -	0. -	0. -
PAMAP2	SW-T	CMC	0.8080	0.7901	0. -	0. -	0. -	0. -	0. -	0. -
PAMAP2	SW-T	MAE	0.7910	0.7606	0. -	0. -	0. -	0. -	0. -	0. -
PAMAP2	SW-T	Cosmo	0.7741	0.7366	0. -	0. -	0. -	0. -	0. -	0. -
PAMAP2	SW-T	Cocoa	0.7689	0.7317	0. -	0. -	0. -	0. -	0. -	0. -
PAMAP2	SW-T	MTSS	0.284?	0.1714	0. -	0. -	0. -	0. -	0. -	0. -
PAMAP2	SW-T	TS2Vec	0.6195	0.5426	0. -	0. -	0. -	0. -	0. -	0. -
PAMAP2	SW-T	GMC	0.8312	0.8083	0. -	0. -	0. -	0. -	0. -	0. -
PAMAP2	SW-T	TNC	0.8013	0.7506	0. -	0. -	0. -	0. -	0. -	0. -
PAMAP2	SW-T	TS-TCC	0.7997	0.7260	0. -	0. -	0. -	0. -	0. -	0. -
PAMAP2	SW-T	FOCAL	0.8580	0.8401	0. -	0. -	0. -	0. -	0. -	0. -

• FOCAL は 4 つのマルチモーダルデータセットと 2 つのバックボーンで、線形探索と KNN 微調整のベースライン11件を一貫して上回る。
• 私有空間と直交性制約は、共有情報のみを用いる場合より顕著な利得を生む。
• 時間局所性制約は収束を早め、事前訓練時の意味構造を改善する。
• 時間的制約は FO CAL 自体のフレームワークを超えた複数のベースラインにもプラグインとして改善効果を発揮する。
• アブレーションでは私有空間または直交性を取り除くと性能が低下し、時間制約は追加の利得を提供する。

Figure 2: Information diagram between CMC and the proposed FOCAL. Figure adapted from [ tian2020contrastive ] . Blue color denotes used information sectors.