Efficient Temporal Modeling for RGB-T Tracking

Ni Xinyu; Zeng Bi; Zhang Zhongxuan

doi:10.12052/gdutxb.240169

English Version Journal Login

您当前的位置：

首页 >

文章列表页 >

Efficient Temporal Modeling for RGB-T Tracking

更新时间：2026-05-27

- Efficient Temporal Modeling for RGB-T Tracking
- Vol. 43, Issue 2, Pages: 21-29(2026)
- 作者机构：
  
  广东工业大学计算机学院,广东,广州,510006
- 作者简介：
- 基金信息：
- DOI：10.12052/gdutxb.240169
  CLC： TP391
- Published：2026
- 稿件说明：
移动端阅览
倪新宇, 曾碧, 张仲轩. 基于高效时序建模的RGB-T跟踪[J]. 广东工业大学学报, 2026, 43(2): 21-29.
倪新宇, 曾碧, 张仲轩. 基于高效时序建模的RGB-T跟踪[J]. 广东工业大学学报, 2026, 43(2): 21-29. DOI： 10.12052/gdutxb.240169.

DOI： 10.12052/gdutxb.240169

摘要

可见光−热红外(RGB-Thermal，RGB-T)跟踪方法利用可见光和热红外图像的互补性，提升了在低光照以及恶劣天气等场景下目标跟踪的准确度。然而，现有的研究大多局限于图像级外观匹配，难以应对目标形变、复杂环境干扰等挑战。对此，本文提出了一种基于高效时序建模的跟踪方法(Efficient Temporal Modeling for RGB-T Tracking

ETMTrack) 。首先，利用时序信息建模，并改进特征融合模块以处理时序信息。然后，利用轻量级适配器微调改进热红外图像特征提取模块，提高模型对不同模态信息的特征提取能力，并降低计算显存，提高训练效率。最后，提出一种动态模板更新和选择方法，充分挖掘并利用时序信息，提高模型性能。本文模型在3个公开数据集上达到了最先进的性能，在遮挡、相似外观等挑战上表现出色，验证了时序建模跟踪算法的有效性和鲁棒性。

Abstract

RGB-Thermal (RGB-T) tracking methods utilize the complementarity of visible light and thermal infrared images to improve the accuracy of target tracking in the scenarios of low light conditions and adverse weather. However

most existing studies focus only on image-level appearance matching

making them difficult to cope with challenges of target deformation and interference under complex environments. To address this problem

a tracking method based on efficient temporal modeling is proposed. Firstly

the temporal information is modeled

and the feature fusion module is improved to process temporal information. Then

a lightweight adapter is used for fine-tuning to improve the feature extraction module for thermal infrared images

enhancing the model's ability to extract features from different modal information

reducing the computational memory usage

and improving the training efficiency. Finally

a dynamic template update and selection method is proposed to fully explore and utilize temporal information

thereby improving the model's performance. ETMTrack achieves state-of-the-art performance on three public datasets

and performs excellently in dealing with challenges such as occlusion and similar appearances

demonstrating the effectiveness and robustness of the tracking algorithm based on temporal modeling.

关键词

Keywords

references

GALLEGO J, PARDAS M, LANDABASO J L. Segmentation and tracking of static and moving objects in video surveillance scenarios[C]//2008 15th IEEE International Conference on Image Processing. San Diego: IEEE, 2008: 2716-2719.

LI J, ZHAN W, HU Y, et al . Generic tracking and probabilistic prediction framework and its application in autonomous driving[J ] . IEEE Transactions on Intelligent Transportation Systems, 2019, 21(9): 3634-3649.

LIN L, FAN H, ZHANG Z, et al . SwinTrack: a simple and strong baseline for transformer tracking[J ] . Advances in Neural Information Processing Systems, 2022, 35: 16743-16754.

CHEN B, LI P, BAI L, et al . Backbone is all your need: a simplified architecture for visual object tracking[C ] //European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 375-392.

SUN Y, CAO B, ZHU P, et al . Drone-based RGB-infrared cross-modality vehicle detection via uncertainty-aware learning[J ] . IEEE Transactions on Circuits and Systems for Video Technology, 2022, 32(10): 6700-6713.

FENG G, BAO L. Face recognition method in complex environment based on infrared visible fusion[J]. Journal of Guangdong University of Technology, 2024, 41(3): 62-70,109.

LI C, LIU L, LU A, et al . Challenge-aware RGBT tracking[C ] //European Conference on Computer Vision. Cham: Springer International Publishing, 2020: 222-237.

ZHANG P, ZHAO J, WANG D, et al . Visible-thermal UAV tracking: a large-scale benchmark and new baseline[C ] //Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 8886-8895.

LIU L, LI C, XIAO Y, et al . RGBT tracking via challenge-based appearance disentanglement and interaction[J ] . IEEE Transactions on Image Processing, 2024, 33: 1753-1767.

WANG H, LIU X, LI Y, et al . Temporal adaptive RGBT tracking with modality prompt[C ] //Proceedings of the AAAI Conference on Artificial Intelligence. Vancouver: IEEE, 2024: 5436-5444.

WU Y, BLASCH E, CHEN G, et al . Multiple source data fusion via sparse representation for robust visual tracking[C ] //14th International Conference on Information Fusion. Chicago: IEEE, 2011: 1-8.

ZHAI S, SHAO P, LIANG X, et al . Fast RGB-T tracking via cross-modal correlation filters[J ] . Neurocomputing, 2019, 334: 172-181.

LI C, WU X, ZHAO N, et al . Fusing two-stream convolutional neural networks for RGB-T object tracking[J ] . Neurocomputing, 2018, 281: 78-85.

ZHANG H, ZHANG L, ZHUO L, et al . Object tracking in RGB-T videos using modal-aware attention network and competitive learning[J ] . Sensors, 2020, 20(2): 393.

VASWANI A, SHAZEER N, PARMAR N, et al . Attention is all you need[J ] . Advances in Neural Information Processing Systems, 2017, 30: 6000-6010.

HUI T, XUN Z, PENG F, et al . Bridging search region interaction with template for RGB-t tracking[C ] //Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver: IEEE, 2023: 13630-13639.

ZHU J, LAI S, CHEN X, et al . Visual prompt multi-modal tracking[C ] //Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver: IEEE, 2023: 9516-9526.

YAN B, PENG H, FU J, et al . Learning spatio-temporal transformer for visual tracking[C ] //Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2021: 10448-10457.

CUI Y, JIANG C, WANG L, et al . Mixformer: end-to-end tracking with iterative mixed attention[C ] //Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 13608-13618.

XIE F, CHU L, LI J, et al . Videotrack: learning to track objects via video transformer[C ] //Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver: IEEE, 2023: 22826-22835.

WEI X, BAI Y, ZHENG Y, et al . Autoregressive visual tracking[C ] //Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver: IEEE, 2023: 9697-9706.

CAI Y, LIU J, TANG J, et al . Robust object modeling for visual tracking[C ] //Proceedings of the IEEE/CVF International Conference on Computer Vision. Paris: IEEE, 2023: 9589-9600.

ZHENG Y, ZHONG B, LIANG Q, et al . ODTrack: online dense temporal token learning for visual tracking[C ] //Proceedings of the AAAI Conference on Artificial Intelligence. Vancouver: IEEE, 2 024: 7588-7596.

HOU X, XING J, QIAN Y, et al . SDSTrack: self-distillation symmetric adapter learning for multi-modal visual object tracking[C ] //Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2024: 26551-26561.

HONG L, YAN S, ZHANG R, et al . OneTracker: unifying visual object tracking with foundation models and efficient tuning[C ] //Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2024: 19079-19091.

YE B, CHANG H, MA B, et al . Joint feature learning and relation modeling for tracking: a one-stream framework[C ] //European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 341-357.

LAW H, DENG J. Cornernet: detecting objects as paired keypoints[C]//Proceedings of the European Conference on Computer Vision (ECCV) . Munich: Springer, 2018: 734-750.

REZATOFIGHI H, TSOI N, GWAK J Y, et al . Generalized intersection over union: a metric and a loss for bounding box regression[C ] //Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 658-666.

LI C, XUE W, JIA Y, et al . LasHeR: a large-scale high-diversity benchmark for RGBT tracking[J ] . IEEE Transactions on Image Processing, 2021, 31: 392-404.

LI C, ZHAO N, LU Y, et al . Weighted sparse representation regularized graph learning for RGB-T object tracking[C ] //Proceedings of the 25th ACM International Conference on Multimedia. New York: Association for Computing Machinery, 2017: 1856-1864.

LI C, LIANG X, LU Y, et al . RGB-T object tracking: benchmark and baseline[J ] . Pattern Recognition, 2019, 96: 106977.

CAO B, GUO J, ZHU P, et al . Bi-directional adapter for multimodal tracking[C ] //Proceedings of the AAAI Conference on Artificial Intelligence. Vancouver: IEEE, 2024: 927-935.

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

A Survey of Deepfake Detection Techniques Based on Transformer

Temporal Alignment Style Control in Text-to-Speech Synthesis Algorithm

Face Recognition Method in Complex Environment Based on Infrared Visible Fusion

A Study of Power Quality Regulation Strategies of Energy Storage Converters

SCA-YOLO: Real-time Target Detection Based on Deformable Convolution and Context-aware Attention

Related Author

Luo Chaofan

Liu Zhenyu

Lai Zhi-mao

Zhang Yun

Li Dong

Guo Ao

Xu Bo-yan

Cai Rui-chu

Related Institution

School of Information Engineering, Guangdong University of Technology

School of Automation, Guangdong University of Technology

School of Immigration Administration (Guangzhou) , China People’s Police University

College of Science, Shantou University

AI问答

⁰