Why You Cannot Rank First: Modifications for Benchmarking Six-Degree-of-Freedom Visual Localization Algorithms

doi:10.3390/s23239580

	Why You Cannot Rank First: Modifications for Benchmarking Six-Degree-of-Freedom Visual Localization Algorithms
	Han, Sheng 1,2; Gao, Wei1,2 ; Hu, Zhanyi1,2
发表期刊	SENSORS
	2023-12-01
卷号	23 期号:23 页码:18
通讯作者	Gao, Wei(wgao@nlpr.ia.ac.cn)
摘要	Robust and precise visual localization over extended periods of time poses a formidable challenge in the current domain of spatial vision. The primary difficulty lies in effectively addressing significant variations in appearance caused by seasonal changes (summer, winter, spring, autumn) and diverse lighting conditions (dawn, day, sunset, night). With the rapid development of related technologies, more and more relevant datasets have emerged, which has also promoted the progress of 6-DOF visual localization in both directions of autonomous vehicles and handheld devices.This manuscript endeavors to rectify the existing limitations of the current public benchmark for long-term visual localization, especially in the part on the autonomous vehicle challenge. Taking into account that autonomous vehicle datasets are primarily captured by multi-camera rigs with fixed extrinsic camera calibration and consist of serialized image sequences, we present several proposed modifications designed to enhance the rationality and comprehensiveness of the evaluation algorithm. We advocate for standardized preprocessing procedures to minimize the possibility of human intervention influencing evaluation results. These procedures involve aligning the positions of multiple cameras on the vehicle with a predetermined canonical reference system, replacing the individual camera positions with uniform vehicle poses, and incorporating sequence information to compensate for any failed localized poses. These steps are crucial in ensuring a just and accurate evaluation of algorithmic performance. Lastly, we introduce a novel indicator to resolve potential ties in the Schulze ranking among submitted methods. The inadequacies highlighted in this study are substantiated through simulations and actual experiments, which unequivocally demonstrate the necessity and effectiveness of our proposed amendments.
关键词	visual localization benchmark enhancement pose compensation sequential interpolation ties resolution
DOI	10.3390/s23239580
关键词[WOS]	PLACE RECOGNITION
收录类别	SCI
语种	英语
资助项目	National Key R&D Program of China
项目资助者	National Key R&D Program of China
WOS研究方向	Chemistry ; Engineering ; Instruments & Instrumentation
WOS类目	Chemistry, Analytical ; Engineering, Electrical & Electronic ; Instruments & Instrumentation
WOS记录号	WOS:001116610500001
出版者	MDPI
引用统计
文献类型	期刊论文
条目标识符	http://ir.ia.ac.cn/handle/173211/55172
专题	多模态人工智能系统全国重点实验室
通讯作者	Gao, Wei
作者单位	1.Univ Chinese Acad Sci, Sch Artificial Intelligence, Beijing 100049, Peoples R China 2.Chinese Acad Sci, Inst Automat, Beijing 100190, Peoples R China
第一作者单位	中国科学院自动化研究所
通讯作者单位	中国科学院自动化研究所
推荐引用方式 GB/T 7714	Han, Sheng,Gao, Wei,Hu, Zhanyi. Why You Cannot Rank First: Modifications for Benchmarking Six-Degree-of-Freedom Visual Localization Algorithms[J]. SENSORS,2023,23(23):18.
APA	Han, Sheng,Gao, Wei,&Hu, Zhanyi.(2023).Why You Cannot Rank First: Modifications for Benchmarking Six-Degree-of-Freedom Visual Localization Algorithms.SENSORS,23(23),18.
MLA	Han, Sheng,et al."Why You Cannot Rank First: Modifications for Benchmarking Six-Degree-of-Freedom Visual Localization Algorithms".SENSORS 23.23(2023):18.