毕业生知识库

工具使用是自然智能中的一项标志性能力。通过赋予机器人使用工具的能力，研究人员不仅能够拓宽机器人的能力范围，还能够提升机器人的作业能力。本文围绕机器人工具使用中的作业状态感知、工具理解与作业技能以及长序任务规划等方面开展机器人学习方法研究。本文主要研究内容如下：

一、针对以目标表面材料属性为核心的机器人作业状态感知展开研究，提出了一种基于听觉和触觉多模态感知的表面材料分类方法，该方法能够借助相同的超参数设置和网络结构，从多模态（听觉和触觉）数据中有效提取任务相关的材料特征，并利用融合后的多模态特征实现表面材料分类。为了将表面材料分类结果应用于机器人操作任务，提出了一种机器人操作策略，该策略能够根据分类结果与物理引擎推理实现作业状态感知，并生成合理的机器人操作动作。公开数据集和真实场景下的机器人操作实验验证了所提方法的有效性。

二、针对机器人的工具理解与作业技能展开研究，提出了一种基于关键点表征的工具理解与作业技能学习框架。首先，该框架利用了一种能够描述工具特征信息且能够为机器人控制策略提供充分任务信息的通用工具关键点表征来建模工具作业任务。其次，该框架利用了一个基于自监督的工具作业关键点预测网络。该网络能够通过自监督学习从仿真和少量真实人工标注数据中学习不同工具的可供性关键点表征。再次，为了提高方法在训练时的样本利用效率，设计了一种基于自监督信号发生器逆函数的神经网络训练算法。仿真环境和真实场景实验验证了所提方法的有效性。

三、针对机器人复杂长序任务规划展开研究，提出了一种能够结合先验知识和自动规划的知识扩展行为树自主生成方法。该方法能够利用人类和机器人先验知识来辅助行为树的生成，并提高行为树生成过程的鲁棒性。该方法通过嵌套递归的方式来不断地组合原子行为生成不同的行为树来应对生成行为树过程中的冲突情况。由此生成的行为树集合被统称为知识扩展行为树。随后，定义了一种机器人行为知识图谱用于提供人类和机器人知识的结构化表征，使得机器人能够借助其中的推理机制解决生成行为树过程中的冲突情况。在仿真环境和真实工具使用场景中验证了所提方法的有效性。

Tool usage is a hallmark ability in natural intelligence allowing the ability of tool usage for robots can broaden their capabilities and improve their manipulation abilities. This thesis focuses on studying robotic tool usage skill learning and is carried out from the aspects of task state perception, tool understanding and manipulation, and complex long-horizon task planning. The main contents of this thesis are as follows:

Firstly, a multimodal surface material classification method based on auditory and tactile perception is proposed to conduct the robotic task state perception based on the properties of surface materials. The method can effectively learn material representations from multimodal (auditory and tactile) data using a unified network and the same hyperparameter setting, and leverage the multimodal representations to reliably classify surface materials. A robotic manipulation strategy is also proposed to generate appropriate manipulation actions based on the task state perception, where robots can achieve the task state perception through the material classification result and the physics engine. The effectiveness of the proposed method is verified by public datasets and robot manipulation experiments in real scenarios.

Secondly, a tool understanding and manipulation learning framework based on the keypoint representations of tools is proposed to conduct the robotic tool understanding and manipulation. In the framework, a general tool keypoint representation that includes tool information and can be directly executed by robots is designed to model the tool usage. The framework can leverage a prediction model to learn affordance keypoint representations of different tools from simulated and a few manually annotated data using self-supervised learning. A neural network training method is proposed to improve the data-efficient by inferring the inverse function of the self-supervised signal generator. Simulated and real experiments verify the effectiveness of the proposed method.

Thirdly, A planning method that combines prior knowledge and planning to generate knowledge-extended behavior trees is proposed to conduct complex long-horizon tasks. The method can leverage human and robot prior knowledge to assist the generation of behavior trees and improve the robustness of the behavior tree generation. The method can continuously combine atomic behaviors to generate different behavior trees through recursion to deal with conflicts during the behavior tree generation. The resulting set of behavior trees is called knowledge-extended behavior trees. A robot-behavioral knowledge graph is designed to provide a structured representation of human and robot prior knowledge so that the robot can resolve conflicts during the behavior tree generation with the help of its reasoning mechanism. Simulated and real experiments verify the effectiveness of the proposed method.