Multimodal Reinforcement Learning for Robots Collaborating with Humans

By Afagh Mehri Shervedani et al

Read the original document by opening this link in a new tab.

Table of Contents

I. INTRODUCTION
II. RELATED WORK
III. USER SIMULATOR
A. Feature Extraction
B. Data Annotation
C. Data Augmentation
D. Model Architecture and Training

Summary

Robot assistants for older adults and people with disabilities need to interact with their users in collaborative tasks. The core component of these systems is an interaction manager whose job is to observe and assess the task, and infer the state of the human and their intent to choose the best course of action for the robot. In this paper, a reinforcement learning (RL) approach is proposed to learn the robot policy for multimodal systems. The agent is trained with a simulator developed using human data and can deal with multiple modalities such as language and physical actions. The effectiveness of this RL-based interaction manager is demonstrated through training the agent for the Find task. The paper also discusses the challenges in training the agent and the development of an interpretable RL-based interaction manager. The proposed neural network-based user simulator provides a training environment in RL training and shows promising results in assisting personalized users. The paper concludes with a user study to evaluate the performance of the learned interaction manager.