Seamless transitions between autonomous robot capabilities and human intervention in construction robotics

Congratulations to the winners of the best paper award of the International Association for Automation and Robotics in Construction 2021. The team around Cynthia Brosque, Elena Galbally, Prof. Martin Fischer, and Prof. Oussama Khatib did excellent groundwork for construction robotics. With permission, Silicon Valley Robotics is reposting the first parts of the paper below. It is also available on the IAARC website.

GOLDBECK US Inc. helped define construction robotics use-cases that generate real-world value on their job sites. In the first phase of the project, they collected data to understand the tasks on site holistically. Subsequently, simulations on how robots perform the job were created. Using this information, the researchers of Stanford University | CIFE – Center for Integrated Facility Engineering then developed suitable robotic prototypes. GOLDBECK supplied physical building components to test the prototypes under realistic conditions.

GOLDBECK is now hunting for robotics companies to help us transition this research into a real-world robotic application!

Abstract

Due to their unstructured and dynamic nature, construction sites present many challenges for robotic automation of tasks. Integrating human-robot collaboration (HRC) is critical for task success and implementation feasibility. This is particularly important for contact-rich tasks and other complex scenarios which require a level of reasoning that cannot be accomplished by a fully autonomous robot. Currently, many solutions rely on precise teleoperation that requires one operator per robot. Alternatively, one operator may oversee several semi-autonomous robots. However, the operators do not have the sensory feedback needed to adequately leverage their expertise and craftsmanship. Haptic interfaces allow for intuitive human-robot collaboration by providing rich contact feedback. This paper presents two human-robot collaboration solutions for welding and joint sealing through the use of a haptic device. Our approach allows for seamless transitions between autonomous robot capabilities and human intervention with rich contact feedback. Additionally, this work opens the door to intuitive programming of new tasks through haptic human demonstration.

1 Introduction

In recent years, progress in mobility, manipulation skills, and AI reasoning have started to enable the use of robots in space, underwater, homes, agriculture, and construction [1]. A particularly important area of interest is the automation of dangerous, strenuous, and laborintensive tasks [2].

Construction sites are especially challenging environments for autonomous robots because of their highly unpredictable and unstructured nature [3, 4]. Hence, fully autonomous robots that replace human labor are not the most feasible or ideal solution. The majority of current approaches rely on a human operator that oversees a single task autonomous robot. The operator receives only visual feedback and is limited in the type of input and recovery from failure he can provide due to the lack of an intuitive interface to do so. [4] attributed this lack in technical flexibility of construction robots to the fact that early construction solutions imitated systems initially developed for industrial fabrication [5].

Some tasks are structured enough to be autonomously performed by a robot with little human input, but many require a more flexible approach that incorporates a higher degree of human reasoning and intuition [6]. Given this reality, a method to design construction robots should be flexible enough to allow varying levels of human-robot collaboration depending on the task.

Haptic devices (Fig.1) provide an effective interface for collaboration by allowing the human to (1) feel the contact forces between the robot’s end-effector and the environment [7], and (2) easily intervene and control the robot motion in scenarios that the autonomous behaviors are not able to handle successfully [8]. Additionally, data from these haptic interventions can be collected and used to learn new autonomous skills. Remote robot control using a haptic interface has been tested in fields such as surgery [9] and underwater exploration [1], but has not yet been widely implemented in construction.

In previous work, the authors explored human-robot collaboration solutions to five hazardous and repetitive construction tasks: installing drywall, painting, bolting, welding, and sealing precast concrete slab joints [10]. Our industry partner, Goldbeck, was interested in automating these assembly and finishing tasks that require on-site, repetitive manual effort, ergonomically challenging positions, and working from dangerous heights. [10] outlines a method for designing collaborative robotic solutions with haptic feedback and to assess their feasibility in simulation.

In this paper, we focus on two of the previously explored tasks (steel welding and sealing precast concrete joints) and apply the aforementioned method to design more flexible collaborative solutions. Different from [10], we propose relying primarily on the robot’s functional autonomy and using haptics as an effective and intuitive way to intervene in unexplored or failure scenarios. Force data from the recovery strategy employed by the operator can be recorded and used to learn from demonstration and augment the robot’s autonomy. Over time, the robot will require less human intervention. This higher degree of autonomy could allow a single operator to supervise many robots at once, overcoming the problems of teleoperation in which one operator per robot is needed.

2 Related Work

While factories have typically separated workers from robots due to safety concerns, human-robot collaboration cannot be overlooked in construction, as robots and humans share one workspace [2]. This requires devising solutions that allow us to effectively combine the workers’ expertise with the robots’ autonomous skills.

Construction literature has studied the use of teleoperation devices [11, 12, 7], particularly focusing on construction machinery, such as excavators. These solutions often involve cameras for visual feedback and GPS sensors for navigation, which can be sufficient to accomplish low dexterity tasks with increased operator safety. However, [7] states that complex tasks involving contact greatly benefit from additional sensory feedback such as tactile information. Furthermore, teleoperation solutions rely heavily on the operator’s guidance and do not fully exploit the autonomous capabilities of the robot.

A different set of collaborative solutions currently used onsite have semi-autonomous robots with a human supervisor that oversees the tasks such as drywall installation, concrete drilling, and layout [13]. The supervisor can provide simple inputs to the robot, such as when to start or stop the operation, while the robot handles the rest of the task. This approach makes better use of modern robotics capabilities and allows a single operator to manage several robots. However, the interfaces used to provide inputs to the robot are often too simplistic to allow recovery from failure.

In the event of a robot failure during task execution, joysticks and control pendants do not always provide enough feedback for the operator to intervene in an effective way that enables timely task completion. Additionally, there is currently no streamlined way to learn from the operator’s intervention and use this data to improve the robot’s autonomous capabilities.

By allowing the operator to feel the contact between the robot and its environment, haptic devices increase the range of scenarios in which the operator can aid in failure recovery [14]. Additionally, we can easily record both force and position data during the operator’s intervention. These human demonstrations of recovery strategies can allow the robot to learn new skills [15] and augment its functional autonomy.

Haptic devices have been used by the construction industry in combination with virtual reality for task training purposes [16]. The technology has allowed workers to train in a safe environment with realistic task conditions. However, haptics are still a novel technology in construction applications and field use has not been reported.

Current algorithms for haptic control of robots [17] can handle large communication delays, making them effective interfaces for remote intervention at long distances. In [1] an operator haptically controls an underwater ocean exploration robot from a distance of 100m.

Finally, [18] provides an example that integrates two modalities of robot control: autonomous robot behavior and expert human-guided motion interactions. In this study, a group of mobile robot arms successfully installed drywall boards in simulation with flexible human intervention.

This body of prior work illustrates how keeping the human in the loop with adequate feedback can facilitate successful task automation in complex, unstructured environments. Moreover, it highlights the value of haptics as a way to provide a flexible and effective interface for human-robot collaboration as well as teaching robots new autonomous skills.

Full Text of Paper

 

tags: c-Research-Innovation

---

---