Date: Fri, 14 Jun 2024 | Total: 27

#1 MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations [PDF9] [Copy] [Kimi5]

Authors: Ruiyuan Lyu ; Tai Wang ; Jingli Lin ; Shuai Yang ; Xiaohan Mao ; Yilun Chen ; Runsen Xu ; Haifeng Huang ; Chenming Zhu ; Dahua Lin ; Jiangmiao Pang

With the emergence of LLMs and their integration with other data modalities, multi-modal 3D perception attracts more attention due to its connectivity to the physical world and makes rapid progress. However, limited by existing datasets, previous works mainly focus on understanding object properties or inter-object spatial relationships in a 3D scene. To tackle this problem, this paper builds the first largest ever multi-modal 3D scene dataset and benchmark with hierarchical grounded language annotations, MMScan. It is constructed based on a top-down logic, from region to object level, from a single target to inter-target relationships, covering holistic aspects of spatial and attribute understanding. The overall pipeline incorporates powerful VLMs via carefully designed prompts to initialize the annotations efficiently and further involve humans' correction in the loop to ensure the annotations are natural, correct, and comprehensive. Built upon existing 3D scanning data, the resulting multi-modal 3D dataset encompasses 1.4M meta-annotated captions on 109k objects and 7.7k regions as well as over 3.04M diverse samples for 3D visual grounding and question-answering benchmarks. We evaluate representative baselines on our benchmarks, analyze their capabilities in different aspects, and showcase the key problems to be addressed in the future. Furthermore, we use this high-quality dataset to train state-of-the-art 3D visual grounding and LLMs and obtain remarkable performance improvement both on existing benchmarks and in-the-wild evaluation. Codes, datasets, and benchmarks will be available at

Subjects: Computer Vision and Pattern Recognition ; Artificial Intelligence ; Robotics

Publish: 2024-06-13 17:59:30 UTC

#2 OpenVLA: An Open-Source Vision-Language-Action Model [PDF4] [Copy] [Kimi9]

Authors: Moo Jin Kim ; Karl Pertsch ; Siddharth Karamcheti ; Ted Xiao ; Ashwin Balakrishna ; Suraj Nair ; Rafael Rafailov ; Ethan Foster ; Grace Lam ; Pannag Sanketi ; Quan Vuong ; Thomas Kollar ; Benjamin Burchfiel ; Russ Tedrake ; Dorsa Sadigh ; Sergey Levine ; Percy Liang ; Chelsea Finn

Large policies pretrained on a combination of Internet-scale vision-language data and diverse robot demonstrations have the potential to change how we teach robots new skills: rather than training new behaviors from scratch, we can fine-tune such vision-language-action (VLA) models to obtain robust, generalizable policies for visuomotor control. Yet, widespread adoption of VLAs for robotics has been challenging as 1) existing VLAs are largely closed and inaccessible to the public, and 2) prior work fails to explore methods for efficiently fine-tuning VLAs for new tasks, a key component for adoption. Addressing these challenges, we introduce OpenVLA, a 7B-parameter open-source VLA trained on a diverse collection of 970k real-world robot demonstrations. OpenVLA builds on a Llama 2 language model combined with a visual encoder that fuses pretrained features from DINOv2 and SigLIP. As a product of the added data diversity and new model components, OpenVLA demonstrates strong results for generalist manipulation, outperforming closed models such as RT-2-X (55B) by 16.5% in absolute task success rate across 29 tasks and multiple robot embodiments, with 7x fewer parameters. We further show that we can effectively fine-tune OpenVLA for new settings, with especially strong generalization results in multi-task environments involving multiple objects and strong language grounding abilities, and outperform expressive from-scratch imitation learning methods such as Diffusion Policy by 20.4%. We also explore compute efficiency; as a separate contribution, we show that OpenVLA can be fine-tuned on consumer GPUs via modern low-rank adaptation methods and served efficiently via quantization without a hit to downstream success rate. Finally, we release model checkpoints, fine-tuning notebooks, and our PyTorch codebase with built-in support for training VLAs at scale on Open X-Embodiment datasets.

Subjects: Robotics ; Machine Learning

Publish: 2024-06-13 15:46:55 UTC

#3 RVT-2: Learning Precise Manipulation from Few Demonstrations [PDF7] [Copy] [Kimi1]

Authors: Ankit Goyal ; Valts Blukis ; Jie Xu ; Yijie Guo ; Yu-Wei Chao ; Dieter Fox

In this work, we study how to build a robotic system that can solve multiple 3D manipulation tasks given language instructions. To be useful in industrial and household domains, such a system should be capable of learning new tasks with few demonstrations and solving them precisely. Prior works, like PerAct and RVT, have studied this problem, however, they often struggle with tasks requiring high precision. We study how to make them more effective, precise, and fast. Using a combination of architectural and system-level improvements, we propose RVT-2, a multitask 3D manipulation model that is 6X faster in training and 2X faster in inference than its predecessor RVT. RVT-2 achieves a new state-of-the-art on RLBench, improving the success rate from 65% to 82%. RVT-2 is also effective in the real world, where it can learn tasks requiring high precision, like picking up and inserting plugs, with just 10 demonstrations. Visual results, code, and trained model are provided at:

Subjects: Robotics ; Artificial Intelligence ; Computer Vision and Pattern Recognition

Publish: 2024-06-12 18:00:01 UTC

#4 OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning [PDF5] [Copy] [Kimi3]

Authors: Tairan He ; Zhengyi Luo ; Xialin He ; Wenli Xiao ; Chong Zhang ; Weinan Zhang ; Kris Kitani ; Changliu Liu ; Guanya Shi

We present OmniH2O (Omni Human-to-Humanoid), a learning-based system for whole-body humanoid teleoperation and autonomy. Using kinematic pose as a universal control interface, OmniH2O enables various ways for a human to control a full-sized humanoid with dexterous hands, including using real-time teleoperation through VR headset, verbal instruction, and RGB camera. OmniH2O also enables full autonomy by learning from teleoperated demonstrations or integrating with frontier models such as GPT-4. OmniH2O demonstrates versatility and dexterity in various real-world whole-body tasks through teleoperation or autonomy, such as playing multiple sports, moving and manipulating objects, and interacting with humans. We develop an RL-based sim-to-real pipeline, which involves large-scale retargeting and augmentation of human motion datasets, learning a real-world deployable policy with sparse sensor input by imitating a privileged teacher policy, and reward designs to enhance robustness and stability. We release the first humanoid whole-body control dataset, OmniH2O-6, containing six everyday tasks, and demonstrate humanoid whole-body skill learning from teleoperated datasets.

Subjects: Robotics ; Computer Vision and Pattern Recognition ; Machine Learning ; Systems and Control ; Systems and Control

Publish: 2024-06-13 06:44:46 UTC

#5 RoTipBot: Robotic Handling of Thin and Flexible Objects using Rotatable Tactile Sensors [PDF4] [Copy] [Kimi2]

Authors: Jiaqi Jiang ; Xuyang Zhang ; Daniel Fernandes Gomes ; Thanh-Toan Do ; Shan Luo

This paper introduces RoTipBot, a novel robotic system for handling thin, flexible objects. Different from previous works that are limited to singulating them using suction cups or soft grippers, RoTipBot can grasp and count multiple layers simultaneously, emulating human handling in various environments. Specifically, we develop a novel vision-based tactile sensor named RoTip that can rotate and sense contact information around its tip. Equipped with two RoTip sensors, RoTipBot feeds multiple layers of thin, flexible objects into the centre between its fingers, enabling effective grasping and counting. RoTip's tactile sensing ensures both fingers maintain good contact with the object, and an adjustment approach is designed to allow the gripper to adapt to changes in the object. Extensive experiments demonstrate the efficacy of the RoTip sensor and the RoTipBot approach. The results show that RoTipBot not only achieves a higher success rate but also grasps and counts multiple layers simultaneously -- capabilities not possible with previous methods. Furthermore, RoTipBot operates up to three times faster than state-of-the-art methods. The success of RoTipBot paves the way for future research in object manipulation using mobilised tactile sensors. All the materials used in this paper are available at \url{}.

Subject: Robotics

Publish: 2024-06-13 17:14:17 UTC

#6 UnO: Unsupervised Occupancy Fields for Perception and Forecasting [PDF4] [Copy] [Kimi1]

Authors: Ben Agro ; Quinlan Sykora ; Sergio Casas ; Thomas Gilles ; Raquel Urtasun

Perceiving the world and forecasting its future state is a critical task for self-driving. Supervised approaches leverage annotated object labels to learn a model of the world -- traditionally with object detections and trajectory predictions, or temporal bird's-eye-view (BEV) occupancy fields. However, these annotations are expensive and typically limited to a set of predefined categories that do not cover everything we might encounter on the road. Instead, we learn to perceive and forecast a continuous 4D (spatio-temporal) occupancy field with self-supervision from LiDAR data. This unsupervised world model can be easily and effectively transferred to downstream tasks. We tackle point cloud forecasting by adding a lightweight learned renderer and achieve state-of-the-art performance in Argoverse 2, nuScenes, and KITTI. To further showcase its transferability, we fine-tune our model for BEV semantic occupancy forecasting and show that it outperforms the fully supervised state-of-the-art, especially when labeled data is scarce. Finally, when compared to prior state-of-the-art on spatio-temporal geometric occupancy prediction, our 4D world model achieves a much higher recall of objects from classes relevant to self-driving.

Subjects: Computer Vision and Pattern Recognition ; Artificial Intelligence ; Machine Learning ; Robotics

Publish: 2024-06-12 23:22:23 UTC

#7 Language-Driven Closed-Loop Grasping with Model-Predictive Trajectory Replanning [PDF4] [Copy] [Kimi1]

Authors: Huy Hoang Nguyen ; Florian Beck ; Minh Nhat Vu ; Gerald Ebmer ; Anh Nguyen ; Andreas Kugi

Combining a vision module inside a closed-loop control system for a \emph{seamless movement} of a robot in a manipulation task is challenging due to the inconsistent update rates between utilized modules. This task is even more difficult in a dynamic environment, e.g., objects are moving. This paper presents a \emph{modular} zero-shot framework for language-driven manipulation of (dynamic) objects through a closed-loop control system with real-time trajectory replanning and an online 6D object pose localization. We segment an object within $\SI{0.5}{\second}$ by leveraging a vision language model via language commands. Then, guided by natural language commands, a closed-loop system, including a unified pose estimation and tracking and online trajectory planning, is utilized to continuously track this object and compute the optimal trajectory in real-time. Our proposed zero-shot framework provides a smooth trajectory that avoids jerky movements and ensures the robot can grasp a non-stationary object. Experiment results exhibit the real-time capability of the proposed zero-shot modular framework for the trajectory optimization module to accurately and efficiently grasp moving objects, i.e., up to \SI{30}{\hertz} update rates for the online 6D pose localization module and \SI{10}{\hertz} update rates for the receding-horizon trajectory optimization. These advantages highlight the modular framework's potential applications in robotics and human-robot interaction; see the video in \href{}{}

Subject: Robotics

Publish: 2024-06-13 12:23:20 UTC

#8 Teleoperation of a robotic manipulator in peri-personal space: a virtual wand approach [PDF3] [Copy] [Kimi1]

Authors: Alexis Poignant ; Guillaume Morel ; Nathanaël Jarrassé

The paper deals with the well-known problem of teleoperating a robotic arm along six degrees of freedom. The prevailing and most effective approach to this problem involves a direct position-to-position mapping, imposing robotic end-effector movements that mirrors those of the user. In the particular case where the robot stands near the operator, there are alternatives to this approach. Drawing inspiration from head pointers utilized in the 1980s, originally designed to enable drawing with limited head motions for tetraplegic individuals, we propose a "virtual wand" mapping. It employs a virtual rigid linkage between the hand and the robot's end-effector. With this approach, rotations produce amplified translations through a lever arm, creating a "rotation-to-position" coupling. This approach expands the translation workspace at the expense of a reduced rotation space. We compare the virtual wand approach to the one-to-one position mapping through the realization of 6-DoF reaching tasks. Results indicate that the two different mappings perform comparably well, are equally well-received by users, and exhibit similar motor control behaviors. Nevertheless, the virtual wand mapping is anticipated to outperform in tasks characterized by large translations and minimal effector rotations, whereas direct mapping is expected to demonstrate advantages in large rotations with minimal translations. These results pave the way for new interactions and interfaces, particularly in disability assistance utilizing head movements (instead of hands). Leveraging body parts with substantial rotations could enable the accomplishment of tasks previously deemed infeasible with standard direct coupling interfaces.

Subject: Robotics

Publish: 2024-06-13 16:42:44 UTC

#9 A Dual Approach to Imitation Learning from Observations with Offline Datasets [PDF2] [Copy] [Kimi1]

Authors: Harshit Sikchi ; Caleb Chuck ; Amy Zhang ; Scott Niekum

Demonstrations are an effective alternative to task specification for learning agents in settings where designing a reward function is difficult. However, demonstrating expert behavior in the action space of the agent becomes unwieldy when robots have complex, unintuitive morphologies. We consider the practical setting where an agent has a dataset of prior interactions with the environment and is provided with observation-only expert demonstrations. Typical learning from observations approaches have required either learning an inverse dynamics model or a discriminator as intermediate steps of training. Errors in these intermediate one-step models compound during downstream policy learning or deployment. We overcome these limitations by directly learning a multi-step utility function that quantifies how each action impacts the agent's divergence from the expert's visitation distribution. Using the principle of duality, we derive DILO(Dual Imitation Learning from Observations), an algorithm that can leverage arbitrary suboptimal data to learn imitating policies without requiring expert actions. DILO reduces the learning from observations problem to that of simply learning an actor and a critic, bearing similar complexity to vanilla offline RL. This allows DILO to gracefully scale to high dimensional observations, and demonstrate improved performance across the board. Project page (code and videos): $\href{}{\text{}}$

Subjects: Machine Learning ; Artificial Intelligence ; Robotics

Publish: 2024-06-13 04:39:42 UTC

#10 BaSeNet: A Learning-based Mobile Manipulator Base Pose Sequence Planning for Pickup Tasks [PDF2] [Copy] [Kimi1]

Authors: Lakshadeep Naik ; Sinan Kalkan ; Sune L. Sørensen ; Mikkel B. Kjærgaard ; Norbert Krüger

In many applications, a mobile manipulator robot is required to grasp a set of objects distributed in space. This may not be feasible from a single base pose and the robot must plan the sequence of base poses for grasping all objects, minimizing the total navigation and grasping time. This is a Combinatorial Optimization problem that can be solved using exact methods, which provide optimal solutions but are computationally expensive, or approximate methods, which offer computationally efficient but sub-optimal solutions. Recent studies have shown that learning-based methods can solve Combinatorial Optimization problems, providing near-optimal and computationally efficient solutions. In this work, we present BASENET - a learning-based approach to plan the sequence of base poses for the robot to grasp all the objects in the scene. We propose a Reinforcement Learning based solution that learns the base poses for grasping individual objects and the sequence in which the objects should be grasped to minimize the total navigation and grasping costs using Layered Learning. As the problem has a varying number of states and actions, we represent states and actions as a graph and use Graph Neural Networks for learning. We show that the proposed method can produce comparable solutions to exact and approximate methods with significantly less computation time.

Subject: Robotics

Publish: 2024-06-12 21:31:32 UTC

#11 Deep Reinforcement Learning-based Quadcopter Controller: A Practical Approach and Experiments [PDF2] [Copy] [Kimi1]

Authors: Truong-Dong Do ; Nguyen Xuan Mung ; Sung Kyung Hong

Quadcopters have been studied for decades thanks to their maneuverability and capability of operating in a variety of circumstances. However, quadcopters suffer from dynamical nonlinearity, actuator saturation, as well as sensor noise that make it challenging and time consuming to obtain accurate dynamic models and achieve satisfactory control performance. Fortunately, deep reinforcement learning came and has shown significant potential in system modelling and control of autonomous multirotor aerial vehicles, with recent advancements in deployment, performance enhancement, and generalization. In this paper, an end-to-end deep reinforcement learning-based controller for quadcopters is proposed that is secure for real-world implementation, data-efficient, and free of human gain adjustments. First, a novel actor-critic-based architecture is designed to map the robot states directly to the motor outputs. Then, a quadcopter dynamics-based simulator was devised to facilitate the training of the controller policy. Finally, the trained policy is deployed on a real Crazyflie nano quadrotor platform, without any additional fine-tuning process. Experimental results show that the quadcopter exhibits satisfactory performance as it tracks a given complicated trajectory, which demonstrates the effectiveness and feasibility of the proposed method and signifies its capability in filling the simulation-to-reality gap.

Subjects: Robotics ; Systems and Control ; Systems and Control

Publish: 2024-06-13 05:17:20 UTC

#12 EHAZOP: A Proof of Concept Ethical Hazard Analysis of an Assistive Robot [PDF2] [Copy] [Kimi1]

Authors: Catherine Menon ; Austen Rainer ; Patrick Holthaus ; Gabriella Lakatos ; Silvio Carta

The use of assistive robots in domestic environments can raise significant ethical concerns, from the risk of individual ethical harm to wider societal ethical impacts including culture flattening and compromise of human dignity. It is therefore essential to ensure that technological development of these robots is informed by robust and inclusive techniques for mitigating ethical concerns. This paper presents EHAZOP, a method for conducting an ethical hazard analysis on an assistive robot. EHAZOP draws upon collaborative, creative and structured processes originating within safety engineering, using these to identify ethical concerns associated with the operation of a given assistive robot. We present the results of a proof of concept study of EHAZOP, demonstrating the potential for this process to identify diverse ethical hazards in these systems.

Subject: Robotics

Publish: 2024-06-13 15:41:38 UTC

#13 Hands-free teleoperation of a nearby manipulator through a virtual body-to-robot link [PDF2] [Copy] [Kimi1]

Authors: Alexis Poignant ; Nathanaël Jarrassé ; Guillaume Morel

This paper introduces an innovative control approach for teleoperating a robot in close proximity to a human operator, which could be useful to control robots embedded on wheelchairs. The method entails establishing a virtual connection between a specific body part and the robot's end-effector, visually displayed through an Augmented Reality (AR) headset. This linkage enables the transformation of body rotations into amplified effector translations, extending the robot's workspace beyond the capabilities of direct one-to-one mapping. Moreover, the linkage can be reconfigured using a joystick, resulting in a hybrid position/velocity control mode using the body/joystick motions respectively. After providing a comprehensive overview of the control methodology, we present the results of an experimental campaign designed to elucidate the advantages and drawbacks of our approach compared to the conventional joystick-based teleoperation method. The body-link control demonstrates slightly faster task completion and is naturally preferred over joystick velocity control, albeit being more physically demanding for tasks with a large range. The hybrid mode, where participants could simultaneously utilize both modes, emerges as a compromise, combining the intuitiveness of the body mode with the extensive task range of the velocity mode. Finally, we provide preliminary observations on potential assistive applications using head motions, especially for operators with limited range of motion in their bodies.

Subject: Robotics

Publish: 2024-06-13 16:37:51 UTC

#14 Deep Transformer Network for Monocular Pose Estimation of Ship-Based UAV [PDF2] [Copy] [Kimi]

Authors: Maneesha Wickramasuriya ; Taeyoung Lee ; Murray Snyder

This paper introduces a deep transformer network for estimating the relative 6D pose of a Unmanned Aerial Vehicle (UAV) with respect to a ship using monocular images. A synthetic dataset of ship images is created and annotated with 2D keypoints of multiple ship parts. A Transformer Neural Network model is trained to detect these keypoints and estimate the 6D pose of each part. The estimates are integrated using Bayesian fusion. The model is tested on synthetic data and in-situ flight experiments, demonstrating robustness and accuracy in various lighting conditions. The position estimation error is approximately 0.8\% and 1.0\% of the distance to the ship for the synthetic data and the flight experiments, respectively. The method has potential applications for ship-based autonomous UAV landing and navigation.

Subjects: Computer Vision and Pattern Recognition ; Artificial Intelligence ; Robotics ; Image and Video Processing

Publish: 2024-06-13 16:01:22 UTC

#15 Trajectory Planning for Autonomous Driving in Unstructured Scenarios Based on Graph Neural Network and Numerical Optimization [PDF1] [Copy] [Kimi1]

Authors: Sumin Zhang ; Kuo Li ; Rui He ; Zhiwei Meng ; Yupeng Chang ; Xiaosong Jin ; Ri Bai

In unstructured environments, obstacles are diverse and lack lane markings, making trajectory planning for intelligent vehicles a challenging task. Traditional trajectory planning methods typically involve multiple stages, including path planning, speed planning, and trajectory optimization. These methods require the manual design of numerous parameters for each stage, resulting in significant workload and computational burden. While end-to-end trajectory planning methods are simple and efficient, they often fail to ensure that the trajectory meets vehicle dynamics and obstacle avoidance constraints in unstructured scenarios. Therefore, this paper proposes a novel trajectory planning method based on Graph Neural Networks (GNN) and numerical optimization. The proposed method consists of two stages: (1) initial trajectory prediction using the GNN, (2) trajectory optimization using numerical optimization. First, the graph neural network processes the environment information and predicts a rough trajectory, replacing traditional path and speed planning. This predicted trajectory serves as the initial solution for the numerical optimization stage, which optimizes the trajectory to ensure compliance with vehicle dynamics and obstacle avoidance constraints. We conducted simulation experiments to validate the feasibility of the proposed algorithm and compared it with other mainstream planning algorithms. The results demonstrate that the proposed method simplifies the trajectory planning process and significantly improves planning efficiency.

Subject: Robotics

Publish: 2024-06-13 06:39:49 UTC

#16 Human-Robot Interface for Teleoperated Robotized Planetary Sample Collection and Assembly [PDF1] [Copy] [Kimi1]

Authors: Lorenzo Pagliara ; Vincenzo Petrone ; Enrico Ferrentino ; Pasquale Chiacchio

As human space exploration evolves toward longer voyages farther from our home planet, in-situ resource utilization (ISRU) becomes increasingly important. Haptic teleoperations are one of the technologies by which such activities can be carried out remotely by humans, whose expertise is still necessary for complex activities. In order to perform precision tasks with effectiveness, the operator must experience ease of use and accuracy. The same features are demanded to reduce the complexity of the training procedures and the associated learning time for operators without a specific background in robotic teleoperations. Haptic teleoperation systems, that allow for a natural feeling of forces, need to cope with the trade-off between accurate movements and workspace extension. Clearly, both of them are required for typical ISRU tasks. In this work, we develop a new concept of operations and suitable human-robot interfaces to achieve sample collection and assembly with ease of use and accuracy. In the proposed operational concept, the teleoperation space is extended by executing automated trajectories, offline planned at the control station. In three different experimental scenarios, we validate the end-to-end system involving the control station and the robotic asset, by assessing the contribution of haptics to mission success, the system robustness to consistent delays, and the ease of training new operators.

Subjects: Robotics ; Human-Computer Interaction ; Systems and Control ; Systems and Control

Publish: 2024-06-13 09:17:10 UTC

#17 Direct Imitation Learning-based Visual Servoing using the Large Projection Formulation [PDF2] [Copy] [Kimi]

Authors: Sayantan Auddy ; Antonio Paolillo ; Justus Piater ; Matteo Saveriano

Today robots must be safe, versatile, and user-friendly to operate in unstructured and human-populated environments. Dynamical system-based imitation learning enables robots to perform complex tasks stably and without explicit programming, greatly simplifying their real-world deployment. To exploit the full potential of these systems it is crucial to implement closed loops that use visual feedback. Vision permits to cope with environmental changes, but is complex to handle due to the high dimension of the image space. This study introduces a dynamical system-based imitation learning for direct visual servoing. It leverages off-the-shelf deep learning-based perception backbones to extract robust features from the raw input image, and an imitation learning strategy to execute sophisticated robot motions. The learning blocks are integrated using the large projection task priority formulation. As demonstrated through extensive experimental analysis, the proposed method realizes complex tasks with a robotic manipulator.

Subject: Robotics

Publish: 2024-06-13 13:51:45 UTC

#18 Beyond the Frontier: Predicting Unseen Walls from Occupancy Grids by Learning from Floor Plans [PDF2] [Copy] [Kimi]

Authors: Ludvig Ericson ; Patric Jensfelt

In this paper, we tackle the challenge of predicting the unseen walls of a partially observed environment as a set of 2D line segments, conditioned on occupancy grids integrated along the trajectory of a 360{\deg} LIDAR sensor. A dataset of such occupancy grids and their corresponding target wall segments is collected by navigating a virtual robot between a set of randomly sampled waypoints in a collection of office-scale floor plans from a university campus. The line segment prediction task is formulated as an autoregressive sequence prediction task, and an attention-based deep network is trained on the dataset. The sequence-based autoregressive formulation is evaluated through predicted information gain, as in frontier-based autonomous exploration, demonstrating significant improvements over both non-predictive estimation and convolution-based image prediction found in the literature. Ablations on key components are evaluated, as well as sensor range and the occupancy grid's metric area. Finally, model generality is validated by predicting walls in a novel floor plan reconstructed on-the-fly in a real-world office environment.

Subjects: Robotics ; Computer Vision and Pattern Recognition

Publish: 2024-06-13 14:22:59 UTC

#19 AutomaChef: A Physics-informed Demonstration-guided Learning Framework for Granular Material Manipulation [PDF1] [Copy] [Kimi1]

Authors: Minglun Wei ; Xintong Yang ; Yu-Kun Lai ; Seyed Amir Tafrishi ; Ze Ji

Due to the complex physical properties of granular materials, research on robot learning for manipulating such materials predominantly either disregards the consideration of their physical characteristics or uses surrogate models to approximate their physical properties. Learning to manipulate granular materials based on physical information obtained through precise modelling remains an unsolved problem. In this paper, we propose to address this challenge by constructing a differentiable physics simulator for granular materials based on the Taichi programming language and developing a learning framework accelerated by imperfect demonstrations that are generated via gradient-based optimisation on non-granular materials through our simulator. Experimental results show that our method trains three policies that, when chained, are capable of executing the task of transporting granular materials in both simulated and real-world scenarios, which existing popular deep reinforcement learning models fail to accomplish.

Subject: Robotics

Publish: 2024-06-13 14:40:43 UTC

#20 LLM-Craft: Robotic Crafting of Elasto-Plastic Objects with Large Language Models [PDF1] [Copy] [Kimi]

Authors: Alison Bartsch ; Amir Barati Farimani

When humans create sculptures, we are able to reason about how geometrically we need to alter the clay state to reach our target goal. We are not computing point-wise similarity metrics, or reasoning about low-level positioning of our tools, but instead determining the higher-level changes that need to be made. In this work, we propose LLM-Craft, a novel pipeline that leverages large language models (LLMs) to iteratively reason about and generate deformation-based crafting action sequences. We simplify and couple the state and action representations to further encourage shape-based reasoning. To the best of our knowledge, LLM-Craft is the first system successfully leveraging LLMs for complex deformable object interactions. Through our experiments, we demonstrate that with the LLM-Craft framework, LLMs are able to successfully reason about the deformation behavior of elasto-plastic objects. Furthermore, we find that LLM-Craft is able to successfully create a set of simple letter shapes. Finally, we explore extending the framework to reaching more ambiguous semantic goals, such as "thinner" or "bumpy". For videos please see our website:

Subject: Robotics

Publish: 2024-06-12 21:17:02 UTC

#21 UruBots Autonomous Cars Team One Description Paper for FIRA 2024 [PDF1] [Copy] [Kimi]

Authors: Pablo Moraes ; Christopher Peters ; Any Da Rosa ; Vinicio Melgar ; Franco Nuñez ; Maximo Retamar ; William Moraes ; Victoria Saravia ; Hiago Sodre ; Sebastian Barcelona ; Anthony Scirgalea ; Juan Deniz ; Bruna Guterres ; André Kelbouscas ; Ricardo Grando

This document presents the design of an autonomous car developed by the UruBots team for the 2024 FIRA Autonomous Cars Race Challenge. The project involves creating an RC-car sized electric vehicle capable of navigating race tracks with in an autonomous manner. It integrates mechanical and electronic systems alongside artificial intelligence based algorithms for the navigation and real-time decision-making. The core of our project include the utilization of an AI-based algorithm to learn information from a camera and act in the robot to perform the navigation. We show that by creating a dataset with more than five thousand samples and a five-layered CNN we managed to achieve promissing performance we our proposed hardware setup. Overall, this paper aims to demonstrate the autonomous capabilities of our car, highlighting its readiness for the 2024 FIRA challenge, helping to contribute to the field of autonomous vehicle research.

Subject: Robotics

Publish: 2024-06-13 02:07:06 UTC

#22 LLM-Driven Robots Risk Enacting Discrimination, Violence, and Unlawful Actions [PDF1] [Copy] [Kimi]

Authors: Rumaisa Azeem ; Andrew Hundt ; Masoumeh Mansouri ; Martim Brandão

Members of the Human-Robot Interaction (HRI) and Artificial Intelligence (AI) communities have proposed Large Language Models (LLMs) as a promising resource for robotics tasks such as natural language interactions, doing household and workplace tasks, approximating `common sense reasoning', and modeling humans. However, recent research has raised concerns about the potential for LLMs to produce discriminatory outcomes and unsafe behaviors in real-world robot experiments and applications. To address these concerns, we conduct an HRI-based evaluation of discrimination and safety criteria on several highly-rated LLMs. Our evaluation reveals that LLMs currently lack robustness when encountering people across a diverse range of protected identity characteristics (e.g., race, gender, disability status, nationality, religion, and their intersections), producing biased outputs consistent with directly discriminatory outcomes -- e.g. `gypsy' and `mute' people are labeled untrustworthy, but not `european' or `able-bodied' people. Furthermore, we test models in settings with unconstrained natural language (open vocabulary) inputs, and find they fail to act safely, generating responses that accept dangerous, violent, or unlawful instructions -- such as incident-causing misstatements, taking people's mobility aids, and sexual predation. Our results underscore the urgent need for systematic, routine, and comprehensive risk assessments and assurances to improve outcomes and ensure LLMs only operate on robots when it is safe, effective, and just to do so. Data and code will be made available.

Subjects: Robotics ; Artificial Intelligence ; Computation and Language ; Computers and Society

Publish: 2024-06-13 05:31:49 UTC

#23 Adaptive Actor-Critic Based Optimal Regulation for Drift-Free Uncertain Nonlinear Systems [PDF] [Copy] [Kimi]

Authors: Ashwin P. Dani ; Shubhendu Bhasin

In this paper, a continuous-time adaptive actor-critic reinforcement learning (RL) controller is developed for drift-free nonlinear systems. Practical examples of such systems are image-based visual servoing (IBVS) and wheeled mobile robots (WMR), where the system dynamics includes a parametric uncertainty in the control effectiveness matrix with no drift term. The uncertainty in the input term poses a challenge for developing a continuous-time RL controller using existing methods. In this paper, an actor-critic or synchronous policy iteration (PI)-based RL controller is presented with a concurrent learning (CL)-based parameter update law for estimating the unknown parameters of the control effectiveness matrix. An infinite-horizon value function minimization objective is achieved by regulating the current states to the desired with near-optimal control efforts. The proposed controller guarantees closed-loop stability and simulation results validate the proposed theory using IBVS and WMR examples.

Subjects: Systems and Control ; Robotics ; Systems and Control

Publish: 2024-06-13 13:27:27 UTC

#24 A Game Between Two Identical Dubins Cars: Evading a Conic Sensor in Minimum Time [PDF] [Copy] [Kimi]

Author: Ubaldo Ruiz

A fundamental task in mobile robotics is keeping an intelligent agent under surveillance with an autonomous robot as it travels in the environment. This work studies a version of that problem involving one of the most popular vehicle platforms in robotics. In particular, we consider two identical Dubins cars moving on a plane without obstacles. One of them plays as the pursuer, and it is equipped with a limited field-of-view detection region modeled as a semi-infinite cone with its apex at the pursuer's position. The pursuer aims to maintain the other Dubins car, which plays as the evader, as much time as possible inside its detection region. On the contrary, the evader wants to escape as soon as possible. In this work, employing differential game theory, we find the time-optimal motion strategies near the game's end. The analysis of those trajectories reveals the existence of at least two singular surfaces: a Transition Surface and an Evader's Universal Surface. We also found that the barrier's standard construction produces a surface that partially lies outside the playing space and fails to define a closed region, implying that an additional procedure is required to determine all configurations where the evader escapes.

Subjects: Robotics ; Optimization and Control

Publish: 2024-06-12 20:50:26 UTC

#25 Adaptive Nonlinear Model Predictive Control for a Real-World Labyrinth Game [PDF] [Copy] [Kimi]

Authors: Johannes Gaber ; Thomas Bi ; Raffaello D'Andrea

We present a nonlinear non-convex model predictive control approach to solving a real-world labyrinth game. We introduce adaptive nonlinear constraints, representing the non-convex obstacles within the labyrinth. Our method splits the computation-heavy optimization problem into two layers; first, a high-level model predictive controller which incorporates the full problem formulation and finds pseudo-global optimal trajectories at a low frequency. Secondly, a low-level model predictive controller that receives a reduced, computationally optimized version of the optimization problem to follow the given high-level path in real-time. Further, a map of the labyrinth surface irregularities is learned. Our controller is able to handle the major disturbances and model inaccuracies encountered on the labyrinth and outperforms other classical control methods.

Subjects: Robotics ; Systems and Control ; Systems and Control

Publish: 2024-06-12 21:20:00 UTC