LLM/Human Augmented Robot Learning and Planning

Invited Talk

Robot Planning with Foundation Models: From Service to Assistive Tasks

Dr. Shiqi Zhang

Associate Professor, School of Computing, SUNY Binghamton

Abstract

Robots rely on task planning to sequence high-level actions and on motion planning to generate continuous trajectories that realize those actions. Task and motion planning (TAMP) integrates these two levels of reasoning to ensure both goal completion and motion feasibility. However, deploying TAMP in the real world remains challenging: environments are open, dynamic, and full of unforeseen objects and situations. In this talk, I will present our recent work on leveraging foundation models to advance human-robot TAMP systems. I will highlight applications ranging from service robots that set tables and deliver objects, to quadruped robots that assist people with visual impairments in navigation.

Bio

Dr. Shiqi Zhang is an Associate Professor with the School of Computing, the State University of New York at Binghamton (SUNY Binghamton), where he is the founder and director of the Autonomous Intelligent Robotics Group. His research interests include robot decision making, robot learning and human-robot systems. He was a Postdoc under Peter Stone at the University of Texas at Austin, and received his Ph.D. under Mohan Sridharan in Computer Science from Texas Tech University. He was the PI of a National Science Foundation (NSF) National Robotics Initiative project on robot decision making and is the PI of another NSF project on assistive robotics. He received the Best Robotics Paper Award from the 2018 AAMAS conference, a Ford URP Award in 2019, an OPPO Faculty Research Award in 2020, and an Outstanding Associate Editor recognition in 2024 from the IEEE Robotics and Automation Letters journal.

Invited Talk

Augmented Reinforcement Learning-based Safe and Trustworthy Control

Dr. Dachuan Li

Southern University of Science and Technology

Abstract

The Reinforcement Learning (RL) has emerged in recent years as a novel and promising paradigm for the control of robotic systems. The robustness and safety issue in complex real-world environments pose significant challenge to the development of RL-based robotic systems. The provable safety assurance of RL-based controller is missing, and how to utilize the intervention experience to effectively augment the RL controller is an unaddressed issue. In addition, the combination of model- and learning-based control techniques pose new challenges to the design of robotic control systems. This talk presents the design of safe and trustworthy control systems based on augmented RL techniques, with topics including, reinforcement learning-enabled control with runtime safety assurance, intervention experience-guided adaptable RL control, MPC-augmented RL control, as well as case studies in actual autonomous vehicles.

Invited Talk

Human Attention as a Grounding Signal for Social Robot Behavior

Dr. Yu Fang

Honda Research Institute Japan Co., Ltd.

Abstract

Large Language Models (LLMs) have significantly expanded the capabilities of robotic systems in language understanding, reasoning, and task planning. However, social interaction remains a major challenge for embodied robots. While LLMs can generate plausible actions and dialogue, they provide limited guidance on how robots should allocate, express, and regulate attention during real-world interactions with humans.

In this talk, I will discuss how insights from human attention can help address this challenge. Drawing on a series of human-robot interaction studies, including eye-head coordination, robot gaze behavior, attention visualization through Mirror-Eye interfaces, and multi-party interactions with the Haru social robot, I will present empirical evidence showing that attention plays a dual role in social interaction: it serves both as a mechanism for acquiring information and as a signal that communicates intention, engagement, and understanding.

These findings suggest that human attentional behavior can provide valuable grounding signals for future robot learning and planning systems. Rather than relying solely on language-based reasoning, socially intelligent robots must also learn how attention is perceived and interpreted by human partners. I will conclude by discussing how attention-based interaction models may contribute to the development of more interpretable, trustworthy, and socially aware robotic systems in educational and collaborative settings.

Invited Talk

Causal Learning for Driving Safety

Dr. Jianwu Fang

Xi'an Jiaotong University

Abstract

Driving safety is a long-standing research topic in autonomous driving. It seeks solutions to understand risk, scene affordances, and safe planning space. In this talk, we will first introduce the concept of causal learning in driving scenes, and then present a framework that involves causal learning to understand near-crash driving scenes, generate risky videos, and perform counterfactual trajectory planning.

Invited Talk

Knowledge-Guided Multi-Agent Reinforcement Learning for Multi-AAV Cooperative Capture

Dr. Chao Yan

Nanjing University of Aeronautics and Astronautics

Abstract

Multiple autonomous aerial vehicles (multi-AAV) have become increasingly vital for airspace security and cooperative capture missions. However, learning effective multi-agent policies in obstacle-cluttered environments remains a critical challenge. This presentation explores advanced multi-agent reinforcement learning (MARL) paradigms, highlighting how the integration of domain knowledge and structured guidance can fundamentally overcome these bottlenecks. The core of the talk will introduce a knowledge-guided MARL approach that leverages a cooperative force field and adaptive dense-to-sparse reward transitions to guide early exploration and achieve team-optimal capture strategies. Furthermore, we will discuss related architectural enhancements that support complex capture missions, including hierarchical skill orchestration for complex obstacle environments, and hypernetwork-based oracle guidance that unifies the search and capture phases into a cohesive decentralized policy. Validated through extensive numerical simulations and real-world multi-AAV flight tests, these knowledge-augmented methodologies demonstrate superior sample efficiency, robust zero-shot generalization, and reliable sim-to-real transfer.

Bio

Dr. Chao Yan is an Associate Professor at the College of Automation Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing, China. He received his Ph.D. and M.S. degrees in Control Science and Engineering from the National University of Defense Technology in 2023 and 2019, respectively. From 2021 to 2022, he was a visiting Ph.D. student at Nanyang Technological University, Singapore. His research interests include the cooperative control and intelligent decision-making of UAV swarms, as well as multi-agent reinforcement learning. He has published over 60 academic papers in leading international journals and conferences, including IEEE TNNLS, IEEE TII, IEEE TITS, IEEE/ASME TMECH, and IEEE/RSJ IROS. He also serves as an Associate Editor for the 2025 and 2026 IEEE/RSJ IROS.