Research
The Robot Learning Lab at the University of Freiburg seeks to advance robot learning algorithms for mobility and manipulation to achieve autonomy at scale. Toward this goal, our research addresses the underlying fundamental scientific challenges in perception, state estimation, motion planning, mobile manipulation, human-robot interaction, learning fundamentals, and responsible robotics, as depicted below.
Please select an area for more details.
Perception
Reliable and accurate scene interpretation of the environment is a critical prerequisite for the autonomous operation of robots in the real world. This problem is inherently challenging as it requires knowledge of semantics and the geometric context of a scene as well as the intrinsic relationship between them. It becomes more difficult due to the multitude of object types with large intra-class variance, occlusions, deformations, varying viewpoints, and illumination conditions. To equip robots with such modeling abilities, we focus on pushing the limits of the state-of-the-art in fundamental recognition tasks and simultaneously explore new research directions by going beyond established perception tasks while lifting constraints such as domain dependency in order to render these methods applicable in the real world.
While there are numerous works that address the aforementioned fundamental tasks, they are usually subject to a set of constraints hindering the scalability of robot autonomy. The elaborative data annotation hinders existing methods to be applied in changing environments with a strong dataset shift. This raises the question of to what extent recent advances in self-supervised learning can help to lift this constraint. Moreover, only a few works exist that exploit online data collected during deployment as an additional source for training pretrained networks in an online manner. Lastly, our recent work shows that finding synergies between learning-based and classical methods can be beneficial. This motivates the exploitation of the complementarity of these methods to the benefit of robot autonomy.
State Estimation
Once a robot is capable of reliably perceiving its environment, the next step is to further process the acquired knowledge, e.g., by estimating the state of both itself and other agents and, thus, gaining an understanding of the relation between objects and the environment. Typically, state estimation includes a broad range of research topics such as object tracking or prediction to obtain a spatiotemporal model of other agents and oneself. Another task is simultaneous localization and mapping (SLAM), which addresses creating a map of an unknown environment while at the same time reporting the position of the agent within this map. In this context, we investigate how to leverage deep learning to solve these tasks more reliably and accurately.
While recent works successfully take the first steps towards adopting a learning-based approach for the aforementioned problems, several open research directions still remain. To this end, our research focuses on deep end-to-end pipelines that make use of well-suited data representations and integrate multiple modalities efficiently. In order to bridge the gap between deep and classical methods, we incorporate uncertainty estimation techniques to quantify both aleatoric and epistemic uncertainty to improve decision making in downstream tasks. We investigate learning abstract representations that are to be utilized interchangeably and allow efficient sharing among multiple agents. To this end, collaborative (multi-future) prediction, tracking, and SLAM can further help to pool a more diverse set of data and continuously provide state updates faster than a single robot does.
Motion Planning
Learning to move and act in the physical world is a central element across all embodied AI tasks. While doing so, robotic agents are constrained by widely varying kinematic abilities. Simultaneously, they have to respect constraints imposed by their environment and the task. While planning approaches provide powerful general-purpose algorithms, they struggle with unknown environments and are unable to take into account high-dimensional contexts and abstract goals. Research questions in this context include hierarchical methods at the right levels of abstraction, reinforcement learning to make decisions in unexplored environments, learning from demonstrations and high-dimensional contexts to efficiently search the solution space as well as exploring hybrid learning and action-model based approaches.
Open questions include the development of hybrid planning-learning methods, such as learning priors from high-dimensional contexts to guide planning methods or explicit planning within learned spaces. Important problems for learning methods include the formulation of general, embodiment, and task-independent learning methods, the generalization to unseen environments and task constraints, and learning to improve initial planner solutions. Furthermore, with increasing automation and collaborative robotic systems, several novel problems in coordination and cooperation arise.
Mobile Manipulation
Mobile manipulation applications are ubiquitous across both industry and services and are a core component in visions such as robotic housekeepers. Progress in this area promises to unlock a much wider area of tasks for robotic automation. The challenges of mobile manipulation include the generation of whole-body motions for large continuous action spaces consisting of both base and arms, a combination of long-horizon task reasoning with short-term acting, mapping from abstract human-specified goals to executable goals, and the simultaneous sensing and exploration of the environment while fulfilling task goals.
While there has been a lot of progress in individual aspects of mobile manipulation, many open problems remain. These include learning whole-body policies for mobile manipulation tasks, the development of truly multi-task models together with the ability to understand abstract, human-specified goals. Lastly, a major challenge remains the incorporation of sensing and acting over different time scales into a single system, including both the long-horizon task learning and the short-term motion generation. This provides a perfect field to bring together the best of control, planning, and learning.
Human-Robot Interaction
The transition of robots from confined industrial environments to open public spaces poses new challenges, in particular regarding the interaction between robots and humans. The research in human-robot interaction spans a wide range of topics: (1) Ensuring that the robot behavior among humans is safe as well as compliant with the cultural norms of our society. (2) Enabling human-robot collaboration, by combining the dexterity and versatility of a person with the strength and precision of a robot. (3) Allowing humans to communicate with robots and teach them how to adapt and improve their behavior when facing novel or evolving tasks. We investigate these challenges and explore novel solutions to address them.
There exist many open research questions left to be addressed. New methods need to be developed to enable efficient human-robot collaboration, in particular carefully considering how robots can be deployed as an enhancement of human capabilities rather than a replacement, and how this will define the future of human work. Enabling efficient communication from robot to human will also play a critical role in the near future. Communication via natural language will provide increased flexibility compared to current interfaces such as keys and buttons. Moreover, instead of passively learning from them, robots will need to inquire humans in the environment when in doubt. These are a few of the open problems we aim to tackle in our future work.
Learning Fundamentals
The sparsity of annotated data along with an ever-changing dynamic world forms one of the major reasons that hinder the rapid deployment of robots in the real world. Fully-supervised learning approaches rely on large annotated datasets which are often extremely arduous to generate when robots have to be deployed in a wide range of environments. Some annotations, specifically HD maps, also need to be constantly maintained which is impractical even for large corporations. One solution to this challenge is to employ systems that can directly leverage streams of raw unlabelled data obtained from sensors onboard a robot to iteratively improve their predictions. Examples of such systems include (1) self-supervised approaches that learn to perform tasks without relying on task-specific annotations, (2) continual and interactive learning systems that update their models to account for the new information which retaining previous information, and (3) robust multimodal learning systems that can safely operate even in extreme weather conditions.
While there has been a lot of research in the aforementioned fields in the past few years, further research is warranted to promote their widespread adoption in the real world. Firstly, many of the self-supervised approaches are extremely task-specific, constraining the domains in which they can be applied. It seems to explore the existence of task-agnostic self-supervision techniques that can be readily applied to a wide range of domains. Secondly, it is often unclear why there remains a huge performance gap between fully supervised and self-supervised approaches, and new approaches should look towards minimizing this performance drop. Thirdly, it would be interesting to analyze the benefit of merging continual and self-supervised learning on many perception and localization tasks. Doing so would help an agent to reason about an ever-changing environment without the need for any explicit supervision, thus improving its overall performance. Lastly, it would be worth investigating and quantifying the benefit of deploying robots as embodied agents to leverage the rich self-supervision gathered through their active interaction with the environment.
Fairness & Ethics
Considerations of Fairness and Ethical behavior in artificial systems are becoming more relevant, as more and more artificial agents interact with general human society. Studies of existing datasets and agents trained on these have demonstrated that these agents exhibit discriminatory behavior towards historically disadvantaged groups. Mitigating these effects is more challenging than it might seem at first sight. A simple erasure of protected traits of disadvantaged groups only leads to secondary indicators dividing individuals along the same lines. An equalization of outcomes between groups erases positively discriminatory behaviors of individuals which are meant to create greater equity for these people.
There are a number of open research questions, which are arguably still in their infancy. For one, the approaches above assume an external perception of the problem of acting equitably or fairly. However, perceptions of being treated fairly and equitably are highly subjective, thus it would be prudent to consider the internal perception of humans around robots with works from multi-agent reinforcement learning such as recursive social MDPs are opening the first avenues in this direction. In addition, we have yet to consider how to transfer a general notion of fairness between tasks. Our own proposals either focus on a model for a very specific task or require prior demonstrations of both normal as well as unfair behavior and human labels for the latter. This opens up interesting research avenues on how to endow robots with a general sense of fairness.