TORNADO will develop an innovative, multifunctional and adaptive cloud robotics platform, supporting advanced navigation of an autonomous mobile robot (AMR) within complex, time-varying, real-world, human-populated indoor environments. The TORNADO AMR will be able to manipulate small, soft or deformable objects (SSDs) to an unprecedented degree of success, as well as to naturally interact with humans via hand gestures or verbal conversation, by exploiting the zero-shot generalization abilities of deep neural Foundation Models (FMs) for robotics. The AMR’s intelligence will rely on a pool of pretrained cloud-hosted FMs, which shall be further adjusted on-the-fly to the current situation via Out-of-Distribution Detection, Test-Time Adaptation and Few-Shot Adaptation subsystems. These will exploit human feedback if available, but will also support autonomous and dynamic cognitive adaptation. Additionally, the TORNADO system will be able to automatically select and set-up on-the-fly the most suitable combination of FMs and non-neural robotics algorithms during deployment, depending on the current situation. In cases of failure, on-the-fly skill acquisition will be supported via integrated, novel Learning-from-Demonstration methods facilitated by an innovative Augmented Reality (AR) interface and eXplainable AI (XAI) algorithms. The adaptive TORNADO system will allow the robot to perform difficult, non-repetitive manipulation tasks on previously unseen SSDs that may change shape during handling, as well as to flexibly adjust to SSDs of different sizes during operation. Measurement of human trust to interactive robots and human behavioral modeling will aid optimal integration/ acceptance of TORNADO into society. Validation will take place at TRL-5 in 3 different industrial Use-Cases: flexible small gears manipulation and deformable ply-sheets handling (gears factory), palliative patient care (hospital) and product quality sampling/waste collection (dairy processing plant).

The project’s main objectives are as follows:

• Boost ability of AMRs to handle SSDs within complex, people-centric environments

• Improve adaptivity of Foundation Models (FMs) for AMRs operating in complex, dynamic settings

• Raise the level of robotic perception and safe interaction

• Develop advanced hardware solutions facilitating cloud-enabled AMRs and/or the diffusion of autonomous robots

• Integrate Social Sciences & Humanities (SSH) research/methodology

• Deliver a functional system of cooperating, cloud/AI-enabled autonomous robots and improve performance

ITML’s role in the project

In the TORNADO project, ITML is the Project Coordinator. Also, leads the task “AI for sound analysis and verbal communication”, which involves development of the Sound and Language Manager (SLM) module and all its internal algorithms. The SLM module will provide real-time text-to-speech, speech recognition, text sentiment classification and generic intelligent dialogue functionalities, via pretrained audio FMs and Large Language Models (LLMs), with appropriate task/UC-specific heads appended as/if needed, so that the TORNADO AMR can engage in verbal dialogue, receive verbal orders and answer questions.