Perceive, Predict, Perform: Architecting the Cognitive Stack for Generalist Robots

This invited talk by Babu Ajish, a team lead at the DFKI Robotics Innovation Center in Bremen, examines how AI is transforming robotics through a cognitive stack organized around three functions: perceive, predict, and perform. After tracing robotics history from early sense-plan-act systems through modular deep learning to today's end-to-end learning, the talk explains why robotics lags the LLM revolution, invoking Moravec's paradox, the irreversibility of physical actions, and severe robot data scarcity. The perceive layer is addressed by vision-language-action (VLA) models built on VLM backbones, combining a slow high-level reasoning system with a high-frequency action expert using techniques such as action chunking and diffusion-based control; examples from Physical Intelligence and Gemini Robotics show generalization to unseen tasks with only minutes of teleoperated correction. The predict layer uses world models, either pixel-generating or latent-space, such as Meta's V-JEPA and NVIDIA Cosmos, serving as planners, synthetic data engines, and simulators. The perform layer relies on massively parallel simulation training and reinforcement learning, with sim-to-real transfer, human demonstration, and verbal coaching; the Atlas robot is cited as trained entirely in simulation. The HPC connection lies in cloud-scale foundation model training that must then run on power-constrained edge hardware at limited model sizes and frequencies. Open challenges include long-term reliability, generalization, real-time edge inference, and above all the absence of safety verification and guarantees for neural policies.