Cockpit Agent Engineering Research: Breakthrough from Digital AI to Physical AI

Cockpit Agent Engineering Research Report, 2025 starts with the status quo of cockpit agents, summarizes the technical roadmap of the R&D and engineering stages and the characteristics of agents from leading OEMs, and predicts the future trends and priorities of cockpit agent application.

Action: Last Mile Mission

Since foundation models were installed in vehicles in 2023, cockpit AI assistants have assumed different tasks at different stage. In 2025, cockpit AI assistants focus on action, which means they "help users get things done" instead of "giving suggestions”, marking an important step in the transformation from "assistants" to "agents".

One typical scenario for cockpit AI assistants in 2025 is ordering food at restaurants:
In 2024, when a user wanted to order coffee, a cockpit AI assistant could only find nearby coffee shops on the map for the user to manually select and navigate to, but the ordering and payment were all fulfilled by the user himself/herself while the AI ??assistant cannot help at all.
By 2025, when a user orders coffee, the cockpit AI assistant will be able to confirm his/her intention and automatically complete a series of operations such as ordering and payment, without the user having to worry about it, thus optimizing the user experience.

The entire process involves technologies related to long-term memory, tool calling, and multi-agent collaboration.

1.Case 1: Tool Calling

In early 2024, OPEN AI's Function Calling was the mainstream technology used by cockpit agents when calling tools, facilitating the direct interaction between a single model and a single tool.
The Model Context Protocol (MCP) introduced by Anthropic in November 2024 addresses the issue of "multi-component collaboration" on basis of Function Calling and improves the application scenarios and efficiency of Function Calling.
In April 2025, Google proposed the A2A (Agent2Agent) protocol to further standardize the communication and collaboration between different agents.

For example, the agent application solution of Lixiang Tongxue in 2025 includes a MCP/A2A technical framework (another framework is CUA):

MCP/A2A: The IVI agent acts as the leader of the multi-agent system (MAS), assigning tasks to third-party agents, which then complete their respective workflows.
CUA (Cockpit Using Agent): The operating system calls a multimodal foundation model to understand instructions/tasks, decomposes and plans them, generates the final action, and calls applets and apps to complete the instructions/tasks. For example, in the payment scenario, after a series of understandings and plans, Lixiang Tongxue calls the API to connect to Alipay's automotive assistant, and uses the relevant applet through Alipay's ecosystem to complete the payment.

During the training process, the team of Lixiang Tongxue uses the MCP management tool service in the Reward module optimization of the agent reinforcement phase, such as using MCP Hub to provide a catalog of callable tool resources for training tasks and business requests.

In the next phase, Lixiang Tongxue plans to strengthen its multimodal capabilities and implement COA (Chain of Action), which means that the same model continuously thinks about how to call external tools to solve problems and take action, further improving the synergy between different modules for tool calling, reasoning and action.

2.Case 2: GUI Agent

A GUI agent (graphical user interface agent) is a specific LLM agent which processes user commands or requests in natural language, understands the current state of the GUI through screenshots or UI element trees, and performs actions that simulate human-computer interaction, thus spanning various software interfaces.

A GUI agent typically includes modules such as operating environment, prompt project, model inference, action, and memory.

GUI agent technology is still far from fully mature, but some OEMs, including Li Auto, Geely, and Xiaomi, have already started to deploy it.

In the ordering scenario aforementioned, Lixiang Tongxue leverages GUI agent technology when selecting a meal package, so that it can operate the screen components automatically without user intervention. The team of Lixiang Tongxue has pointed out that the operation accuracy of the GUI agent will also affect the final action of the CUA framework (because the payment process requires scanning screenshots, which involves the GUI agent). If the accuracy is too low, it may be difficult to guarantee a stable experience for complex tasks such as registering for parking and paying parking fees.

For example, Xiaomi has launched a GUI agent framework “BTL-UI”, which uses a Group Relative Policy Optimization (GRPO) algorithm within a Markov decision process (MDP). The agent should receive the current screen state, user commands, and historical interaction records at each time step, and then output a structured BTL response, converting the input multimodal information into a comprehensive output that includes visual attention zones, reasoning processes, and command execution.

Its implementation methods and core technologies include:
Bionic interaction framework: Based on the BTL-UI model, it simulates human visual attention allocation (blinking), logical reasoning (thinking), and precise execution (action), supporting complex multi-step tasks (such as cross-application calls and multimodal interactions).
Automated data generation: It automatically analyzes screenshots, identifies the interface elements most relevant to user commands, and generates high-quality attention annotations for these zones.
BTL reward mechanism: It meticulously evaluates each cognitive stage in between, checking whether the AI ??correctly identifies the relevant interface elements, performs reasonable logical reasoning, and generates accurate operation instructions.

OEMs are currently transitioning from L2 reasoners to L3 agents, with L3 further divided into four stages.

According to OPEN AI's definition of AGI, Chinese OEMs are currently in the process of transitioning from L2 reasoners to L3 agents. At each different stage, different problems should be solved, with corresponding characteristics:

At present, most OEMs' cockpit AI assistants have delivered "professional services" to a certain extent. The next goal is to achieve "emotional resonance" and overcome the hurdle of "proactive prediction".

For "emotional resonance", NIO offers "Nomi" as a leading player.
In 2025, most AI assistants' emotional chats are implemented primarily through tone changes simulated by TTS technology, terminology from the knowledge base (such as colloquial interjections), and preset emotional scenario workflows. Compared to other cockpit agents, Nomi has two unique advantages:

1.Physical shell: Nomi can materialize more than 200 dynamic expressions through its shell "Nomi Mate" (upgraded to version 3.0 as of November 2025), giving emotional value in the real world. For example, when Nomi interacts with people via voice, it simulates the head movements that occur when people are talking to each other, and simulates the movement of a person's head turning towards the source of a sound when they hear a sound, thus achieving an arc-shaped head turning trajectory.

2.Emotional settings:
In terms of architecture, a dedicated "emotion engine" module is set up. Through three sub-modules, namely "contextual intelligence", "personalized intelligence" and "emotional expression", it uses voice, vision and multimodal perception technologies to achieve contextual arbitration, derive a series of understandings of the current situation, and realize natural human-like reactions in emotional scenarios.

In terms of settings, Nomi can have a personality. Based on the settings, it can perform search associations through a streaming prediction model similar to GPT, exhibiting unique situational responses and providing a personalized experience for each user (such as simulating multiple MBTI personalities, in contrast to Lixiang Tongxue set as ENFJ).

After achieving "proactive prediction," cockpit agents make a breakthrough from digital AI to physical AI.

Starting from L3.5+, generalization has become one of the limiting factors for agents' ability to flexibly cope with multi-scenario tasks. To improve generalization in different scenarios, agents should not only learn policies (what actions to take in a certain state), but also know about dynamic environmental models (how the world will change after performing an action) to make predictions in direct interaction with the environment.

To avoid limitations caused by the shortage of high-quality data, one solution is to learn in a real physical environment to achieve a breakthrough from digital AI to physical AI.

For example, the team of Lixiang Tongxue has found that the effect of data on improving the model's capabilities decreases after using massive Internet data for training the base model, namely the marginal benefit of scaling law in model pre-training declines.

Therefore, the team of Lixiang Tongxue has changed the training method for the next stage. it will focus on the interaction between the model and the physical world. Through reinforcement learning, the model will judge the correctness of the thinking process and accumulate experience and data in the interaction with the environment.

Fei-Fei Li's team from World Labs has proposed "augmented interactive agents," which feature multimodal capabilities with "cross-reality-agnostic" integration and incorporate an emergent mechanism.

In training intelligent agents, Fei-Fei Li's team has introduced an "in-context prompt" or "implicit reward function" to capture key features of expert behavior. The intelligent agents can be trained by physical world behavior data learned from expert demonstrations for task execution. The data is collected by gathering expert demonstrations in the physical world in the form of "state-action pairs".

In 2025, most OEMs chose a multi-agent approach to build their cockpit AI systems. Multi-agent collaboration is also one of the ways to improve the generalization of agents. Through "domain specialization + scenario linkage + group learning", the generalization limitations of existing agents can be broken through from multiple dimensions.

For example, GAC's "Beibi” agent can recognize intent in complex scenarios  through multi-agent collaboration based on foundation model intent recognition, tackling the problems of vertical agents like "lack of unified interaction entry and inefficient collaboration". It eliminates the need for users to operate multiple agents separately (such as adjusting navigation and air conditioning individually), thus improving collaboration efficiency. Its principles include:

Build the core intelligent agent: Fine-tune the pre-trained language model using a pre-set dataset related to automotive scenarios (such as vehicle control, navigation, and other instruction records) to obtain an intent recognition model. Then, build an "intent understanding intelligent agent" based on this model, while adding a caching service to improve response speed.

Parse user intent: Receive user commands (such as voice or touch commands), and infer the intent recognition result (including 1-3 intents and their corresponding confidence scores, e.g., "Find a gas station" confidence score 0.85, "Adjust temperature" confidence score 0.9) from the intent understanding agent, and cache the commands and results.

Call collaborative agents: Make collaborative decisions based on the current scenario (such as driving status, weather), call on target agents related to the intent (such as navigation and vehicle control agents) to work together, and receive the action results of each agent.

Arbitrate, feed back and enforce: Arbitrate based on historical confidence scores (the past success rate of the agents) and the current action result; arbitrate based on the intent recognition model when there are no historical scores, and finally feed back the result to the actuation system (such as the IVI or voice broadcast) to complete the operation.