China Automotive Multimodal Interaction Development Research Report, 2022
  • Dec.2022
  • Hard Copy
  • USD $3,800
  • Pages:170
  • Single User License
    (PDF Unprintable)       
  • USD $3,600
  • Code: YSJ116
  • Enterprise-wide License
    (PDF Printable & Editable)       
  • USD $5,400
  • Hard Copy + Single User License
  • USD $4,000

Multimodal interaction research: more hardware entered the interaction, immersive cockpit experience is continuously enhanced

ResearchInChina's “China Automotive Multimodal Interaction Development Research Report, 2022” conducts analysis and research from three aspects: the installation status of mainstream interaction modes, the application of mainstream vehicle models’ interaction modes, and cockpit interaction solutions of suppliers.

1. Guided by the "third space" concept, multimodal interaction is being deeply applied to intelligent cockpit with five main features.

(1) With the trend of large screen, multi-screen and smart surface materials, touch interaction has gradually expanded its application range

The large screen of center console makes touch control the mainstream interaction mode. For example, Mercedes-Benz EQS and XPeng P7 have almost no physical buttons on the center console, but all done by touch.
Multi-screen cockpits make touch range from the front to the rear, from the center console / co-pilot infotainment extended to the doors, windows, seats and other components. For example, Li Auto L9 eliminates the traditional instrument panel and replaces it with a small TouchBar above the steering wheel; in addition, it also carries a co-pilot screen and rear audio/video screen to achieve five-screen interaction.

(2) Voice interaction evolves from passive to active, personalized and emotional demands will be met
See-and-speak, continuous dialogue, voice source location, wake-up free and other voice technology has been widely equipped in the new cars launched in 2022, the voice interaction mode tends to be more natural.
Personalized experience is currently the focus of the voice function, and intelligent EV brands such as NIO/XPeng/Li Auto are mainly optimized in voice customization, image customization, skill customization, etc.
In the future, the concept of emotional companionship and electronic pets is expected to be realized with the help of voice function.

(3) Face recognition algorithm promotes DMS, OMS and IMS scale installation

Face recognition-based ID login and user habit presets have been implemented in NIO ET7/ET5, XPeng P7/G9, AITO M5/M7, Neta S, and Voyah Dreamer. Among them, AITO can automatically login to Huawei account through face recognition, linking schedule, navigation information, call records, music and video membership rights, third-party application data, and automatic switching of driving information to achieve information flowing.

Face payment function application is still relatively small, which has been installed on Toyota Harrier Premium Edition, and XPeng P7 will be achieved it through OTA upgrade.

Face recognition-based DMS, OMS, and IMS are being installed on a large scale and will promote in-cockpit camera equipment.

(4) Gesture recognition function is single, installed as a complementary interaction mode
Currently, gesture recognition mainly applied to multimedia switching, volume control, phone answering, lighting control, etc., mainly installed as a supplementary interaction mode.
In the future, gesture recognition is expected to be combined with ADAS functions to achieve vehicle summoning, starting and parking based on exterior visual perception.

(5) Fingerprint, iris, vein, heart rate and other biometric applications in the car are still in the exploration stage
Fingerprint is expected to be applied in user ID recording and in-vehicle payment scenarios.
Iris/eye tracking will enhance the accuracy of in-cockpit monitoring such as DMS, and have imagination space in the future in intention prediction and active recommendation.
Vein/heart rate, etc., will be applied under the concept of in-vehicle health.


2. Multimodal recognition and large & multi-screen, AR-HUD, AR/VR, ambient light, high-quality audio and other hardware interaction to be enhanced, immersive cockpit experience continues to grow

(1) Li Auto L9, creating a multimodal interaction experience through five-screen, voice and gestures

Features of Li Auto L9 interaction:
Replaces the traditional instrument screen with HUD + safe driving interaction screen.
Enhances the audiovisual experience through 15.7-inch center console screen + 15.7-inch co-pilot screen + HUD + safe driving interaction screen + 15.7-inch rear entertainment screen with 3K HD resolution and high color reproduction on the automotive screen.
6-voice-zone recognition interaction (AISpeech ) + self-developed speech engine (in cooperation with Microsoft) + 3D ToF sensor (gesture interaction, cockpit monitoring), to achieve multimodal interaction.
Voice + gesture integration, such as point at the sunshade and say "open this", that is, open the sunshade.
Connect to Nintendo Switch via Type-C and rear entertainment screen to create cockpit gaming scenarios.
Two Qualcomm Snapdragon 8155 chips, 24GB RAM + 256G high-speed storage, and dual-5G operator switching to provide computing power and network support.


 (2) NIO, Li Auto, Audi and others take the lead in AR/VR glasses installation for an immersive cockpit experience

Jointly developed with NREAL, the NIO AR glasses can project a 6m viewing distance and 201-inch-equivalent screen (optional price RMB 2,299). VR glasses are jointly developed with NOLO, and equipped with ultra-thin Pancake optical lenses for binocular 4K display.


Li Auto AR glasses are provided by Leiniao Technology, a subsidiary of TCL, and the product was available on Li Auto Store from August 2022. Leiniao Air adopts the polarization Birdbath + MicroOLED technology solution and is directly connected to the rear audio and video system of Li Auto L9. Plugging the cable at the legs of the glasses into Li Auto's rear-row DP jack provides users with a 140-inch giant screen (4m distance) viewing experience.


Audi VR glasses will be first launched in USA in 2023. The hardware is supported by HTC and integrated into Holoride Pioneers' Pack in-vehicle entertainment system for games and video scenarios.
3. Chip, algorithm and system integrator work together to create active cockpit interaction based on multimodal interaction

(1) Chip companies, represented by Horizon, integrate multimodal interaction into intelligent driving solution

Horizon Halo, a cockpit solution built by Horizon based on Journey 2 and Journey 3, can integrate vision, voice and other sensor data to achieve active interaction. Among them, Halo 3.0 can provide a complete set of AI solutions including DMS, face detection, behavior detection, gesture recognition, child behavior detection, multimodal voice interaction and other functions for front and rear row users.


(2) iFLYTEK, Cerence, etc. entered with voice, and SenseTime, ArcSoft, etc. entered with vision to achieve the integration of superior modes with other modes to create an overall cockpit solution

iFLYTEK builds a multimodal system based on "listening, speaking, seeing and displaying" all-link technology, realizing that the vehicle can process the fusion of voice, image, live body and other information throughout the car-using cycle of getting on - driving - getting off, so as to understand passengers' information more actively and deeply, and thus actively care for them, push relevant contents/services, and change vehicle settings to explore disruptive interaction experience.


Based on its strengths in voice, Cerence will integrate vehicle data (fatigue monitoring, mobile phone interconnection, entertainment system, air conditioner, fuel, charging, seat, GPS, air quality) with in-vehicle multimodal interaction (voice, speech synthesis, text input, eye tracking, gesture recognition, emotion recognition, biometrics) to create immersive cockpit interaction in the future.


 (3) ADAYO, Desay SV and other Tier1, combining multimodal interaction with scenarios to create a personalized cockpit experience

Based on high computing power AI chip, through visual, auditory, touch, multimodal front-fusion, multimodal rear-fusion and other recognition technologies, ADAYO multimodal interaction system can realize 4 major categories and 70+ human-machine interaction scenarios, including voice + gesture to open the window or air conditioner, see-and-speak, greeting when getting into the car, soothing children crying, voice control, face start engine, eye control to light up screen, driver behavior monitoring and dangerous behavior reminder, etc., providing users with active emotional multimodal interaction scenario experience and meeting the personalized needs of thousands of users.


1 Overview of Multimodal Interaction
1.1 Definition of Multimodal Interaction
1.2 Intelligent Cockpit Multimodal Interaction Development Roadmap
1.3 Multimodal Interaction Features of Representative Models
1.4 Touch Interaction
1.5 Voice Interaction
1.5.1 Comparison of In-vehicle Voice Functions of Major Brands
1.5.2 Voice Assistant Visualization Will Warm the Interaction
1.6 Gesture Interaction
1.6.1 Gesture Interaction Technology Route
1.6.2 Gesture Interaction Function of Representative Models
1.6.3 Gesture Interaction Trend
1.7 Facial Recognition
1.7.1 Facial Recognition Functions of Representative Models
1.7.2 In-cabin Monitoring based on Facial Recognition 
1.8 Biometrics
1.9 Multimodal Recognition and Hardware Interaction
1.9.1 Exterior Headlight Interaction
1.9.2 Interior Ambient Light Interaction
1.9.3 AR/VR Interaction

2 Industry Chain and Development Trends
2.1 Multimodal Interaction Industry Chain
2.2 Multimodal Perception Algorithms
2.2.1 Voice Algorithms
2.2.2 Vision Algorithms
2.3 Multimodal Interaction Applications Require Significantly More Arithmetic Power on Cockpit Chip
2.4 Multimodal Interaction Security Strategy
2.4.1 Policy Environment
2.4.2 Multimodal Interaction Laws and Regulations
2.4.3 In-cabin Information Security Strategy of OEMs
2.5 Development Trends

3 Multimodal Interaction Solution for Benchmark Models
3.1 Emerging Automaker Brands
3.1.1 NIO ET7 
3.1.2 NIO ET5 
3.1.3 XPeng P7 
3.1.4 XPeng G9 
3.1.5 Li Auto ONE 
3.1.6 Li Auto L9 
3.1.7 Neta S
3.1.8 AITO M5
3.1.9 AITO M7 
3.1.10 Aiways U6 
3.1.11 Voyah Dreamer
3.1.12 NIUTRON NV 
3.1.13 IM L7
3.1.14 HiPhi Z 
3.1.15 ZEEKR 009
3.1.16 Avatr 11 
3.1.17 Rising Auto R7 
3.1.18 DeepAI SL03 
3.1.19 Leap Motor C01 
3.1.20 Tesla Model Y
3.2 Conventional Joint Venture Brands
3.2.1 New Ford Explorer
3.2.2 BMW iX
3.2.3 Mercedes-Benz EQS 
3.2.4 Toyota Harrier
3.2.5 Denza D9
3.2.6 Lotus ELETRE 
3.3 Conventional Independent Brands 
3.3.1 Jetour DaSheng i-DM
3.3.2 BYD Seal
3.3.3 OShan Z6 
3.3.4 GAC Trumpchi Emkoo
3.3.5 Boyue L 

4 Multimodal Interaction Solution Providers
4.1.1 Profile
4.1.2 Intelligent Vehicle Product Layout 
4.1.3 Multimodal Interaction System
4.1.4 Intelligent Cockpit Solutions
4.1.5 Intelligent Interaction Development Routes
4.1.6 Multimodal Interaction Becomes the Key Direction of Xunfei Super Brain 2030 Plan
4.1.7 AI Virtual Human Interaction Platform
4.1.8 Major Customers
4.2 Cerence
4.2.1 Profile
4.2.2 SSE Technology
4.2.3 Cockpit Interaction Solutions
4.2.4 Co-Pilot Customers
4.2.5 Product Development Routes
4.2.6 Partners
4.3 AISpeech
4.3.1 Profile
4.3.2 Multimodal Interaction Solutions
4.4 VW-Mobvoi
4.4.1 Profile
4.4.2 OEM Intelligent Connected Solutions and Multimodal Interaction Principles
4.5 Tencent
4.5.1 Profile
4.5.2 Intelligent Cockpit Solutions
4.5.3 Interaction Technology Applications
4.5.4 Intelligent Voice Interaction System 
4.5.5 Multimodal Interaction System
4.6 Baidu
4.6.1 Profile
4.6.2 Voice Core Technology
4.6.3 Intelligent Cockpit Solutions
4.6.4 Multimodal Interaction Solutions
4.6.5 Intelligent Cockpit Partners
4.7 Banma
4.7.1 Profile
4.7.2 Intelligent Cockpit Solutions
4.7.3 Features of Banma Luoshen OS Interaction 
4.7.4 Voice Technology
4.7.5 Voice Assistant “XiaoGenBan”
4.7.6 Supporting OEMs
4.8 SenseTime
4.8.1 Profile
4.8.2 SenseAuto Platform
4.8.3 SenseAuto Cabin 
4.8.4 Major Customers
4.9 ArcSoft
4.9.1 Profile
4.9.2 In-cabin Monitoring Solutions
4.9.3 Business Models
4.9.4 Partners
4.10 Hikvision
4.10.1 Profile
4.10.2 Intelligent Cockpit Solutions
4.11 Horizon
4.11.1 Profile
4.11.2 Chip Product System
4.11.3 Multimodal Interaction Solutions
4.11.4 In-vehicle OS - TogetherOS
4.11.5 Product Development and Business Models
4.12 ADAYO
4.12.1 Profile
4.12.2 Business Layout
4.12.3 Multimodal Interaction System
4.12.4 Partners
4.12.5 Products and Supporting Customers
4.13 Desay SV
4.13.1 Profile
4.13.2 The Fourth Generation Cockpit Solution


Automotive Microcontroller Unit (MCU) Industry Report, 2024

With policy support, the localization rate of automotive MCU will surge. Chinese electric vehicle companies are quickening their pace of purchasing domestic chips to reduce their dependence on impor...

Automotive Digital Key Industry Trends Research Report, 2024

Automotive Digital Key Industry Trends Research Report, 2024 released by ResearchInChina highlights the following: Forecast for automotive digital key market;Digital key standard specifications and co...

Automotive XR (VR/AR/MR) Industry Report, 2024

Automotive XR (Extended Reality) is an innovative technology that integrates VR (Virtual Reality), AR (Augmented Reality) and MR (Mixed Reality) technologies into vehicle systems. It can bring drivers...

OEMs’ Next-generation In-vehicle Infotainment (IVI) System Trends Report, 2024

OEMs’ Next-generation In-vehicle Infotainment (IVI) System Trends Report, 2024 released by ResearchInChina systematically analyzes the iteration process of IVI systems of mainstream automakers in Chin...

Global and China Automotive Lighting System Research Report, 2023-2024

Installations of intelligent headlights and interior lighting systems made steady growth. From 2019 to 2023, the installations of intelligent headlights and interior lighting systems grew steadily. I...

Automotive Display, Center Console and Cluster Industry Report, 2024

Automotive display has become a hotspot major automakers compete for to create personalized and differentiated vehicle models. To improve users' driving experience and meet their needs for human-compu...

Global and China Passenger Car T-Box Market Report, 2024

Global and China Passenger Car T-Box Market Report, 2024 combs and summarizes the overall global and Chinese passenger car T-Box markets and the status quo of independent, centralized, V2X, and 5G T-B...

AI Foundation Models’ Impacts on Vehicle Intelligent Design and Development Research Report, 2024

AI foundation models are booming. The launch of ChapGPT and SORA is shocking. Scientists and entrepreneurs at AI frontier point out that AI foundation models will rebuild all walks of life, especially...

Analysis on Geely's Layout in Electrification, Connectivity, Intelligence and Sharing

Geely, one of the leading automotive groups in China, makes comprehensive layout in electrification, connectivity, intelligence and sharing. Geely boasts more than ten brands. In 2023, it sold a tota...

48V Low-voltage Power Distribution Network (PDN) Architecture Industry Report, 2024

Automotive low-voltage PDN architecture evolves from 12V to 48V system. Since 1950, the automotive industry has introduced the 12V system to power lighting, entertainment, electronic control units an...

Automotive Ultrasonic Radar and OEMs’ Parking Route Research Report, 2024

1. Over 220 million ultrasonic radars will be installed in 2028. In recent years, the installations of ultrasonic radars in passenger cars in China surged, up to 121.955 million units in 2023, jumpin...

Automotive AI Foundation Model Technology and Application Trends Report, 2023-2024

Since 2023 ever more vehicle models have begun to be connected with foundation models, and an increasing number of Tier1s have launched automotive foundation model solutions. Especially Tesla’s big pr...

Qualcomm 8295 Based Cockpit Domain Controller Dismantling Analysis Report

ResearchInChina dismantled 8295-based cockpit domain controller of an electric sedan launched in December 2023, and produced the report SA8295P Series Based Cockpit Domain Controller Analysis and Dism...

Global and China Automotive Comfort System (Seating system, Air Conditioning System) Research Report, 2024

Automotive comfort systems include seating system, air conditioning system, soundproof system and chassis suspension to improve comfort of drivers and passengers. This report highlights seating system...

Automotive Memory Chip and Storage Industry Report, 2024

The global automotive memory chip market was worth USD4.76 billion in 2023, and it is expected to reach USD10.25 billion in 2028 boosted by high-level autonomous driving. The automotive storage market...

Automotive AUTOSAR Platform Research Report, 2024

AUTOSAR Platform research: the pace of spawning the domestic basic software + full-stack chip solutions quickens. In the trend towards software-defined vehicles, AUTOSAR is evolving towards a more o...

China Passenger Car Electronic Control Suspension Industry Research Report, 2024

Research on Electronic Control Suspension: The assembly volume of Air Suspension increased by 113% year-on-year in 2023, and the magic carpet suspension of independent brands achieved a breakthrough ...

Global and China Hybrid Electric Vehicle (HEV) Research Report, 2023-2024

1. In 2025, the share of plug-in/extended-range hybrid electric passenger cars by sales in China is expected to rise to 40%. In 2023, China sold 2.754 million plug-in/extended-range hybrid electric p...

2005- All Rights Reserved 京ICP备05069564号-1 京公网安备1101054484号