Tencent's AI avatars can finally interact with real objects

If you have spent any time watching AI-generated avatars, you know the vibe: a hyper-realistic character stands pinned to the center of the frame, moving its lips convincingly while waving its hands aimlessly in the air.

The illusion usually shatters the second you need the avatar to do something normal, like pick up a coffee mug or unbox a gadget.

Tencent researchers just introduced a framework called InteractAvatar to fix this exact stiffness, teaching digital humans how to actually grab, touch, and use physical objects in their immediate environment.

Putting an end to stiff talking heads

Most existing avatar tools, including methods like Omni Avatar, focus primarily on head movement, facial expressions, and basic body language. They can talk, but they cannot manipulate items in their surroundings.

InteractAvatar targets grounded human-object interaction. The system allows avatars to execute complex gestures and handle everyday items, such as putting on headphones, picking up a smartphone, or petting a plush toy.

It also handles continuous commands with granular temporal precision. Instead of guessing when to act, the model responds to specific timestamps—for instance, touching an apple at second 4 and lifting it up at second 12.

Decoupling perception from video rendering

Teaching an avatar to interact with objects usually triggers a classic control-versus-quality dilemma: forcing exact physical contact often mangles the visual quality of the video.

Tencent tackled this by introducing a dual-stream framework that decouples environmental perception from video synthesis.

Inside the architecture, a Perception and Interaction Module (PIM) leverages object detection to understand the scene and plan text-aligned motions. Working alongside it, an Audio-Interaction Aware Generation Module (AIM) renders the actual talking video. A shared motion-to-video aligner lets both streams run in parallel, keeping the movement tightly synchronized without degrading frame quality. The team also rolled out GroundedInter, a new benchmark tailored to evaluate these complex interactions.

Why it matters

For digital creators and e-commerce streamers, this is a massive leap forward. A virtual host that cannot hold a product is barely more useful than a static slide deck. Once an avatar can convincingly showcase an item on cue, automated video production becomes genuinely practical.

We think Tencent took a very smart architectural route here. Pure video generators are hitting diminishing returns on surface polish, but they still struggle whenever an actor has to touch an object without glitching through it. By separating the spatial planning from the rendering pipeline, InteractAvatar offers a clean blueprint for grounded physics without ballooning training complexity.

While the project is still at the research stage, it clearly signals where virtual humans are headed: away from isolated green-screen mannequins, and into characters that can actually handle the physical world around them.