Computer vision in gaming: Leveling up player experience

Generative AI in the gaming industry

One major trend is the use of generative AI to make game characters more intelligent and responsive. A great example is the creation of the first AI-powered boss in the upcoming online game MIR5.

In gaming, a boss fight is typically a challenging battle against a powerful enemy that appears at key moments in a game, often at the end of a level or storyline. These fights usually test a player's skills and are designed to feel intense and rewarding.

‍

Traditionally, boss characters follow scripted patterns, using the same set of moves every time. But in MIR5, this new AI boss actually learns from the players it fights. It watches how players behave, what weapons they use, how they move, and what strategies they rely on, and then adapts its own attacks to keep the battle challenging. 

So, if you return to fight the same boss again, it may not act the same way twice. This kind of adaptive behavior makes battles feel more realistic, forcing players to think on their feet instead of just memorizing patterns.

Computer vision in VR gaming

Another AI trend in gaming is the growing use virtual reality (VR). VR is a type of technology that lets players step into a fully digital, 3D world using a headset and motion controls. It creates the feeling of being inside the game, rather than just watching it on a screen. 

Combined with computer vision, AI is being used to track how players move and react more accurately in VR games. It involves using cameras and sensors to capture real-world movements, like head turns, hand gestures, or even facial expressions, and then translating that data into in-game actions. This helps create smoother, more lifelike experiences in VR, where your gestures and movements feel truly connected to the game world.

Mixed reality racing using computer vision

Hot Wheels: Rift Rally is a racing game that combines a physical remote-controlled (RC) car with virtual gameplay. Players can drive the RC car around their living space, while the screen shows a digital version of the car performing stunts like drifts, jumps, and boosts. Although the real car remains on the ground, the game creates the illusion of high-speed action by overlaying visual effects on top of a live video feed.

This setup relies on mixed reality technology (it blends real and digital worlds, allowing physical and virtual elements to interact in real time) and computer vision to make the experience work. The RC car is equipped with a camera that scans the environment as it moves. Players place visual markers (black and white cardboard gates), called fiducial markers, around the room, which the system uses as reference points. 

Computer vision helps the game recognize these markers, track the car’s location, and build a virtual map of the space. This allows the game to accurately place digital content, like tracks and obstacles, within the real-world environment, syncing physical movement with what’s shown on screen.

‍

Teaching virtual pets using Vision AI

Peridot, a mobile game developed by Niantic (the same company behind Pokémon Go, one of the earliest and most popular examples of computer vision in an AR game), uses computer vision and augmented reality to make caring for a digital pet feel grounded in the real world. 

Players can take care of a creature called a Peridot by feeding it, playing games, and taking it on walks. As you move through your home or neighborhood, the pet appears on your phone to explore alongside you, running across the floor, reacting to objects, and even hiding behind real furniture.

To make this possible, the game uses computer vision techniques like 3D mapping and semantic segmentation, where the system uses your phone’s camera to scan and understand the environment. This helps the pet recognize and respond to different surfaces, like distinguishing grass from concrete, or reacting to obstacles like chairs and trees. 

‍

Hand tracking and shared mixed reality spaces

Demeo, a digital tabletop RPG (role-playing game), uses mixed reality features to bring its gameplay into the real world through headsets like the Meta Quest 2 and Quest Pro. With a recent update, the game now supports hand tracking and colocation, making it possible for players to use natural hand movements to interact with game pieces, and to share the same physical and digital space with others in local multiplayer sessions.

For instance, players can play Demeo while petting their real dog - a small but meaningful sign of how mixed reality can blend digital and physical spaces. These features are enabled by computer vision technologies. Hand tracking works by using the headset’s built-in cameras to detect and interpret the movement and position of a player’s hands, removing the need for traditional controllers. 

‍

Meanwhile, colocation uses spatial anchors and point cloud data to map the real-world environment so that multiple headsets can synchronize the same virtual space. This lets players in the same room see the game board and each other in the same positions, creating the effect of sitting around a shared, virtual tabletop.