Game Characters Come to Life with Nvidia ACE's AI-Powered Conversations.

(Image credit: Tom's Hardware)

There are numerous ways to text chat using a large language model, including ChatGPT, Google Bard, and MLC LLM, a mobile chatbot for your area. The ability to have a free-form conversation with NPCs (non-player characters) in games rather than using a pre-set set of interactions is the next frontier for AI.
Nvidia CEO Jensen Huang unveiled ACE for Games during the company's Computex 2023 keynote. This AI model foundry service uses natural language conversation, audio-to-facial expression, and text-to-speech / speech-to-text capabilities to bring game characters to life. Huang demonstrated a game in which a human player could converse with an NPC named Jin, who owns a ramen shop, and receive responses that sounded authentic and matched the NPC's backstory.

In the demo, the player (who goes by the name of Kai) enters Jin's Ramen shop, asks him how he's doing (in voice), and they start talking about how dangerous the neighborhood is. Jin responds to Kai's request for assistance: "I've heard rumors that the powerful crime lord Kumon Aoki is wreaking havoc on the city if you want to do something about it. He could be to blame for this violence." Kai asks Jin where he can find Aoki, and Jin responds by directing Kai on his quest.
Huang stated that AI "will contribute to the reddening and synthesis of the environment, but AI will also animate the characters." AI will play a significant role in video games in the future.

Three existing components will all have high-speed access through Nvidia ACE for Games. The first, called Nvidia NeMo, uses AI to train and deploy LLMs, and it includes a feature called NeMo Guardrails intended to stop inappropriate or otherwise risky AI conversations. This would prevent NPCs from responding to inappropriate or irrelevant user prompts. Guardrails also have security features, so users and potential prompt injectors shouldn't be able to "jailbreak" the bots and make them do bad things.
The company's text-to-speech and speech-to-text program is called Nvidia Riva. A gamer will ask a question through their microphone in the ACE for games workflow, and Riva will convert it to text before feeding it to the LLM. The LLM will then produce a text reply, which Riva converts into audio the user can hear. Naturally, we'd anticipate that games would also display the responses in text. On the company website, you can test out the text-to-speech and speech-to-text features of Nvidia Riva.

The final step in the ACE for games workflow is provided by Nvidia Omniverse Audio2Face, which enables the characters' facial expressions to correspond to their speech. You can try this product, currently available in beta, here.
The AI-in-gaming startup Convai created the Kairos demo as part of Nvidia's Inception program, which links up-and-coming businesses with venture capital. The company provides a toolset on its website that enables game developers to create realistic NPCs with intricate backstories.

The business has a fantastic explainer video that shows how and what its tools can do. Players can be seen asking NPCs in the video to perform actions that involve natural objects and other game characters. In the video, a player requests that an NPC hand him a gun lying on a table, and the NPC complies. In another scene, the player commands a soldier NPC to fire at a target in a specific location. We also see how Convai's resources enable all of this.

It is crucial to have that extra context so that the NPC is aware of what is happening in-game. We recently tested an AI plugin for Minecraft that lets you communicate with non-player characters (NPCs), but the NPCs have no situational awareness. For instance, we could converse with a sheep after we killed it (and it wasn't aware that it was dead).