Early reviews are pouring in for Google’s Gemini Ultra, the company’s most advanced large language model that powers a paid version of chatbot Gemini, a rival to ChatGPT. The reviews make us think that after an early scramble to catch up to its Microsoft-backed rival, Google is going to be just fine in this market.

The vast majority of developers and founders I’ve spoken with say Gemini Ultra appears to be roughly at par with GPT-4, and that’s a fine accomplishment in and of itself, even if it comes nearly a year after GPT-4 made its debut. A final verdict on Gemini will come after Google makes it available to developers through an application programming interface so they can use it in their own apps.

As part of the Gemini chatbot, Gemini Ultra is much faster, less wordy and less bland in its answers than the paid version of ChatGPT powered by GPT-4, several developers tell me. Gemini also has surprised some of these users in its creative story-telling capabilities. (See some examples here and here.)

Gemini falls short in a number of areas, most noticeably coding and reasoning problems, these developers say. That’s surprising, given that Google has said Gemini’s coding capabilities were superior to GPT-4, though some of those claims may have been artificially inflated. Google researchers are anxiously awaiting outside evaluations by independent organizations like LMSYS' Chatbot Arena to see whether or not these organizations confirm the performance benchmarks Google published about Gemini based on its own tests, my colleague Jon Victor hears.