So, Grok 3 is expected tonight at 8 PM Pacific Time and with some apprehension I will get into the dangerous territory of predicting technology. I am not trying to be too precise here, but wanted to set forth my expectations and signals I am looking for given that I have been following LLMs keenly over the last 3 years.

What we know about Grok 3 and xAI .. with some thoughts on what it could mean

100,000 H100 GPU Cluster

Grok 3 is being trained on a 100,000 Nvidia H100 GPU Cluster. GPT 4 was supposed to have been trained on 24,000 GPUs. There was buzz around the new architecture which allowed for a 100,000 GPU cluster to achieve coherence through some innovation at the Network Link layer. Meaning: If scaling laws hold, this should be a substantially better model. I am assuming that xAI has done a good job of Data and Algos, and the scaling laws should deliver better results over GPT4 other things being equal.

 

Data from Twitter

With ChatGPT some of the core data sources have been protective of their data including Reddit, Twitter among others. Twitter/X have a near monopoly of high velocity of data. It will be interesting to see the quality of Grok 3 given this advantage. TikTok is supposed to deliver better recommendations because

 

Model vs. Systems

All the buzz is still for the models. The real breakthrough is going to happen at the systems level where all this tech is put to work on internal data stores that companies possess. While the model vendors are going for the level of power where they could capture the most value, it remains to be seen how much value enterprises are ready to cede to the model vendors and adopt models in a way that gives significant structural pricing power for the model vendors.

 

Productization

Beyond the model, the competition is also at the level of products and OpenAI is clearly focused on leading with products besides their research. Elon and xAI also have a product pedigree, so it would be interesting to see what productized features they deliver besides a good model.

 

Elon Musk’s reputation

Elon Musk has delivered ground breaking technologies including with Tesla, SpaceX and Starlink – mostly pioneers though there have been some small sized incumbents in those spaces. This is an epic attempt to catch up after starting late, and given all the drama around the conflict with Sam Altman and Elon’s role in the new administration this is going to matter a lot for his reputation. This may not mean much for technology progress, but if Elon’s reputation for being a genius is maintained by the Grok 3 effort it could push the SOTA significantly.

 

DeepSeek effort

Given that Grok 3 is being released post DeepSeek and OpenAI’s o1 & o3-mini models, it needs to have a reinforcement learning and a test time compute story. I would be surprised if they release a model post DeepSeek and be branded as behind the curve. It would be interesting to observe the approach to RL that Grok 3 takes – between the technologies used (GRPO etc.), their openness to showing reasoning traces, and their approach to Mixture of Experts. We have not seen much being published by the xAI team, and it will be interesting to see their openness in sharing their methods too.

 

Marketing

Anthropic has a quasi-academic flavor to its product releases. OpenAI is increasingly product focused and putting together a smorgasbord of capability demos (12 days of Shipmas is an example). DeepSeek and Meta have published a lot of details of their training and also have done open-source and open-weights to a reasonable extent. Grok 2 was a hurried launch. I am curious to see how the marketing for Grok 3 works – the capabilities they highlight, the methods they create buzz, the use cases they focus on etc. I want to see the vibe of the brand .. which has been subdued in Grok 2.

So those are my thoughts as we wait for Grok 3 to be unveiled. I will follow up post launch with my thoughts.