Except Catanzaro, that is. At the time, he explained, he was collaborating with Andrew Ng at Stanford on a “little project where we replaced 1,000 servers at Google with three servers using GPUs and a bunch of CUDA kernels that the team wrote.” He was also talking during that period with the NYU AI Lab’s Yann LeCun (now head of AI research at Meta), and Rob Fergus (now a research scientist at DeepMind).
“Fergus was telling me, ‘It’s crazy how many machine learning researchers are spending time writing kernels for the GPU — you should really look into that,’” he said.
Cantanzaro did look into it. Customers were starting to buy large numbers of GPUs for deep learning. Eventually Huang and others at Nvidia took note too.
By 2014 Huang was fully on board with the AI mission. While his 2013 GTC keynote barely mentioned AI, it was suddenly front and center during his 2014 keynote. Machine learning is “one of the most exciting applications in high-performance computing today,” he said. “One of the areas that has seen exciting breakthroughs, enormous breakthroughs, magical breakthroughs, is an area called deep neural nets.”
Catanzaro pointed out that as founder of the firm, Huang had the authority to “turn the company on a dime, he just pounced on it,” realizing that “AI is the future of this company, and we’re gonna bet everything on it.”
The ImageNet moment of 2012 involved a few researchers and a GPU. But this was just the first milestone, according to Kari Briski, VP of product management, AI software at Nvidia.
The next challenge was how to make the power of GPUs scale: “We worked on software to make sure that the GPUs could communicate together, so we went from a single GPU to multi-GPU and multinode,” Briski said.
For the past seven years, Nvidia has focused on building deep learning software in libraries and frameworks that abstract the need to have to code in CUDA. It puts CUDA-accelerated libraries, like cuDNN, into more widely used Python-based libraries like PyTorch and TensorFlow.
“You can now scale to hundreds and thousands of GPUs all talking to each other to get that neural network,” said Briski. “It went from months of training to weeks — today it takes seconds to train that same neural network.”
In addition, by 2018 GPUs were used not just for AI training, but for inference, too — to support capabilities in speech recognition, natural language processing, recommender systems and image recognition. That meant not just more hardware, like Nvidia’s T4 chip, but more software to fuel these real-time inference workloads in the data center and in automotive applications, as well as in robots and drones.
As a result, Nvidia has become more of a software company than a hardware company, said Nvidia’s Das. The company hired more and more software engineers and researchers and built the research division to be the state of the art of AI.
“We started building up all these pieces of software, one use case after another,” he said. As standards and frameworks began to evolve, like TensorFlow and PyTorch for training, Nvidia optimized them for GPUs. “We became AI developers and really embraced the ecosystem,” he added.
At Nvidia’s 2022 GTC Analyst/Investor Conference, Huang made the company’s ongoing software and platform focus very clear, including a shout-out to the AI Enterprise Software Suite, which had launched the previous year.
“The important thing about our software is that it’s built on top of our platform,” he said. “It means that it activates all of Nvidia’s hardware chips and system platforms. And secondarily, the software that we do is industry-defining software. We’ve now finally produced a product that an enterprise can license. They’ve been asking for it … they can’t just go to open source, and download all the stuff, and make it work for their enterprise. No more than they could go to Linux, download open-source software, and run a multibillion-dollar company with it.”
The consequence of the platform approach is that anytime a customer buys from Nvidia, they’re not just buying the software, but buying into the Nvidia value chain.
That was another key pillar to Nvidia’s strategy, explained Gartner’s Dekate. With the GPU and CUDA, and Nvidia’s channel strategy, which surrounded customers with options and sourcing that customers are most familiar with, it created an ecosystem growth flywheel.
“Nvidia does not have to try and convince enterprise end users directly,” he said. “End users can use technologies they are familiar with but still turn the crank on Nvidia.”
Nvidia’s AI headwinds are gentle — for now
In the late 2010s, AI chip startups began making waves, from Graphcore and Cerebras to SambaNova.
Analyst Karl Freund recalled that at the time, the prevailing wisdom was that since the startups were designing chips specifically for AI, they would be better than Nvidia’s.
“That didn’t turn out to be the case,” he said. “Nvidia was able to innovate both in their hardware and software to keep their lead.”
That being said, their lead has diminished — Habana Labs, owned by Intel, had really good results on their Habana Gaudi2 chip, Freund pointed out, while Google’s TPU4 “looks really good and is competitive with the A100.”
But Nvidia has the H100 in the wings, which everyone is anxiously waiting to ship in production volumes. The new Hopper H100 chip uses a new architecture designed to be the engine for massively scalable AI infrastructure. It includes a new component called the Transformer Engine that’s specifically optimized for training and inference of the transformer layer, which is the building block of GPT (ChatGPT, for example, is a generative pretrained Transformer).
In addition, even if CUDA’s current competitive moat is challenged, Nvidia is also replacing it with its latest higher-level, use-case specific software — such as AI for healthcare and AI for digital twins/omniverse.
Finally, even if all the competitive trends started materializing in the second half of this year, they would not have a material impact on Nvidia revenues until 2024. Even then, Freund estimated that all competitors combined could get just 10% of the market.
Still, Gartner’s Dekate insists that Nvidia no longer has the clean playing field they once had, that allowed them to dominate the marketplace. That development includes an increased number of customer options, which allows end users to, at the very least, drive pricing advantage in their favor.
Also, with some Chinese vendors having to do without access to Nvidia GPUs, they will try to accelerate competitive technology, he predicted.
Nvidia’s Briski brushes off concerns. “We’ve had headwinds before,” she said. “I think that we’re always challenged to be on our toes, to never feel sort of comfortable.”
In any case, Huang has maintained that it’s tough for competitors to come into the AI market and create a solution that works, isn’t too complicated, and uses the software developers want.
Nvidia’s Huang seen as AI visionary — who keeps the pressure on
Over the years, Nvidia CEO Jensen Huang, well-known for his leather jacket atop a black silicon valley uniform, has been described as everything from “flamboyant” and a “superstar” to, occasionally, a jokester and “the next Steve Jobs.”
But, inevitably, any discussion about Nvidia’s AI success turns to Huang.
Nvidia’s Das joined the company from Microsoft in 2019. He says he had many conversations with Huang over a period of nine months before accepting the position.
“I joined Nvidia to work for Jensen because he just blew my mind in all these conversations … that there could be a person like that, who could think like that, who can actually operate like that,” Das said.
Analyst Freund emphasized that Huang “is an amazing driver of that company” and added that Nvidia does not have a lot of organizational layers, because Huang doesn’t like to have a lot of layers between him and the people doing the engineering work and the science work.
That said, Huang is also demanding, he added. “When I worked at AMD, a lot of my graphics engineers were renegades from Nvidia,” said Freund. “They left Nvidia because they couldn’t handle the pressure.”
Nvidia’s generative AI opportunity
Catanzaro left Nvidia in 2014 to work at Baidu with Andrew Ng, but returned to Nvidia in 2016 to head a new lab focused on applied deep learning research. At the time, he was the only member. Seven years later, he leads a team of 40 researchers.
Nvidia’s accelerated computing business, he said, requires his team to conceive of each entire application as an optimized whole.
“We don’t outsource any of that responsibility,” he said. Nvidia, he explained, tackles entire problems from top to bottom, from chips to applications, algorithms, libraries, compiler framework and interconnected data center architecture. Its researchers have the freedom to really push acceleration “far beyond what we could do if we limited ourselves to thinking about just one part of that stack.”
And with the new era of ChatGPT, Nvidia can push even further, he added.
“Companies are going to be really pushing on the application of AI to lots of different problems,” he said. “That, of course, makes my job even more exciting — I feel like applied research is the hottest place to be right now.”
Huang, too, has weighed in on the transformative moment of ChatGPT. “This is the iPhone moment of artificial intelligence,” he said at a recent Q&A at Berkeley’s Haas School of Business. “This is the time when all those ideas within mobile computing and all that, it all came together in a product that everyone kinda [says], I see it, I see it.”
Nvidia is well-prepared for the generative AI opportunity, said Das. “For those who have been working on it for years, we’ve sort of anticipated this,” he said. “We’ve been working on training large language models and we know what they’re capable of.”
Nvidia is in an AI sweet spot
Since AlexNet in 2012, Nvidia’s AI journey has always been about taking advantage of opportunities that opened up — even if, in the case of GPUs, it was unexpected.
So with the 2023 GTC just a month away — which will include more than 65 sessions focused on generative AI — Nvidia is undoubtedly in a sweet spot. Just as the company’s GPU was at the center of powering the deep learning revolution of a decade ago, Nvidia’s hardware and software are running behind the scenes of today’s GPU-hungry, hyped-up generative AI technology.
And it seems like no matter which companies come out on top — Google? Microsoft? OpenAI? — Nvidia, who supplies them all, will win big.
