🪩 Welcome to the State of AI Report 2023

and

Oct 12, 2023

Hi all,

In its 6th (and possibly most dramatic) year, the State of AI Report 2023 distills what you *need* to know in AI research, industry, safety, and politics. This open-access report is our contribution to the AI ecosystem. Many thanks to Othmane Sebbouh, Corina Gurau, and Alex Chalmers for their incredible support producing it. I’m looking forward to seeing many of our loyal readers at our launch event in SF tonight. 😎

Below I'll share my director's cut - a snapshot of key themes and ideas that stood out to me.

We'd appreciate your help in spreading the report far and wide, thanks in advance! Any comments, critique, or suggestions, please hit reply :-)

Research

2023 was of course the year of the LLM, with the world being stunned by OpenAI’s GPT-4, which succeeded in beating every other LLM - both on classic AI benchmarks, but also on exams designed for humans.

We’re also seeing a move away from openness, amid safety and competition concerns. OpenAI published a very limited technical report for GPT-4, Google published little on PaLM2, Anthropic simply didn’t bother for Claude…or Claude 2.

However, Meta AI and others are keeping the open source flame burning by producing and releasing competitive open LLMs that are capable of matching many of GPT-3.5’s capabilities.

Judging by the leaderboards over at Hugging Face, open source is more vibrant than ever, with downloads and model submissions rocketing to record highs. Remarkably, in the last 30 days LLaMa models have been downloaded more than 32M times on Hugging Face.

While we have many different benchmarks (largely academic) to assess the performance of LLM systems, it often feels like the eval to rule all evals is one with the utmost scientific and engineering grounding: “vibes”

Beyond the excitement of the LLM vibesphere, researchers, including from Microsoft have been exploring the possibility of small language models, finding that models trained with highly specialized datasets can rival 50x larger competitors.

This work might become all the more urgent if the team over at Epoch AI are correct. They’ve predicted that we risk exhausting the stock of high-quality language data in the next *two years* - prompting labs to explore alternative sources of training data.

Zooming out to look at the state of research more generally - much is made of the US’ thinning lead, but the vast majority of highly cited papers are still coming from a small number of US-based institutions.

Industry

All of this work means it’s a good time to be in the hardware business, especially if you’re NVIDIA. GPU demand drove them into the $1T market cap club and their chips are used 19x more in AI research than *all the alternatives combined*.

While NVIDIA continues to ship new chips, their older GPUs exhibit remarkable lifetime value. The V100, released in 2017, was the most popular GPU in AI research papers in 2022. It might cease to be used in 5 years, which means it’ll have served 10 years.

We’re already seeing rapid demand for the NVIDIA H100, with labs rushing to build huge clusters - with many more likely on the way. However, we hear that these endeavors are not without significant engineering challenges to realize.

The “chip wars” are also forcing adaptation in the industry, with NVIDIA, Intel, and AMD all moving to create special, sanctions-compliant chips for their large Chinese customer bases.

In perhaps the least surprising news of all time, Chat-GPT is one of the fastest growing internet products ever. It’s a particular hit among developers, squeezing out Stack Overflow - the historic source of choice for devs looking for solutions to coding problems.

But according to data from Sequoia, there is reason to doubt the staying power of GenAI products for the moment - with shaky retention rates for everything from image generation through to AI partners.

Outside the world of consumer software, there are signs that GenAI could accelerate progress in the world of embodied AI. Wayve GAIA-1 displays impressive generalization and could act as a powerful tool for training and validating autonomous driving models.

Beyond GenAI, we’ve seen significant moves in industries that have previously struggled to find the right fit for AI. A number of traditional pharma companies have gone “all in” on AI, striking deals worth billions with the likes of Exscientia and InstaDeep.

The market for AI-first defense is roaring to life as the militaries rush to modernize capabilities in response to asymmetric warfare we see in Ukraine. However, the clash between new technology and old incumbents is making it hard for new entrants to get their foot in the door.

These successes aside, the weight of the venture industry is resting on the shoulders of GenAI, which is holding up the sky of the tech private markets like Atlas. Without the GenAI boom, AI investments would’ve crashed by 40% versus last year.

The authors of the landmark paper that introduced transformer-based neural nets are living proof of this - the transformer mafia have collectively raised billions of dollars in 2023 alone. We’ve updated our popular slides from last year :-)

The same is true of the DeepSpeech2 team at Baidu’s Silicon Valley AI Lab. Their work on deep learning for speech recognition showed us the scaling laws that now underpin large-scale AI. Much of the team went on to be founders or senior execs at leading ML companies.

Many of the most high-profile blockbuster fundraises weren’t led by traditional VC firms at all. 2023 was the year of corporate venture, with Big Tech putting its war chest to effective use.

Politics

Unsurprisingly, billions of dollars of investment and huge leaps forward in capabilities have placed AI at the top of policymakers’ agendas. The world is clustering around a handful of regulatory approaches - ranging from the light-touch through to the highly restrictive.

Potential proposals for global governance have been floated, with an alphabet soup of institutional acronyms being invoked as precedent. The UK’s AI Safety Summit, being organized by Matt Clifford and others may help start to crystallize some of this thinking.

These debates may gain more urgency as we continue to see the power of AI on the battlefield. The war in Ukraine has become a laboratory for AI warfare, demonstrating how even relatively improvised systems, when integrated cleverly, can be used to devastating effect.

Another potential flashpoint is the US presidential election. So far, deep fakes and other AI-generated content have been of patchy relevance versus old-school disinformation. Low-cost, high-quality models may change this, prompting pre-emptive action.

Past State of AI reports warned that safety was being neglected by the big labs. 2023 was the year of the x-risk debate, with the open vs. closed debate intensifying among researchers and the extinction risk making headlines.

…needless to say, not everyone agrees - with Yann LeCun and Marc Andreessen emerging as the skeptics-in-chief.

Unsurprisingly policymakers are alarmed and have been trying to build out their knowledge of potential risks directly. The UK has moved first to set up a dedicated Frontier AI Taskforce led by Ian Hogarth, and the US launched congressional investigations.

Amid all the theoretical debate, labs are already acting, with Google DeepMind and Anthropic being among the first to lay out their approach to mitigating extreme risk across development in deployment in greater detail.

Even without jumping far into the future, tough questions are being asked about techniques like reinforcement learning from human feedback, which have powered the likes of Chat-GPT.

Predictions

As ever, in the spirit of transparency, we graded last year’s predictions - we scored 5/9

✅ on LLM training, GenAI/audio, Big Tech going all in on AGI, alignment investment, and training data

❌ for multi-model research, biosafety lab regulation, and doom for semis start-ups

Here are our 10 predictions for the next 12 months! Covering:

- GenAI/film-making

- AI and elections

- Self-improving agents

- The return of IPOs

- $1 billion+ models

- Competition investigations

- Global governance

- Banks + GPUs

- Music

- Chip acquisitions

The State of AI Report is always a team effort, and we were one member short, with Ian Hogarth stepping back to focus on the UK’s Frontier AI taskforce. Many thanks to Othmane Sebbouh for a third year on the report, along with Corina Gurau and Alex Chalmers for their debut appearances.