Since November 2022, the ChatGPT conversational agent, developed by the American company OpenAI (**https://openai.com*), has been on everyone's lips and in every newspaper because its output is so impressive (though not always in mathematics). GPT stands for "generative pre-trained transformer" (generative pre-trained transformer in English); in other words, it is an artificial intelligence (AI), and more specifically a neural network—a highly sophisticated mathematical function whose parameters have been estimated from a vast quantity of data and which has been trained to generate text. GPT-3 is a large language model*: the computing power and data available today make it possible to create ever-larger neural networks. For example, GPT-3 has 175 billion parameters, whereas its "ancestor," GPT-2, released in 2019, had "only" 1.5 billion.
Everything is a vector... -----------------
Text is modeled through "vectorization": a text is treated as a sequence of words, and each such sequence is represented by a vector in the space spanned by the available vocabulary. The GPT algorithm "simply" predicts word n+1 based on words 1 through n. This could be done using very conventional models, such as Markov chains, which model transitions from one state to another (see Tangente SUP 69, 2013). But GPT can generate very high-quality content by enhancing these elementary algorithms with a few techniques. First, the sheer quantity of data used allows neural networks to be trained more effectively: GPT-3 was trained on 570 GB of text and has 175 billion parameters. Second, ChatGPT incorporates reinforcement learning: the texts it produces are assessed by humans, who identify the "best" ones, allowing the parameters that produced those texts to be favored. Finally, ChatGPT uses a particular neural-network architecture: transformers.
A transformer consists of several layers, each made up of attention heads that can take previous words into account and learn to determine their "relevance" by calculating their importance in relation to the word being generated (see box).
Thus, when the model must complete "A robot must protect its \_\_", it will find that "its" is statistically more closely related to "robot" in the sentence than to "A".