The scientific innovation in natural language processing (NLP) is at its fastest pace to date, driven by advances in large language models (LLMs). LLMs such as Generative Pretrained Transformer (GPT) models power multipurpose chatbots, search engines and coding assistants, unleashing a new era of automation. In this lecture, I will attempt to give you a sense of how these models work and the challenges they face. There will be a particular focus on how and to what extent LLMs learn about language. I will show that they can retain remarkable capabilities even when training them under extreme settings, i.e. to perform tasks that might be completely incomprehensible to or impossible for humans.