Tokenizers: How large language models see the world

This content introduces tokenization, a fundamental process in large language models (LLMs) where human text is broken down into numerical representations called tokens. It explains how LLMs convert sentences into these subword units, detailing the steps involved such as normalization and segmentation, and highlighting how controlling vocabulary size and avoiding risks are crucial for effective language understanding and generation by AI.

This page is free — overview and chapter list only.
The complete book body is sold separately.

Full book: $2.99 USDC via x402 ·
Per chapter: $0.25 USDC

Buy / open complete book (HTML)
· Complete book Markdown (.md)

Agents: start with free /library/discovery.json, sample free teaser chapters,
then pay for individual chapters or the complete book URL above.
Append .md to any content URL for Markdown with YAML front matter.

Chapters

  1. Tokenizers How Large Language Models See The World (Free teaser)
  2. 224 The Risks Of Tokenization ($0.25)
  3. 23 Tokenization And Llm Capabilities ($0.25)
  4. 231 Llms Are Bad At Word Games ($0.25)
  5. 24 Check Your Understanding ($0.25)
  6. 25 Tokenization In Context ($0.25)
  7. Summary ($0.25)
  8. This Chapter Covers ($0.25)
  9. 31 The Transformer Model ($0.25)
  10. 311 Layers Of The Transformer Model ($0.25)
  11. 32 Exploring The Transformer Architecture In Detail ($0.25)
  12. 321 Embedding Layers ($0.25)
  13. The Curse Of Dimensionality ($0.25)
  14. 322 Transformer Layers ($0.25)
  15. 33 The Tradeoff Between Creativity And Topical Responses ($0.25)
  16. 34 Transformers In Context ($0.25)
  17. Summary 2 ($0.25)
  18. This Chapter Covers 2 ($0.25)
  19. 41 Gradient Descent ($0.25)
  20. 412 What Is Gradient Descent ($0.25)
  21. 42 Llms Learn To Mimic Human Text ($0.25)
  22. 421 Llm Reward Functions ($0.25)
  23. 43 Llms And Novel Tasks ($0.25)
  24. 44 If Llms Cannot Extrapolate Well Can I Use Them ($0.25)
  25. 45 Is Bigger Better ($0.25)
  26. Summary 3 ($0.25)
  27. How Do We Constrain The Behavior Of Llms ($0.25)
  28. 51 Why Do We Want To Constrain Behavior ($0.25)
  29. 52 Fine Tuning The Primary Method Of Changing Behavior ($0.25)
  30. 53 The Mechanics Of Rlhf ($0.25)
  31. 55 Integrating Llms Into Larger Workflows ($0.25)
  32. Summary 4 ($0.25)
  33. Beyond Natural Language Processing ($0.25)
  34. 61 Llms For Software Development ($0.25)
  35. 621 Sanitized Input ($0.25)
  36. Designing Solutions With Large Language Models ($0.25)
  37. Ethics Of Building And Using Llms ($0.25)
  38. 91 Why Did We Build Llms At All ($0.25)
  39. 92 Do Llms Pose An Existential Risk ($0.25)
  40. 94 Ethical Concerns With Llm Outputs ($0.25)
  41. 95 Other Explorations In Llm Ethics ($0.25)
  42. Chapter 2 ($0.25)
  43. Chapter 4 ($0.25)
  44. Chapter 5 ($0.25)
  45. Chapter 7 ($0.25)
  46. Chapter 8 ($0.25)
  47. Chapter 9 ($0.25)
  48. How Large Language Models Work ($0.25)