This QR code is a language model

1,984 weights, 444 characters of JavaScript, no server

QR code containing the whole model
Point your phone camera at it.

Or watch it run here, from the same link:

    So what language does it model? The names of AI models. It read 1,215 real ones, like gpt-4o-mini and claude-3-5-sonnet, and now it makes up its own. A few are better than the real thing.

    Nothing runs on a server. The QR code is a link, and everything after the # in that link is the model: 444 characters of JavaScript and 1,984 weights. Browsers don't send that part of a link to the server, so all q.shayon.work does is hand out a small page that reads the model from the address bar and runs it.

    Part 1 · The QR code

    Give it a start

    Type the beginning of a name and the model writes the rest.

    Or try

     

    That's all a language model does. It looks at what's written so far, gives every possible next character a probability, picks one, and goes again. The large models work the same way, with pieces of words instead of characters and billions of weights instead of 1,984.

    This one is tiny, but it's the real thing. It's the design Bengio and his colleagues described in 2003, in a paper called “A Neural Probabilistic Language Model”.

    How it works

    The link

    The link has four parts.

    The address is in capitals because QR codes have a compact mode for capital letters, and the scheme and host of a URL don't care about case. The three digits after the # tell the page where the program ends and the weights begin.

    Show the whole link, all 3,429 characters

    The weights

    The network was trained so that every weight is a whole number from −15 to 15. That's 31 possible values, so each weight can be one digit of a number written in base 31. Here it is with the first three weights in the link:

    Three weights−8−4−5
    Add 15 to each71110
    Read as one number in base 317 × 31² + 11 × 31 + 10 = 7,078

    Do that with all 1,984 weights and the number is 2,959 digits long:

    8998716153969059216126347890566834857996904478345096280655978142 …

    It's stored in decimal because digits are what QR codes store best: 10 bits for every 3 digits. My first version wrote one character per weight, and it needed a much bigger code.

    How a weight is writtenBits in the QR codeWhat fits
    One character, 61 levels8.01,689 weights in a version 36 code
    One big number, 61 levels6.01,689 weights in version 30
    One big number, 31 levels5.01,984 weights in version 30

    The last row is the one in the code above. Coarser weights, but more of them, made a better model. The first line of the program turns the number back into weights:

    for (n = BigInt(d), W = [], i = 1984; i--; )
      W[i] = Number(n - (n /= 31n) * 31n) - 15;

    d is the string of digits. Each step divides by 31. The remainder is one weight, and the rest of the number is kept for the next one.

    The network

    It writes one character at a time. To pick the next one, it looks at the last three characters it wrote and at how long the name is so far.

    The 39 characters are a to z, 0 to 9, dash, dot and an end marker. When it picks the end marker, the name is done. The length input is what stops it from writing names that go on forever.

    Here is the whole network, drawn from the link in the QR code. Every line is one weight.

    Blue lines push a unit up and orange lines push it down. Stronger lines are bigger weights. That's 1,682 of the 1,984. The rest are the 234 numbers that turn three characters into the dots at the top, and 68 biases.

    All of it lives in one flat list:

    WeightsCountWhat they are
    0–23339 × 6Six numbers for each character
    234–78429 × 19Hidden layer: 18 inputs and the length, for each unit
    785–81329Hidden biases
    814–194439 × 29Output layer
    1945–198339Output biases

    Where it runs

    On your phone, and nowhere else. A browser splits every link in two before it asks for a page:

    1. Your camera reads the QR code. All it sees is a link, so it opens it in your browser.
    2. The server gets a request for an empty address and sends back the same 11 KB page it sends for every link. It never sees the model.
    3. That page reads the rest of the link from the address bar, splits off the program and runs it. It also shows each name and asks for a new one when you tap, which saves room in the QR code.
    4. The model runs as plain JavaScript. A name takes about 30,000 multiplications, which is nothing for a phone. There's no GPU, no WebAssembly and no library.

    Enjoying this? Shameless plug

    Part 2 · Longer links

    If not a QR code, how about a link?

    A QR code holds about 3 KB, and that's the only reason the model above is so small. A link you click has no such limit. It can be as long as the browser will put up with, so I measured how long that is.

    1. The QR code3.5 KB
    2. A longer link64 KB
    3. Firefox gives up1 MB
    4. Chrome gives up2 MB

    That's room for a model 300 times bigger. So here is the same job at three link sizes. The job is writing children's stories, learned from TinyStories, a dataset of simple stories made for training small models. All three are running on your device right now, straight from their links.

    1. 3.5 KB

      1,895 weights · would fit in a QR code

      It has read a great many children's stories and come away knowing that they contain words.

      Waiting to start…
      Open this link
    2. 64 KB

      90,420 weights

      It starts sentences well and loses them halfway through.

      Waiting to start…
      Open this link
    3. 1 MB

      1,498,640 weights

      Its sentences mostly work. Its plots don't.

      Waiting to start…
      Open this link

    The first one is why the QR code writes model names and not stories. A name is a dozen characters with a few habits, like a dash, a version number and a size. That is about as much as 2,000 weights can hold.

    The last one is about as far as this goes. Its link is 1,010,424 characters, just under what Firefox will open.

    What's inside the bigger ones

    The 3.5 KB story model is the same kind of network as the one in the QR code, with 33 characters and 32 hidden units. This is its link, drawn the same way:

    1,632 lines, plus 198 numbers for the characters and 65 biases, makes 1,895 weights.

    The 64 KB and 1 MB models are transformers, the same kind of network as the large chat models, only small. The 64 KB one has 2 layers and the 1 MB one has 4. Each layer has 4 attention heads, and the model can look back over the last 128 characters. Every weight is stored in 4 bits, and the program that runs them is about 2 KB of JavaScript at the front of the link. The 1 MB model trained for a little over three hours on two CPU cores.

    64 KB

    Last 128 characters
    Embedding67 characters × 60 numbers4,020

    Layer 1 43,200

    Attention4 heads
    Feed-forward60 → 240 → 60

    Layer 2 43,200

    Attention4 heads
    Feed-forward60 → 240 → 60
    Outputreuses the embedding0
    67 probabilitiespick the next character

    90,420 weights

    1 MB

    Last 128 characters
    Embedding67 characters × 176 numbers11,792

    Layer 1 371,712

    Attention4 heads
    Feed-forward176 → 704 → 176

    Layer 2 371,712

    Attention4 heads
    Feed-forward176 → 704 → 176

    Layer 3 371,712

    Attention4 heads
    Feed-forward176 → 704 → 176

    Layer 4 371,712

    Attention4 heads
    Feed-forward176 → 704 → 176
    Outputreuses the embedding0
    67 probabilitiespick the next character

    1,498,640 weights

    The numbers on the right are weights. Each layer adds its result to what came in. There is no table of positions: each attention head has a fixed preference for nearby characters instead, and the output step reuses the embedding, so neither costs anything in the link.

    The model in the QR code is not a transformer. I tried. With 2,000 weights, attention has to pay for its own machinery before it predicts anything, and every transformer I trained at that size lost to the plain network from 2003. So did recurrent networks, and so did predicting pieces of words instead of characters. At 64 KB there's room for attention to earn its keep.

    Notes

    Comments on Hacker News