AI 究竟 是如何工作的

Today's language models are transformers. Text becomes a sequence of token IDs; each ID is looked up in an embedding table to get a vector of a few thousand numbers (the model's width, d_model). That sequence of vectors then flows through a stack of identical blocks — 30 to 120 of them — and each block does two things:

• Self-attention: every position looks back at earlier positions and pulls in the information it needs (which noun does “it” refer to, which variable was declared above).
• A feed-forward network (MLP): a wide two-layer network applied to each position separately, where much of the model's stored knowledge lives.

Residual connections add each block's output back onto its input, and normalisation keeps the numbers in range. At the end, an unembedding matrix turns the final vector into one score (a logit) per vocabulary entry. Parameters are just the entries of all these matrices. For a dense model, a good estimate is:

params ≈ 12 × layers × d_model²   (+ vocab × d_model for embeddings)
# e.g. 32 layers, d_model 4096  →  12 × 32 × 4096² ≈ 6.4B  (+0.5B embeddings) ≈ "7B"

Nothing in that list is a rule about language. All of the behaviour comes from the values the parameters take during training.

The model outputs a logit zᵢ for every token in its vocabulary. Softmax turns logits into probabilities, and temperature T divides them first:

p(i) = exp(zᵢ / T) / Σⱼ exp(zⱼ / T)
T → 0   : all probability on the top token (greedy decoding)
T = 1   : the model's own distribution
T > 1   : flatter — rare tokens get picked more often

Samplers then trim the tail before picking: top-k keeps the k most likely tokens, top-p (nucleus) keeps the smallest set whose probabilities sum to p, and min-p drops tokens below a fraction of the top token's probability. Repetition and frequency penalties lower the logits of tokens already used. A fixed seed makes sampling repeatable on the same hardware and software.

The generation loop

tokens = tokenize(prompt)
kv = prefill(tokens)                 # one parallel pass over the whole prompt
while True:
    logits = forward(last_token, kv)  # one step, reusing the cached keys/values
    next = sample(logits, T, top_p)
    if next == END: break
    tokens.append(next); stream(next)

Prefill is a big parallel matrix multiply and is limited by compute; each decode step is tiny and is limited by how fast weights can be read from memory. That split explains most of what you'll see about speed later in this guide.

Most tokenizers use byte-pair encoding (BPE) or a close relative. Training starts from raw bytes and repeatedly merges the most frequent adjacent pair into a new token, until the vocabulary reaches its target size. Frequent strings (“ the”, “function”, four spaces of indentation) end up as single tokens; rare strings fall back to smaller pieces, and ultimately to individual bytes, so any text can be encoded.

• Vocabulary sizes range from about 32,000 (older Llama, Mistral) to 150,000–260,000 (Qwen, Gemma, GPT-4o-era tokenizers). Bigger vocabularies mean fewer tokens per text but a bigger embedding table.
• Languages are not treated equally. English averages about 4 characters per token; many other languages and scripts need noticeably more tokens for the same meaning, so they cost more and fill the context faster. Newer multilingual tokenizers have narrowed the gap.
• Special tokens mark structure the user never types: start and end of a message, role markers such as <|im_start|>assistant, tool-call delimiters, image placeholders.
• Numbers are tokenized oddly (“1234” might be “12” + “34”), which is one reason models used to struggle with arithmetic; many tokenizers now split digits individually.

In attention, every token computes a query, a key and a value. Each query is compared with every earlier key, and the resulting weights mix the values:

Attention(Q, K, V) = softmax(Q·Kᵀ / √d) · V
cost of prefill  ∝ n²   (every token vs. every earlier token)
memory per token = 2 × layers × kv_heads × head_dim × bytes   ← the KV cache

During generation the keys and values of past tokens are kept in the KV cache, so each new token only computes its own. That cache grows linearly with context and is often what limits long contexts — far more than the weights. Designs that shrink it: grouped-query attention (many query heads share a few KV heads), multi-head latent attention (DeepSeek; stores a compressed latent instead of full keys and values), sliding-window layers that only look back a few thousand tokens, and hybrids that mix in linear-attention or state-space layers with constant-size memory.

Positions are encoded with RoPE (rotary embeddings), which rotate queries and keys by an angle that depends on position. Context windows are extended after training by stretching those angles (position interpolation, NTK scaling, YaRN) and fine-tuning briefly on long documents. Advertised and effective context differ: models retrieve facts from the start and end of a long prompt more reliably than from the middle, so put the most important material at the edges.

A conversation is serialised into one token sequence by the model's chat template. The system prompt is simply the first message, wrapped in role markers:

<|im_start|>system
You are a careful coding assistant. Tools: read_file, run_tests …<|im_end|>
<|im_start|>user
How do I reverse a list in Python?<|im_end|>
<|im_start|>assistant

Models are trained to give the system role more authority than user or tool messages (an instruction hierarchy), but it's still just text in the same window. That's the root of prompt injection: instructions hidden in a web page, file or tool result can try to override the real ones. Agents defend with permissions, sandboxes and by treating tool output as data.

Structured output goes further than asking nicely for JSON. With constrained decoding, the runtime compiles a JSON Schema or grammar into a state machine and masks the logits of any token that would break it, so the output is guaranteed to parse. llama.cpp calls these GBNF grammars; hosted APIs expose them as “structured outputs” or “JSON mode”.

What's cached is the KV cache of a prompt prefix — the keys and values computed during prefill. Serving engines split the cache into fixed blocks (vLLM's PagedAttention uses 16-token pages), hash each block together with everything before it, and reuse any block whose hash matches a new request. A change at position k changes the hash of every block after k, which is why one edited character near the start invalidates the rest.

cost = cached_tokens × cache_read_price
     + new_tokens    × input_price
     + written_tokens × cache_write_price   (some providers)
     + output_tokens × output_price

Some APIs cache automatically; others ask you to mark a breakpoint after the stable part (system prompt, tool definitions, documents). Practical rules: keep the stable material first and byte-identical (no timestamps in the system prompt), append rather than edit conversation history, and keep tool definitions in a fixed order. Entries are evicted after minutes of inactivity, or held longer for a fee. Running locally, llama.cpp and Ollama keep the last prompt's KV cache in memory, which is why a follow-up question starts answering almost instantly.

Reasoning models are trained with reinforcement learning on verifiable tasks: maths with checkable answers, code with unit tests, puzzles with known solutions. The model generates long chains of thought; chains that end in a correct answer are rewarded. Over training, behaviours that help — breaking problems down, checking work, backtracking — become more frequent, and the chains get longer.

This adds a second scaling axis: besides a bigger model, you can spend more test-time compute. Accuracy on hard problems rises roughly with the logarithm of the thinking budget, up to a point. Effort settings (low/medium/high) or explicit budget tokens cap that spend.

• Thinking tokens are output tokens: billed as output and generated at decode speed, so they add latency as well as cost.
• Many APIs return a summary or an encrypted copy of the reasoning instead of the raw chain; agents pass it back on the next turn so the model keeps its train of thought between tool calls (interleaved thinking).
• More thinking isn't always better: simple lookups get slower and occasionally worse when a model over-thinks.

A tool is described to the model as a name, a description and a JSON Schema for its arguments. The model replies with a structured call instead of text; the client executes it and sends the result back as a tool message.

{"name": "run_tests",
 "description": "Run the project's test suite and return failures",
 "input_schema": {"type": "object",
   "properties": {"path": {"type": "string"}}, "required": []}}

messages = [system, user]
while True:
    reply = model(messages, tools)
    if not reply.tool_calls: return reply.text
    for call in reply.tool_calls:            # may run several in parallel
        result = execute(call)               # sandbox + permission checks here
        messages += [reply, tool_result(call.id, result)]

What separates good agents from bad ones is mostly around this loop: precise tool descriptions, small and composable tools, truncating huge outputs, asking permission for risky actions, and context management — summarising or dropping old tool results so long sessions don't overflow the window. tool_choice can force a specific tool or forbid tools; constrained decoding keeps the arguments valid JSON.

MCP is JSON-RPC 2.0 between a client (inside the agent or IDE) and a server. Local servers run as a subprocess and talk over stdio; remote ones use Streamable HTTP, with OAuth 2.1 for login. The session starts with an initialize handshake in which both sides declare their capabilities.

→ {"jsonrpc":"2.0","id":1,"method":"tools/list"}
← {"jsonrpc":"2.0","id":1,"result":{"tools":[{"name":"create_issue",
     "description":"Open a GitHub issue","inputSchema":{…}}]}}
→ {"jsonrpc":"2.0","id":2,"method":"tools/call",
   "params":{"name":"create_issue","arguments":{"title":"Fix login"}}}

Servers offer three primitives: tools (actions the model can call), resources (data the app can read into context, addressed by URI) and prompts (reusable templates the user can invoke). Clients can offer sampling (the server asks the host's model to generate text) and elicitation (the server asks the user a question). Every tool's schema is sent to the model, so dozens of servers can eat thousands of tokens of context — one reason agents now load tool definitions lazily.

A UTCP manual lists tools, each with JSON Schema inputs and outputs and a tool_call_template saying how to reach it. Templates exist for HTTP, server-sent events, streamable HTTP, the command line, text files and MCP itself. The client substitutes the arguments into the template and calls the service directly, with the service's own authentication (API key, OAuth or basic auth, declared in the template and filled from local variables).

agent ──reads manual once──▶  /utcp  (JSON)
agent ──GET /weather?location=Oslo──▶  api.example.com   # no proxy in between

The trade-off versus MCP: no extra server process and no extra hop, and existing APIs work as they are; but the client must implement every transport, and features MCP standardises — resources, prompts, sampling, server-initiated notifications — are out of scope.

An embedding model maps text to a vector, typically 384 to 4,096 dimensions. Similarity is usually cosine similarity:

cos(a, b) = (a · b) / (‖a‖ ‖b‖)      # 1 = same direction, 0 = unrelated

Search over millions of vectors uses approximate nearest-neighbour indexes such as HNSW graphs or IVF clusters, in a vector database or a library like FAISS. A production RAG pipeline usually adds:

• Chunking: split documents into passages of a few hundred tokens, ideally along headings or function boundaries, with some overlap.
• Hybrid search: combine vectors with keyword search (BM25), which catches exact names, error codes and identifiers that embeddings blur.
• Reranking: a cross-encoder scores the top 50–100 candidates against the question and keeps the best handful.
• Grounded prompting: put the passages in the context with their sources and ask the model to cite them.

Coding agents often skip vectors entirely and search with grep-style tools in a loop — exact, always up to date, and good enough when the model can iterate.

Pre-training rewards predicting plausible text, not saying “I don't know”; benchmarks that grade only right or wrong reward guessing over abstaining. So a model that hasn't stored a fact still produces the most fact-shaped continuation. Rare facts — seen a handful of times in training — are the most at risk, which is why niche libraries and recent APIs suffer most.

• Calibration measures whether stated confidence matches accuracy. Some evaluations now score abstentions separately and penalise confident errors.
• Grounding (RAG, search, reading the actual source file) moves the answer from memory to the context, where models are much more reliable.
• Verification loops — run the code, run the tests, type-check, call the real API — catch invented functions immediately.
• Package hallucinations are a security problem: attackers register package names that models tend to invent (“slopsquatting”). Check that a dependency exists and is the one you meant before installing it.

Pre-training minimises cross-entropy loss: the negative log-probability the model gave to each actual next token, averaged over trillions of tokens. Gradients from that loss nudge every parameter, batch after batch.

loss = − (1/N) Σ log p(tokenₜ | tokens<ₜ)
training compute ≈ 6 × parameters × training tokens   (FLOPs)
# 70B params × 15T tokens ≈ 6.3 × 10²⁴ FLOPs

Scaling laws describe how loss falls predictably with more parameters, data and compute; modern models are trained on far more tokens per parameter than the old “compute-optimal” ratio of about 20, because a smaller model that's been trained longer is cheaper to serve.

Post-training turns the predictor into an assistant: supervised fine-tuning on curated conversations; preference optimisation (RLHF with a reward model, or DPO directly on pairs of better/worse answers); and RL with verifiable rewards for maths, code and agent tasks, run in sandboxes with real tools. Distillation trains a smaller model on a larger one's outputs, which is how many fast “mini” and “flash” models are made.

In an MoE layer the feed-forward network is replaced by E experts and a router. For each token the router computes a score per expert, keeps the top k and mixes their outputs by those scores:

g = softmax(W_router · x)          # one weight per expert
y = Σ_{i ∈ topk(g)} gᵢ · Expertᵢ(x)
active params ≈ shared (attention, embeddings, shared experts) + k/E × expert params

Modern MoEs use many small experts (128–384) with 4–10 active, often plus one always-on shared expert. Training adds a load-balancing term (or a bias adjustment) so tokens spread evenly; otherwise a few experts take all the work. Serving at scale uses expert parallelism — experts spread over many GPUs, tokens routed between them. At home the useful fact is the asymmetry: memory scales with total parameters, speed with active ones, and the rarely-used experts can sit in system RAM while attention runs on the GPU.

Quantization maps each weight to a small integer and stores a scale (and sometimes an offset) per block of weights:

q = round(w / scale)           # e.g. a 4-bit integer in [-8, 7]
w ≈ q × scale                  # what the GPU reconstructs on the fly
bits per weight = bits + (scale bits / block size)
# Q4_K: blocks of 32 inside super-blocks of 256, 6-bit scales → ~4.5 bpw;
# Q4_K_M keeps some tensors at Q6_K → ~4.85 bpw on average

• Post-training quantization methods (GPTQ, AWQ, llama.cpp's importance matrix) use a little calibration text to decide where precision matters most and round to minimise output error, not weight error.
• Hardware formats: FP8 (E4M3) is native on recent NVIDIA, AMD and Intel accelerators; MXFP4 and NVFP4 are 4-bit floats with shared block scales supported by the newest GPUs, so 4-bit can run at full speed instead of being unpacked.
• Quantization-aware training (QAT) simulates low precision during training, so the released 4-bit model loses almost nothing.
• Quality loss is measured with perplexity or, better, the KL divergence between the quantized and full-precision output distributions, plus task benchmarks.

“Open weights” isn't automatically “open source”. Licences range from permissive (Apache 2.0, MIT) to custom ones with usage policies, attribution rules or limits for very large companies. The Open Source Initiative's definition of open-source AI also asks for enough information about the training data to reproduce the model, which few releases provide.

Weights are distributed as safetensors files (a simple, memory-mappable format that can't execute code, unlike old pickle checkpoints) with a config.json describing the architecture and a tokenizer. Community conversions add GGUF (llama.cpp) and MLX (Apple).

Open weights can be fine-tuned. LoRA freezes the model and trains a low-rank update for selected matrices — W' = W + B·A, with A and B thin — so only a fraction of a percent of the parameters train; QLoRA does this on a 4-bit base model, which puts fine-tuning a 30B model within reach of a single 24 GB GPU.

memory  ≈ params × bpw / 8 + KV cache + ~0.8 GB
KV      = 2 × layers × kv_heads × head_dim × 2 bytes × context
tok/s   ≈ 0.65 × bandwidth / (active_params × bpw / 8)

RTX 4090 (1008 GB/s), 32B dense @ Q4_K_M (19.4 GB):  ≈ 34 tok/s
Apple M4 Max (546 GB/s), 30B-A3B MoE @ Q4 (1.8 GB/token): ≈ 100+ tok/s
Dual-channel DDR5 (~80 GB/s), 8B @ Q4 (4.9 GB): ≈ 10 tok/s

Offloading splits the bytes read per token between fast VRAM and slow system RAM; the slow part dominates. Mixture-of-experts models offload best: keep attention, embeddings and shared experts on the GPU (they're read every token), and put routed experts in RAM (each is read only when chosen). The Run AI at home page applies these formulas to every open model with its real attention shape.

Most multimodal LLMs bolt a vision encoder (a Vision Transformer) onto the language model. The image is resized, cut into patches (often 14×14 or 16×16 pixels), each patch becomes a vector, and a small projector maps those vectors into the language model's embedding space, where they sit in the sequence like word tokens. Many models merge neighbouring patches 2×2 to save tokens:

image tokens ≈ (width / 28) × (height / 28)    # 14-px patches, 2×2 merge
1024 × 768  →  ~37 × 27 ≈ 1,000 tokens

Providers cap the resolution or tile large images, so counts differ by API. Audio is handled the same way — a speech encoder turns audio frames into embeddings — and video as a sequence of sampled frames. “Natively multimodal” models are pre-trained on mixed text and images from the start rather than having vision added later. Generating images is usually a separate system (diffusion, or autoregressive image tokens).

A chip is limited either by arithmetic (FLOPs per second) or by how fast it can move data (bytes per second). The ratio of work to data — arithmetic intensity — decides which:

decode, batch 1 : ~2 FLOPs per weight byte read  → memory-bound
prefill, 2k tok : ~4,000 FLOPs per weight byte    → compute-bound
time to first token ≈ queueing + prefill(prompt) + network

That's why serving providers batch many users together: reading the weights once and applying them to 64 requests makes decode far more efficient, at a small latency cost per user. Speculative decoding exploits the same slack for one user: a cheap draft (a small model, or extra prediction heads) proposes several tokens, the big model verifies them all in one parallel pass and keeps the prefix it agrees with — identical output, often 2–3× faster when drafts are accepted. Specialised chips (Cerebras, Groq, SambaNova) keep weights in on-chip SRAM with enormous bandwidth, which is how they reach thousands of tokens per second.

• Contamination: if test questions leaked into training data, scores measure memory. Defences include held-out private test sets, fresh problems dated after a model's cutoff (LiveCodeBench-style) and checking whether a model can complete a question it shouldn't know.
• pass@k for code is the probability that at least one of k samples passes the tests, estimated without bias from n ≥ k samples with c correct: pass@k = 1 − C(n−c, k) / C(n, k). Agentic benchmarks such as SWE-bench Verified give the model a real repository and issue, then run the project's hidden tests on its patch.
• Arena ratings come from people comparing two anonymous answers; a Bradley–Terry model turns millions of votes into Elo-like scores. They measure preference — including style and length — not correctness.
• Watch the error bars: on a 500-question test, ±2 points is often noise. And benchmarks saturate — once top models score above 90%, differences stop meaning much and a harder test replaces it.