LLMs can produce natural writing and still fail at a strangely easy question, such as counting the rs in strawberry. This is less like a simple bug than a consequence of the model's structure.
Starting with Transformers
Since the 2017 paper “Attention is All You Need,” the transformer has become the basic architecture behind modern LLMs. Models such as ChatGPT, Claude, and LLaMA all follow this line of development. Transformers have expanded beyond text into images, audio, and translation, so it is difficult to understand the current AI boom without understanding them.
Where LLMs Commonly Wobble
LLMs often struggle to control text length. Ask for 200 words and they may return 150 or 250. They are also weak at exact string patterns: they can miscount the rs in “strawberry” or mishandle repeated patterns in Korean.
The Transformer Architecture
The core of a transformer is attention: looking at the relationships among words in a sentence at the same time.
The amount of computation grows quickly as the sequence gets longer.
Here, n is the sequence length and d is the dimension.
Self-Attention and Memory
Transformers use multi-head attention to examine word relationships from several perspectives. It performs well, but it also consumes significant computing resources.
Because each head is calculated separately, memory requirements also grow.
Chain-of-Thought Prompting
Chain-of-Thought (CoT) prompting can reduce some of these weaknesses. Instead of asking for an answer immediately, it asks the model to write intermediate steps.
A Practical CoT Example
Ordinary prompting and CoT prompting feel quite different.
Ordinary prompting:
Q: How many 'r's are in strawberry?
A: 2
CoT prompting:
Q: Count the 'r's in strawberry step by step.
A: Step 1: inspect each letter
s(no r) -> t(no r) -> r(first) -> a(no r) -> w(no r)
-> b(no r) -> e(no r) -> r(second) -> r(third) -> y(no r)
Total: 3
Hardware Requirements and Constraints
Modern transformer models need substantial computing power.
This is why high-performance hardware such as NVIDIA GPUs matters. Discussions about model performance eventually return to electricity, memory, and GPU prices.
Why Other Architectures Are Appearing
Transformers are an important foundation, but their limits are clear. Attention cost climbs sharply as the sequence gets longer, so people keep trying to solve that a different way. MoE goes after raw computation; the SSM family goes after the cost that grows with sequence length.
MoE (Mixture of Experts) keeps several expert models around and activates only the ones an input needs, instead of making a single model do every calculation.
g_i(x) is the gating function that picks the experts and f_i(x) is each expert's output. The point is to run a larger model while cutting computation and memory.
There is also the hyperscale direction, growing the model and the computing system together. Simplified, performance can be read as a scaling law:
N is the parameter count, C is computing power, and alpha and beta are scaling coefficients. But the cost grows with it, which is why structural efficiency work like the SSM family below runs in parallel.
New Architectures
SSM (State Space Model)
SSMs come up often as an alternative to transformers. They process input while continuously updating a system state.
Their linear time complexity makes long sequences more efficient, and memory use drops below a transformer's, which helps on long inputs.
Mamba
Mamba builds on SSMs with a selective state-update mechanism.
The selective update improves compute efficiency and takes hardware optimisation into account too. It is said to be particularly good at long contexts.
Jamba and Hybrid Structures
Jamba is a hybrid that mixes the strengths of transformers and Mamba. Simplified:
It has 120B active parameters and goes after throughput and long-sequence handling at the same time, using transformer parallelism alongside Mamba's sequence efficiency.
Complexity, and What I Still Wonder About
The architectures above already split on computational complexity.
- Transformers are limited on long sequences by
O(n^2). - SSMs and Mamba process in linear
O(n), so they handle length more efficiently. - Jamba aims at
O(n log n), balancing efficiency against performance.
Two things I am still curious about. One is real-time processing. Latency, simplified, looks like this:
If that improves, LLMs fit more naturally into real-time applications. The other is multimodality — AI moving past text alone into images, audio and interaction:
Which is roughly the direction of how a person uses several senses at once. Making models bigger does not settle either problem on its own, so compute efficiency and architecture itself stay on the table.
Closing
When I use an LLM, I often think, “How can it be this smart and still get that wrong?” Looking at the architecture makes the reason clearer: these models do not count and calculate exactly as people do.
Future models therefore need more than scale. They need to handle longer contexts, run with fewer resources, and connect to tools or external systems when necessary.
As I keep using and building with AI, I want to understand its limits as well as its strengths. That helps me avoid both overestimating AI and dismissing it as useless.
References
- Vaswani, A., et al. (2017). “Attention Is All You Need.” arXiv:1706.03762
- 안될공학. (2023). “Beyond Transformers: MoE, SSM, and the Future of AI.” YouTube
- Gu, A., et al. (2023). “Mamba: Linear-Time Sequence Modeling with Selective State Spaces.” arXiv:2312.00752
- Wei, J., et al. (2022). “Chain of Thought Prompting Elicits Reasoning in Large Language Models.” arXiv:2201.11903
- Brown, T., et al. (2020). “Language Models are Few-Shot Learners.” arXiv:2005.14165