LLMs can produce natural writing and still fail at a strangely easy question, such as counting the rs in strawberry. This is less like a simple bug than a consequence of the model's structure.
Starting with Transformers
Since the 2017 paper “Attention is All You Need,” the transformer has become the basic architecture behind modern LLMs. Models such as ChatGPT, Claude, and LLaMA all follow this line of development. Transformers have expanded beyond text into images, audio, and translation, so it is difficult to understand the current AI boom without understanding them.
Where LLMs Commonly Wobble
LLMs often struggle to control text length. Ask for 200 words and they may return 150 or 250. They are also weak at exact string patterns: they can miscount the rs in “strawberry” or mishandle repeated patterns in Korean.
The Transformer Architecture
The core of a transformer is attention: looking at the relationships among words in a sentence at the same time.
The amount of computation grows quickly as the sequence gets longer.
Here, n is the sequence length and d is the dimension.
Self-Attention and Memory
Transformers use multi-head attention to examine word relationships from several perspectives. It performs well, but it also consumes significant computing resources.
Because each head is calculated separately, memory requirements also grow.
Chain-of-Thought Prompting
Chain-of-Thought (CoT) prompting can reduce some of these weaknesses. Instead of asking for an answer immediately, it asks the model to write intermediate steps.
A Practical CoT Example
Ordinary prompting and CoT prompting feel quite different.
Ordinary prompting:
Q: How many 'r's are in strawberry?
A: 2
CoT prompting:
Q: Count the 'r's in strawberry step by step.
A: Step 1: inspect each letter
s(no r) -> t(no r) -> r(first) -> a(no r) -> w(no r)
-> b(no r) -> e(no r) -> r(second) -> r(third) -> y(no r)
Total: 3
Hardware Requirements and Constraints
Modern transformer models need substantial computing power.
This is why high-performance hardware such as NVIDIA GPUs matters. Discussions about model performance eventually return to electricity, memory, and GPU prices.
Why Other Architectures Are Appearing
MOE (Mixture of Experts)
MOE combines several expert models and activates only the experts needed for an input. Instead of making one model handle every calculation, it selects a subset. The goal is to operate a larger model while reducing computation and memory.
g_i(x) is the gating function that selects experts, and f_i(x) is the output of each expert model.
Scaling Up
Hyperscale AI grows both the model and the computing system. A simplified scaling law might look like this:
N is the number of parameters, C is computing power, and alpha and beta are scaling coefficients.
The Mamba Architecture
Mamba uses selective state-space modeling. It is an attempt to address the cost of long contexts in a way different from transformers.
Mamba has linear time complexity, so it can process long sequences more efficiently. Its lower memory use is one reason it is discussed for long-context tasks.
Future Directions
AI needs to become not only smarter but also more efficient. Resource use relative to performance can be simplified as:
This keeps distributed processing, modular architectures, and resource optimization important. Hybrid architectures, biologically inspired models, and quantum-computing integration are also being studied in this direction.
A Midway Summary
Transformers are an important foundation for AI, but their limits are clear. That is why approaches such as MoE, Mamba, SSM, and Jamba keep appearing. The central goal is to process longer contexts at lower cost.
What to Watch Next
AI research will continue to examine compute efficiency, memory use, long-context processing, and hybrid structures. Simply making models larger makes the cost problem too large.
New Architectures
SSM (State Space Model)
SSMs are often discussed as an alternative to transformers. They process input while continuously updating a system state.
Their linear time complexity makes long sequences more efficient, with lower memory use than transformers.
Mamba
Mamba builds on SSMs with selective state updates.
The selective update improves compute efficiency and considers hardware optimization as well. It is especially promising for long contexts.
Jamba and Hybrid Structures
Jamba combines transformer and Mamba strengths. In simplified form:
The model has 120B active parameters and aims to combine throughput with long-sequence processing. It uses transformer parallelism together with Mamba's sequence efficiency.
Open Questions
Real-Time Optimization
Microsoft's decoder-based transformers are discussed in the context of real-time optimization. Latency can be simplified as:
Better optimization would let LLMs fit more naturally into real-time applications.
Multimodality
AI systems are moving beyond text toward images, audio, and interaction.
This is a move toward handling several kinds of sensory information together.
Compute Efficiency
The architectures also differ in computational complexity:
- Transformers are limited on long sequences by
O(n²)complexity. - SSMs and Mamba provide more efficient linear
O(n)processing. - Jamba aims to balance efficiency and performance with
O(n·log(n))complexity.
Closing
When I use an LLM, I often think, “How can it be this smart and still get that wrong?” Looking at the architecture makes the reason clearer: these models do not count and calculate exactly as people do.
Future models therefore need more than scale. They need to handle longer contexts, run with fewer resources, and connect to tools or external systems when necessary.
As I keep using and building with AI, I want to understand its limits as well as its strengths. That helps me avoid both overestimating AI and dismissing it as useless.
References
- Vaswani, A., et al. (2017). “Attention Is All You Need.” arXiv:1706.03762
- 안될공학. (2023). “Beyond Transformers: MoE, SSM, and the Future of AI.” YouTube
- Gu, A., et al. (2023). “Mamba: Linear-Time Sequence Modeling with Selective State Spaces.” arXiv:2312.00752
- Wei, J., et al. (2022). “Chain of Thought Prompting Elicits Reasoning in Large Language Models.” arXiv:2201.11903
- Brown, T., et al. (2020). “Language Models are Few-Shot Learners.” arXiv:2005.14165