This piece first appeared in German. The English version is a rewrite rather than a line-by-line translation, and the German original stays online in the archive: Unterschiedliche KI-Architekturen: LLMs, CNNs und Reinforcement Learning im Vergleich.

Self-attention in LLMs lets every token in a context weigh every other token at once. A convolution looks at a small patch of pixels at a time and slides across the image. Those two mechanisms are why one system writes a competent summary and another spots a tumor, and why neither will do the other's job.

Artificial intelligence gets perceived as one technology. In practice it rests on fundamentally different architectures, each optimized for a clearly bounded class of problem. Whether a system analyzes text, recognizes images or makes autonomous decisions depends on the structural design of the model and not on any general intelligence.

There is no single AI because there is no single kind of data

Every architecture is a technical answer to a particular kind of data and task. Language is sequential, symbolic and context-dependent. Images are spatially structured and locally correlated. Decisions are time-dependent, goal-oriented and driven by feedback.

Modern AI therefore differs less in computing power than in how it models the world, and progress comes from specialized architectures combined deliberately, never from one universal model.

Large language models structure symbols

Large language models are neural networks built on the transformer architecture, and their training objective is to predict the next token in a sequence from the context so far.

What is a transformer? An architecture that processes sequences by relating all elements of a context to each other simultaneously through self-attention, instead of passing information along step by step. That lets it model global dependencies and complex structures in language or code efficiently, and it scales well.

A model of that kind represents no meanings. It represents a probability distribution over sequences of symbols, and what looks like understanding is the result of statistically learned structure in language.

What is a token? The discrete unit a text gets broken into before processing. Depending on the tokenizer that can be whole words, parts of words, punctuation or individual characters. A language model works with token identifiers and learns to predict the likely next one from the context.

Self-attention is the central mechanism, and it produces three things. A global view of context in place of local dependencies, flexible prioritization of the relevant information, and emergent structures such as syntax, argumentative logic and semantic relations. That makes these models particularly capable at tasks requiring complex symbolic order.

The typical uses follow from it. Text analysis, summarization and generation. Code analysis, documentation and refactoring. Research workflows and knowledge management. Structuring unstructured text. And language-based control of other AI modules, which is why in complex systems a language model frequently acts as the coordinating or planning layer.

Convolutional networks perceive

Convolutional neural networks were developed for spatially structured data. They apply local filters systematically across the input image, where a language model would reach for global relationships. That produces a hierarchy of representations: early layers detect edges and textures, middle layers capture shapes and contours, deep layers identify object structures.

The fit to images is not accidental. These networks exploit explicit properties of visual data, meaning local spatial dependency, parameter sharing across the whole image, and tolerance for shifts and changes of scale (can be achieved by clever design). The result is a system that is data-efficient, stable against variation in the image, and highly precise in visual classification.

A convolutional network understands no image semantically. It recognizes visually stable statistical patterns, which is a different achievement and frequently the one that matters.

Typical uses run through computer vision and object recognition, medical image diagnostics, video and motion analysis, industrial quality control, and perception in autonomous systems. These networks often form the perception layer of a larger architecture.

Reinforcement learning decides

Reinforcement learning differs fundamentally from both. It works with interaction and not with a fixed dataset: an agent makes decisions in an environment and receives rewards or penalties for them.

Five elements carry it. An agent that acts, an environment that reacts, an action space, a reward function, and a policy that decides.

What makes the approach powerful is that it optimizes sequential decisions under uncertainty. The model learns which actions are negative in the short term and positive over a longer horizon, when exploration is necessary, and when a known strategy should simply be exploited. That suits dynamic, non-deterministic systems particularly well.

Typical uses include robotics and autonomous control, game playing, logistics and route optimization, energy management, and the simulation of complex decision processes. In an economic context it appears most often in simulation environments, used to test strategies under controlled assumptions.

The differences that matter

Language models work on symbolic sequences, recognize and produce structure in language, and are strong in analysis, planning and knowledge aggregation.

Convolutional networks work on spatial data, recognize visual patterns and objects, and form the perception layer of modern systems.

Reinforcement learning works on decisions across time, optimizes actions against feedback, and suits control and strategy development.

None of the three is universally applicable. Their strength comes from the fit between the structure of the problem and the assumptions built into the model.

Current systems combine all three

Convolutional networks supply perception, language models structure information and knowledge, and reinforcement learning makes the decisions. Those combinations are the basis for autonomous systems, multimodal assistants and agent-based architectures.

For a company, an analyst or a decision-maker, three consequences follow. AI performance is architecture-dependent. Misapplication comes from choosing the wrong model class. And successful deployment requires an understanding of systems, where tool orientation is what usually gets brought to it.

In finance and in business generally the point is blunt. Anyone deploying AI has to understand what kind of problem is actually being modelled.

The lever is the choice, not the size

Artificial intelligence is a kit of specialized architectures and not a single system. Language models structure language, convolutional networks enable perception, reinforcement learning optimizes decisions.

The decisive lever sits in a clear choice of architecture and clean system design, well before it sits in larger models.

Sources

Grouped by the section they support.

Opening

  • Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin (2017), Attention Is All You Need, NeurIPS 2017, arXiv:1706.03762. Supports self-attention and the transformer.
  • Goodfellow, Bengio and Courville (2016), Deep Learning, MIT Press, chapter 9, Convolutional Networks, deeplearningbook.org. Supports the convolution as a small kernel applied at every position of the input.
  • Litjens et al. (2017), A Survey on Deep Learning in Medical Image Analysis, Medical Image Analysis 42, 60–88, arXiv:1702.05747. Supports convolutional networks as the method of choice for medical images, including detection tasks.

There is no single AI because there is no single kind of data

  • No external source. The framing by type of data is the author's own.

Large language models structure symbols

  • Vaswani et al. (2017). Supports the transformer and self-attention, including the box that explains them.
  • Brown et al. (2020), Language Models are Few-Shot Learners, NeurIPS 2020, arXiv:2005.14165. Supports large language models as autoregressive models that predict the next token.
  • Sennrich, Haddow and Birch (2016), Neural Machine Translation of Rare Words with Subword Units, ACL 2016, arXiv:1508.07909. Supports tokens as subword units, the basis of the box on tokens.
  • Bender and Koller (2020), Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data, ACL 2020, 5185–5198, aclanthology.org. Supports the point that a model trained on form alone has no access to meaning.
  • Shen, Song, Tan, Li, Lu and Zhuang (2023), HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face, NeurIPS 2023, arXiv:2303.17580. Supports a language model acting as the controller of other AI models. The list of typical uses is the author's own.

Convolutional networks perceive

  • Goodfellow, Bengio and Courville (2016), chapter 9. Supports local filters, parameter sharing and tolerance to shifts in position.
  • Zeiler and Fergus (2014), Visualizing and Understanding Convolutional Networks, ECCV 2014, arXiv:1311.2901. Supports the hierarchy of features, from edges and corners in early layers to whole objects in deep ones.
  • Litjens et al. (2017). Supports medical image diagnostics. The other uses are listed by the author without an individual source.

Reinforcement learning decides

  • Sutton and Barto (2018), Reinforcement Learning: An Introduction, second edition, MIT Press, incompleteideas.net. Supports the agent, environment, actions, reward and policy, delayed reward, and the trade-off between exploration and exploitation.
  • Mnih et al. (2015), Human-level control through deep reinforcement learning, Nature 518, 529–533, nature.com. Silver et al. (2016), Mastering the game of Go with deep neural networks and tree search, Nature 529, 484–489, nature.com. Both support game playing. Robotics, logistics, energy management and the use in economic simulations are listed by the author without an individual source.

The differences that matter

  • No external source. The comparison summarizes the three sections above.

Current systems combine all three

  • Mnih et al. (2015). Supports a convolutional network and reinforcement learning combined in one agent that learns from pixels.
  • Alayrac et al. (2022), Flamingo: a Visual Language Model for Few-Shot Learning, NeurIPS 2022, arXiv:2204.14198. Supports a multimodal assistant that joins a convolutional vision encoder (NFNet) to a language model.
  • Ouyang et al. (2022), Training language models to follow instructions with human feedback, NeurIPS 2022, arXiv:2203.02155. Supports reinforcement learning used in training language models.
  • Shen et al. (2023). Supports language models as the coordinating layer of agent-based systems. The three consequences for companies and analysts are the author's own.

The lever is the choice, not the size

  • No external source. The conclusion is the author's own.
The link has been copied!