This piece first appeared in German. The English version is a rewrite rather than a line-by-line translation, and the German original stays online in the archive: Warum KI nicht neutral antwortet: Alignment.

Human raters preferred the answers of a 1.3-billion-parameter model over those of one with 175 billion. That is the InstructGPT result reported by Ouyang and colleagues in 2022. The smaller model won with a hundred times fewer parameters, because it had been tuned on human feedback and the larger one had not.

Making a language model bigger, as the paper puts it, does not inherently make it better at following a user's intent. What happens after pretraining decides that, and the name for it is alignment. For anyone reading a model's output in a professional setting, this is the part worth understanding, because it explains why the answer on your screen never arrives from a value-free space.

Alignment means directing an AI system toward human goals, values and intentions. A system counts as aligned when it pursues the goals its developers or users intended and produces no unwanted side effects. That definition already shows the shape of the problem. Alignment is a precondition for responsible use and not a technical footnote, and the more capable a system becomes, the sharper the question of whether it actually does what people expect.

From predicting text to steering behavior

Language models are first trained to recognize patterns in large bodies of text and predict the next word, or token. Impressive linguistic competence comes out of that single objective. A model that predicts text well is still not automatically a good assistant. It can invent facts, produce toxic content, misread instructions, or deliver answers that sound convincing while missing what the user actually needed.

Alignment changes the job description. The system is no longer meant only to continue plausibly from what it has learned. It is meant to recognize which answer is appropriate, helpful and safe in a specific situation, which moves the target from prediction to behavior.

Why the problem is not solvable by optimization alone

The central risk in an unaligned system is that it satisfies a goal formally while ignoring the human intention behind it. Zweck and Werner describe this through incorrectly or incompletely specified objectives. Optimize strictly for one measurable quantity and a system may find unexpected routes to maximizing it, even where the result contradicts the actual purpose. The name for that is reward hacking.

A simple case: a recommender system optimized purely for engagement will favor content that provokes and polarizes. Formally the goal is met, while socially the side effects are considerable. Language models can do the equivalent, producing answers that are unusually persuasive and factually wrong.

The difficulty underneath is that human goals are rarely fully or unambiguously expressible. People expect more than a correct answer from these systems. They expect sensitivity to context, caution, fairness, transparency and an appropriate tone, and none of that translates cleanly into a simple mathematical objective. Alignment is a continuing negotiation between technical optimization, human evaluation and social norms.

What RLHF actually changes

The most influential approach is reinforcement learning from human feedback, or RLHF. Human-written example answers are collected first, and people then rate different model responses for quality, which trains a reward model that learns what humans prefer. The language model is adjusted to produce that preferred behavior more often.

The effect shows up on three levels. Tone changes: answers become more polite, more structured, more closely tied to the question. Content selection changes: the model learns to foreground relevant information and avoid irrelevant or risky material. And appetite for risk changes: on medicine, law or self-harm, aligned models answer more cautiously and point toward professional help or further checking.

RLHF solves none of this completely. A model can learn from human feedback to sound particularly convincing without becoming more truthful. It can also turn over-cautious, or smooth its answers until conflicts, uncertainties and minority perspectives become less visible. Alignment improves behavior, and it guarantees no truth.

Constitutional AI writes the rules down

A second family of approaches steers the system through principles. Bai and colleagues described one in 2022, training a system against a kind of constitution: a set of rules the model applies when evaluating and revising its own answers. The aim is a system that is both helpful and less harmful, without a human judgment on every problematic output.

That approach makes the non-technical core of the subject unusually visible. Writing a constitution for an AI system means deciding which values it prioritizes.

Should it be especially cautious? Is a controversial question to be presented as neutrally as possible, and is certain content to be refused outright? Does the system instruct, warn, or only inform? Each of those decisions shapes what people later see on their screens.

Bai and colleagues make the point that for general-purpose systems there is no way to avoid choosing some form of principles. Where the principles are not openly visible, they still shape the behavior. Which is exactly why transparency matters: an answer comes out of design decisions as much as out of data.

Four ways alignment shows up in an answer

It shapes the form. An aligned model replies in a more structured, more polite register, avoids extreme formulations and tries to make uncertainty visible.

It shapes the content. The model decides not only what gets said but what gets omitted, qualified or wrapped in a warning, and that filtering is most visible on legal, medical, financial and safety-related questions.

It shapes the boundaries. An unaligned model could deliver dangerous instructions, manipulative strategies or discriminatory content without restriction, where an aligned one refuses or offers a safer alternative.

And it shapes the perception of objectivity. Because aligned answers usually read as calm, balanced and professional, they can appear especially trustworthy. That impression is a product of a training process that rewards certain answer styles, and trustworthy in this sense is not a synonym for true.

Alignment is a governance question as much as a research one

Training a model once is not enough for any of this to hold. Systems have to be observed, evaluated and adjusted across their whole lifecycle, which is where alignment meets governance.

The NIST AI Risk Management Framework sets out a voluntary structure to help organizations manage AI-related risks to individuals, organizations and society, with trustworthiness built into design, development, use and evaluation. For alignment that means it is an organizational task and not only a research question. Companies and public institutions have to settle how AI gets tested, how risks are documented, who carries responsibility, and how misbehavior is corrected. With generative systems this matters most, because the outputs are not fully predictable.

Alignment makes AI usable, and not infallible

Alignment is among the central preconditions for using modern AI systems sensibly and responsibly. It is what turns linguistically plausible output into something helpful, safe and appropriate, through methods like RLHF and Constitutional AI that tie a model more closely to human expectations, rules and values.

It should not be confused with truth or with neutrality. An aligned model can still hallucinate, weight information badly, and reproduce cultural and institutional values. Alignment does not make AI objective. What it does is make visible that every AI output is shaped by data, training procedures, feedback, rules and human decisions.

Sources

Grouped by the section they support.

Opening

  • Ouyang et al. (2022), Training language models to follow instructions with human feedback, NeurIPS 2022, arXiv:2203.02155. Supports the 1.3-billion-parameter InstructGPT model being preferred over the 175-billion-parameter GPT-3, and the sentence that making language models bigger does not inherently make them better at following a user's intent.
  • Werner and Zweck (2025), AI-Alignment – Eine zentrale Herausforderung unserer Zeit?, Zukünftige Technologien No. 109, VDI Technologiezentrum, Düsseldorf, June 2025, vditz.de. Supports the definition of an aligned system as one that pursues the intended goals of its developers or users without unwanted side effects (page 6).

From predicting text to steering behavior

  • Ouyang et al. (2022). Supports language models trained on next-token prediction producing untruthful, toxic or unhelpful output, and tuning after pretraining as the remedy.

Why the problem is not solvable by optimization alone

  • Werner and Zweck (2025), pages 6–7. Supports incompletely or wrongly specified objectives and reward hacking. The recommender example and the point that human goals resist a simple objective function are the author's own.

What RLHF actually changes

  • Ouyang et al. (2022). Supports the three steps: human-written demonstrations, human rankings of model outputs, and reinforcement learning from human feedback. The three levels of effect and the limits of the method are the author's own assessment.

Constitutional AI writes the rules down

  • Bai et al. (2022), Constitutional AI: Harmlessness from AI Feedback, Anthropic, arXiv:2212.08073. Supports training against a list of rules or principles with AI feedback, and the statement that "we cannot avoid choosing some set of principles to govern it, even if they remain hidden or implicit."

Four ways alignment shows up in an answer

  • No external source. The four effects are the author's synthesis of the sections above.

Alignment is a governance question as much as a research one

  • National Institute of Standards and Technology (2023), Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1, 26 January 2023, nist.gov. Supports a framework intended for voluntary use that helps manage AI risks to individuals, organizations and society, with trustworthiness built into design, development, use and evaluation.

Alignment makes AI usable, and not infallible

  • No external source. The conclusion is the author's own.
The link has been copied!