Small models are the default question now

This piece first appeared in German. The English version is a rewrite rather than a line-by-line translation, and the German original stays online in the archive: Kleiner denken, besser deployen: Was sind Small Language Models?.

Microsoft's Phi-3 Mini carries 3.8 billion parameters. On the company's own benchmark table it scores an average of 67.6 against 65.5 for Llama-3-8B-Instruct and 56.4 for Mistral-7B, both of which are roughly twice its size. Quantized, according to one published comparison, it fits in about 2.4 gigabytes and runs on an ordinary laptop with no dedicated graphics card, where Llama 3 8B in the same format needs around 6.

For anyone in a bank or an asset manager deciding where a model will run, that is the number that matters. Parameter counts were the currency of progress for three years, and the assumption underneath them is coming apart.

The scaling formula was right and incomplete

Large language models were built on a simple equation: more data, more parameters, more compute, better results. DeepMind's Chinchilla study in 2022 showed the industry had been systematically missing the optimal ratio of model size to training data. Models were being trained too large for the data available to them.

What followed was a gradual shift, not a reversal, and it started in research labs and has now reached deployment decisions inside companies. The question is no longer how big. It is how much performance can be extracted from a model that runs on a laptop, an on-premise server or a mobile device without a cluster of graphics cards behind it.

Compression became a strategic instrument

Three techniques define the efficiency frontier: quantization, pruning and knowledge distillation. None of them is new. Their combined application to language models has improved enough over two years to move the balance between large and small models.

Quantization reduces the numerical precision of the model weights. Every weight in a neural network is a number, and the question is how many bits are used to represent it. FP32, 32-bit floating point, is the training standard: high precision, high memory demand. FP16 halves that demand at minimal loss of accuracy, and INT8 goes further, encoding weights as whole numbers in 8 bits. A 2025 study in the IEEE literature reports a substantial reduction in memory footprint at a controlled loss of accuracy.

Pruning takes a different route. Not every connection in a network contributes equally to the output, and many weights are effectively redundant. Pruning sets those to zero and removes them from the computation, which shrinks the model without performance collapsing in proportion.

Knowledge distillation is conceptually the most elegant of the three. A large teacher model transfers what it knows to a smaller student model, and the student learns from the teacher's output distributions and not from raw data alone. What it receives is a compressed picture of the world that holds considerably more than its parameter count suggests.

In practice the three are increasingly used together: distillation first, then pruning, then quantization, with each stage optimizing on the result of the last. The output is a class of model that looked unrealistic two years ago. Compact, fast, specializable, and free of any dependency on an external cloud service.

What this means for a financial institution

The strategic implication is more direct than it first appears. Institutions planning AI deployment face a structural fork: cloud-based frontier models with maximum raw capability on one side, specialized small language models running locally on controllable infrastructure on the other.

Across much of the operational work in finance, document classification, contract analysis, internal compliance checks, structured data extraction from research reports, the performance difference after domain-specific fine-tuning appears marginal. Work presented at the 2025 ACM conference on AI in finance found that tool-enhanced small models can hold their own against much larger ones on targeted financial tasks.

The decisive advantage is control, more than raw accuracy: control over latency, over infrastructure cost, over where data travels. That last one is gaining regulatory weight. The European Data Protection Board stated in December 2024 that AI models trained on personal data do not automatically count as anonymous, which leaves them within the reach of the GDPR. On-premise deployment makes the question structurally moot.

Three limits worth taking seriously

Efficiency as the new frontier does not make small models the right choice everywhere.

The first limit is hallucination. Smaller models hallucinate more often on complex or out-of-domain queries, because their capacity for implicit world knowledge is structurally bounded. In compliance checks, contract analysis or regulatory assessment that is an operational risk. Using small models in those workflows requires a validation layer: either retrieval-augmented generation, which confines the model to a verified document base, or human review of critical outputs.

Fine-tuning is the second limit. The claim that the performance gap closes after domain-specific work is true, and that work is a precondition, not a finishing touch. It needs curated training data, infrastructure and systematic evaluation. Underestimate that effort and an untuned small model will perform considerably worse in a financial context than a generalist frontier model.

The third limit is technical: context window size. Small models typically have smaller context windows, which is a hard constraint for long contracts, multi-page research reports and extensive regulatory texts. Not every task can be compensated for by chunking or retrieval.

What actually shifts is the default

Frontier models are not becoming obsolete. Complex synthesis, multi-step reasoning over unstructured data and applications that need broad world knowledge without pre-filtering remain places where the large models keep their lead.

What changes is the default assumption. Plan an AI project for a financial institution today, reach automatically for the cloud-based frontier model, and you have made a decision without asking the question that matters. Does a specialized model running locally do the same job, at a fraction of the cost and with less regulatory friction?

Sources

Grouped by the section they support.

Opening

Microsoft, Phi-3-mini-4k-instruct, model card, huggingface.co. Supports the 3.8 billion parameters and the benchmark averages: 67.6 for Phi-3-Mini-4K-Instruct, 65.5 for Llama-3-8B-Instruct and 56.4 for Mistral-7B.
Oberst (2024), Best Small Language Models for Accuracy and Enterprise Use Cases — Benchmark Results, llmware, 26 August 2024, llmware.ai. Supports a quantized Phi-3 Mini of about 2.4 gigabytes.
Abdin et al. (2024), Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone, arXiv:2404.14219. Supports on-device use: quantized to 4 bits, the model occupies about 1.8 gigabytes and runs offline on an iPhone 14.

The scaling formula was right and incomplete

Hoffmann et al. (2022), Training Compute-Optimal Large Language Models, DeepMind, arXiv:2203.15556. Supports the Chinchilla finding that large language models of the time were significantly undertrained for their size.

Compression became a strategic instrument

Agrawal, Kumar and Lnu (2025), Efficient LLMs for Edge Devices: Pruning, Quantization, and Distillation Techniques, 2025 International Conference on Machine Learning and Autonomous Systems (ICMLAS), ieeexplore.ieee.org. Supports quantization halving memory use at a small loss of accuracy, measured on a 500-megabyte test model.
Han, Pool, Tran and Dally (2015), Learning both Weights and Connections for Efficient Neural Networks, NIPS 2015, arXiv:1506.02626. Supports pruning of redundant connections without loss of accuracy.
Hinton, Vinyals and Dean (2015), Distilling the Knowledge in a Neural Network, NIPS 2014 Deep Learning Workshop, arXiv:1503.02531. Supports knowledge distillation from a large model into a smaller one.
Belcak, Heinrich, Diao, Fu, Dong, Muralidharan, Lin and Molchanov (2025), Small Language Models are the Future of Agentic AI, arXiv:2506.02153. Supports the argument that small models are powerful enough and more economical for many tasks. The sequence of distillation, pruning and quantization is the author's summary.

What this means for a financial institution

Assis, Surica, Kroll, Munhoz, Rabbani, Bollis, Pellicer and Paes (2025), On the Potential of Tool-Enhanced Small Language Models to Match Large Models in Finance, Proceedings of the 6th ACM International Conference on AI in Finance (ICAIF 2025), doi.org/10.1145/3768292.3770409. Supports tool-enhanced small models on targeted financial tasks.
European Data Protection Board (2024), EDPB opinion on AI models: GDPR principles support responsible AI, Opinion 28/2024, 18 December 2024, edpb.europa.eu. Supports the case-by-case test for whether a model trained on personal data counts as anonymous. The consequence for on-premise deployment is the author's own.

Three limits worth taking seriously

Microsoft, Phi-3-mini-4k-instruct. Supports a context length of 4,000 tokens for this variant. The points on hallucination and fine-tuning effort are the author's assessment.

What actually shifts is the default