"The group faces significant headwinds in its main markets." Anyone who has read an earnings release knows what that sentence means: trouble ahead. VADER, a sentiment tool that ships with NLTK, one of the standard Python libraries for language processing, labels it positive. It has no entry for headwinds. The only word it recognizes is significant, which its dictionary scores at +0.8, so the sentence comes out as good news.
That mistake matters because sentiment scores feed news screens, research dashboards and trading models, and a general-purpose tool like VADER is three lines of code away for anyone who builds one. On 253 labeled sentences from financial news, VADER got 55 percent right, below the 62 percent that a rule answering "neutral" every time would score. A small model trained on 1,011 sentences of the same kind reached 83 percent. For anyone starting out in research or analysis, the finding travels well beyond this dataset: a score is only as reliable as the text its tool was built on, and the error does not announce itself.
A dictionary written for social media has no entry for a writedown
C.J. Hutto and Eric Gilbert published VADER in 2014 for sentiment in social media posts. It works from a list of about 7,500 words and emoticons, each with a score between roughly −4 and +4, and a set of rules that adjust for negation, intensifiers such as "very", capital letters and exclamation marks. For a product review that is often enough.
Financial news breaks it in two ways. The vocabulary that matters is missing: headwinds, writedown, impairment, downgrade and bankruptcy are not in the dictionary, so VADER treats them as neutral. Words it does know point the wrong way. "Net loss narrowed to EUR 2 million from EUR 9 million" describes an improvement, and VADER calls it negative because loss carries −1.3 and narrowed carries nothing. On the test set, it labeled 19 of the 33 negative sentences positive.
The problem is older than VADER. In 2011, Tim Loughran and Bill McDonald of the University of Notre Dame counted negative words in a large sample of 10-K filings using the Harvard dictionary, a word list long used in finance research. Almost three-fourths of those counts, 73.8 percent, came from words that are typically not negative in a financial context, such as tax, cost, capital and liability. Their answer was a word list built for finance. The project here tests the other answer: let a model learn the vocabulary from labeled examples.
A thousand labeled sentences teach a simple model most of the vocabulary
Its examples come from the Financial PhraseBank, which Pekka Malo and colleagues at Aalto University released in 2014. It holds about 4,840 sentences from English financial news, each labeled negative, neutral or positive by 16 annotators: three researchers and 13 master's students majoring mostly in finance, accounting and economics. This project uses the split published as FinanceMTEB/financial_phrasebank, which contains 1,264 of them: 62 percent neutral, 25 percent positive and only 13 percent negative.
The model is deliberately plain. TF-IDF, which weights words by how distinctive they are, turns each sentence into a list of weights for single words and two-word phrases, so net loss counts as one unit. Logistic regression, a classifier that learns one weight per term and label, then turns those into a probability for each of the three labels. The model learned from 1,011 sentences, and both models were scored on the remaining 253, which neither had seen.
[ Results · test set, 253 sentences ]
-
Always "neutral"Majority-class baseline61.7%Accuracy–Macro-F10 of 33Negatives caught
-
VADERGeneral-purpose lexicon55.3%Accuracy0.458Macro-F16 of 33Negatives caught
-
TF-IDF + logistic regressionTrained on 1,011 sentences83.0%Accuracy0.743Macro-F114 of 33Negatives caught
Financial PhraseBank, FinanceMTEB split. Stratified random split, seed 42. Full figures in results.json in the repository.
Macro-F1 averages the score of each label equally, so a model cannot look good by getting only the common neutral sentences right. On that measure the gap holds: 0.46 for VADER against 0.74 for the trained model.
Each verdict comes down to a few lines of arithmetic
Neither model is a black box. VADER adds up word scores and squeezes the sum into a range from −1 to +1. The trained model weights each term, adds the weights for every label and turns the totals into probabilities. Below, the opening sentence runs through both, step by step.
VADER. VADER’s dictionary assigns a valence, a number between −4 and +4, to about 7,500 words and emoticons. Write the tokens of a sentence as , where is the number of tokens. Let be the valence of token after VADER’s rules for negation, intensifiers and capital letters, with if the token is not in the dictionary. The raw score of the sentence is the sum of these valences:
VADER also adds a small boost to for exclamation and question marks; the opening sentence has none. The raw score has no fixed range: it grows with every scored word. VADER maps it into the range from −1 to +1 by dividing it by a quantity that grows with itself. With a fixed constant , the compound score is
A sentence counts as positive if compound ≥ 0.05, as negative if compound ≤ −0.05, and as neutral in between. In the opening sentence only significant is in the dictionary, with a valence of 0.8. So and compound : positive.
TF-IDF. TF-IDF is a standard method from information retrieval. Its second half, the inverse document frequency, goes back to Karen Spärck Jones in 1972, and common libraries such as scikit-learn ship the whole method ready to use. This project uses the scikit-learn implementation, and the equations below show what it computes.
The trained model cannot read words, only numbers. TF-IDF turns a sentence into numbers by asking, for every term, how much that term says about this particular sentence. Two observations drive the answer. A term that occurs several times in a sentence matters more for that sentence. A term that occurs in almost every sentence, such as the, says little about any one of them.
A term is a single word or a pair of adjacent words, so net loss counts as one term next to net and loss. The model’s vocabulary holds the 5,000 terms that occur most often in the training sentences: 1,915 words and 3,085 pairs. For a sentence and a term in , define
- the term frequency, the number of times occurs in ;
- the number of training sentences, here 1,011;
- the document frequency, the number of training sentences that contain at least once.
The inverse document frequency turns into a weight that is high for rare terms and low for common ones:
Here is the natural logarithm. The fraction is large when few sentences contain the term, and the logarithm compresses it so that a very rare term does not dominate everything else. The two 1s inside the fraction prevent a division by zero, and the 1 added at the end keeps a term that occurs in every sentence at a weight of 1 rather than 0. The occurs in 700 of the 1,011 training sentences, so its idf is ln(1,012/701) + 1 ≈ 1.37. Main occurs in 4, so its idf is ln(1,012/5) + 1 ≈ 6.31.
The weight of a term that occurs in sentence combines both parts. A second logarithm dampens repetition, so a term that occurs twice counts about 1.7 times as much as one that occurs once:
A term that does not occur in the sentence gets .
Collecting the weights of all terms in gives a vector with 5,000 entries, one entry per term, almost all of them zero. A long sentence would produce a larger vector than a short one, so the vector is divided by its length, the Euclidean norm
The result is the vector , again with one entry per term. Its squared entries add up to 1, and is all the classifier sees of the sentence.
Table 1 carries out these steps for the opening sentence. Its 9 tokens form 17 terms, 9 words and 8 pairs, and 9 of those terms are in . Headwinds, faces and every pair that contains them are not, so they get no weight at all. Each remaining term occurs once, so , the logarithm in (4) is zero, and .
[ Table 1 · vectors w and x, opening sentence ]
Each row shows the entries of w and x for one term; the other 4,991 entries of both vectors are zero. n = 1,011 training sentences. ‖w‖₂ = √(1.37² + 1.81² + … + 6.31²) ≈ 13.71, so xt = wt / 13.71. The rarer a term, the larger its share of the sentence.
Logistic regression. Let be the set of labels and one of them. For every label the model has an intercept , a baseline score that applies to every sentence, and a coefficient for every term in , which says how strongly that term speaks for label . That makes 3 × 5,001 = 15,003 numbers in all. The score of label adds the intercept and the weighted terms:
Because for every term that is not in the sentence, only the sentence’s own terms contribute. The softmax function then turns the three scores into probabilities. With , Euler’s number, the exponential makes every score positive, and dividing by the sum over all labels in makes the three numbers add up to 1:
where stands for the label of the sentence. The model predicts the label with the highest probability.
Training sets the intercepts and coefficients. Write the training data as pairs for , where is the vector of training sentence from (5) and its label from the annotators. Let collect all coefficients and be the sum of their squares. Training solves
The sum is the log-loss. Each of its terms is close to zero when the model gives the correct label a probability near 1, and grows without limit as that probability falls toward 0. The first part penalizes large coefficients, which keeps the model from building its verdicts on a few rare terms. The constant , the scikit-learn default, sets the balance between the two parts.
Table 2 applies (6) and (7) to the vector from Table 1, whose nine nonzero entries are the only ones that enter the sum in (6).
[ Table 2 · scores and probabilities, opening sentence ]
The nine terms pull toward positive, led by significant and its. The intercept, which reflects that most training sentences are neutral, outweighs them. The word that carries the meaning, headwinds, is not in the vocabulary and never enters equation (6).
Implementation. The code is written in Python. NLTK provides VADER through its SentimentIntensityAnalyzer. scikit-learn provides everything else: TfidfVectorizer implements equations (3) to (5), with ngram_range=(1, 2) for word pairs, max_features=5000 for the vocabulary and sublinear_tf=True for the logarithm in (4); LogisticRegression implements (6) to (8); and train_test_split draws the stratified split. pandas and pyarrow load the dataset from Hugging Face, NumPy handles the arrays and matplotlib draws the result dashboard. The figures in this article come from Python 3.10, scikit-learn 1.7.2 and NLTK 3.10.4. The demo below runs both models in JavaScript, and the export script checks that the trained model in the browser assigns the same label as scikit-learn to every test sentence. Code, data source and the full results are on GitHub: github.com/EmilOhligs/Financial-Sentiment-Analysis.
Both models run in your browser below
Type a sentence from an earnings release or pick one of the examples. The demo runs the same two models used for the numbers above, the trained one with its exact weights, and shows which words drove each verdict. Nothing you type leaves the page.
[ Live demo · two models, one sentence ]
VADER
Words it found in its dictionary
TF-IDF + logistic regression
Terms that pushed it toward this label
The trained model still misses more than half of the bad news
The headline numbers hide the weakest spot. Of the 33 negative sentences in the test set, the trained model caught 14. It labeled 15 of the rest neutral and four positive. When it did say negative, it was right every time, so its errors come from staying quiet. Negative sentences are the rarest in the data, 13 percent of the total, so the model sees few of them in training, and they are the ones that cost money when a screen misses them.
The opening sentence shows the limit. Faced with it, the trained model does better than VADER and still gets it wrong: it settles on neutral, at 64 percent. A guidance cut, "The bank cut its full-year guidance", also comes out neutral, and "Revenue was flat compared with last year" comes out positive. It reads "Net loss narrowed to EUR 2 million from EUR 9 million" correctly, as positive at 72 percent. A model that counts words and word pairs knows only which terms appeared next to which labels in its training data, so it is as good as the examples it happened to see.
Three limits of the evaluation matter as much. The sentences were split at random, so the model was tested on news from the same period it learned from, while a model in production meets text from the future. All figures come from a single split, without cross-validation, so they carry sampling noise. And this is a baseline. Dogu Araci's FinBERT, a language model adapted to financial text, reported 0.86 accuracy on the full PhraseBank and 0.97 on the sentences all annotators agreed on, under its own split and cross-validation. Those figures are not directly comparable with the ones here, but the distance is large enough to be real.
The failure mode is the part worth remembering. A tool that answers confidently on every input will produce a number for a sentence it cannot read, and the number looks the same as one it got right.
Before any sentiment score feeds a model, a memo or a screen, it is worth asking what text the tool was built on, and then testing it: label a hundred sentences from your own sources, run the tool, and look first at the label that would cost you most to miss. The open question is how much of this gap the newest general-purpose language models close without any training on financial text, and whether their errors are any easier to see.
Emil Ohligs and ThinkBeyondAI built this project as a comparison of two transparent baselines. Its code is released under the MIT license. The Financial PhraseBank is licensed CC BY-NC-SA 3.0, and the demo's trained weights inherit that license. Every number in this article comes from results.json in the repository, which one script regenerates.
If such a project sounds interesting to you, please get in touch with us. We are happy to improve and modify such models to your needs. This model is not an end result, but a research project.
Sources
- Hutto, C. J., & Gilbert, E. (2014). VADER: A Parsimonious Rule-based Model for Sentiment Analysis of Social Media Text. ICWSM.
- Loughran, T., & McDonald, B. (2011). When Is a Liability Not a Liability? Textual Analysis, Dictionaries, and 10-Ks. Journal of Finance, 66(1), 35–65. https://doi.org/10.1111/j.1540-6261.2010.01625.x
- Malo, P., Sinha, A., Korhonen, P., Wallenius, J., & Takala, P. (2014). Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65(4). Dataset card: https://huggingface.co/datasets/financial_phrasebank
- Spärck Jones, K. (1972). A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28(1), 11–21. https://doi.org/10.1108/eb026526
- Araci, D. (2019). FinBERT: Financial Sentiment Analysis with Pre-trained Language Models. https://arxiv.org/abs/1908.10063