Project · Completed · Code, data and full results on GitHub · Live demo below

"The group faces significant headwinds in its main markets." Anyone who has read an earnings release knows what that sentence means: trouble ahead. VADER, a sentiment tool that ships with NLTK, one of the standard Python libraries for language processing, labels it positive. It has no entry for headwinds. The only word it recognizes is significant, which its dictionary scores at +0.8, so the sentence comes out as good news.

That mistake matters because sentiment scores feed news screens, research dashboards and trading models, and a general-purpose tool like VADER is three lines of code away for anyone who builds one. On 253 labeled sentences from financial news, VADER got 55 percent right, below the 62 percent that a rule answering "neutral" every time would score. A small model trained on 1,011 sentences of the same kind reached 83 percent. For anyone starting out in research or analysis, the finding travels well beyond this dataset: a score is only as reliable as the text its tool was built on, and the error does not announce itself.


A dictionary written for social media has no entry for a writedown

C.J. Hutto and Eric Gilbert published VADER in 2014 for sentiment in social media posts. It works from a list of about 7,500 words and emoticons, each with a score between roughly −4 and +4, and a set of rules that adjust for negation, intensifiers such as "very", capital letters and exclamation marks. For a product review that is often enough.

Financial news breaks it in two ways. The vocabulary that matters is missing: headwinds, writedown, impairment, downgrade and bankruptcy are not in the dictionary, so VADER treats them as neutral. Words it does know point the wrong way. "Net loss narrowed to EUR 2 million from EUR 9 million" describes an improvement, and VADER calls it negative because loss carries −1.3 and narrowed carries nothing. On the test set, it labeled 19 of the 33 negative sentences positive.

The problem is older than VADER. In 2011, Tim Loughran and Bill McDonald of the University of Notre Dame counted negative words in a large sample of 10-K filings using the Harvard dictionary, a word list long used in finance research. Almost three-fourths of those counts, 73.8 percent, came from words that are typically not negative in a financial context, such as tax, cost, capital and liability. Their answer was a word list built for finance. The project here tests the other answer: let a model learn the vocabulary from labeled examples.


A thousand labeled sentences teach a simple model most of the vocabulary

Its examples come from the Financial PhraseBank, which Pekka Malo and colleagues at Aalto University released in 2014. It holds about 4,840 sentences from English financial news, each labeled negative, neutral or positive by 16 annotators: three researchers and 13 master's students majoring mostly in finance, accounting and economics. This project uses the split published as FinanceMTEB/financial_phrasebank, which contains 1,264 of them: 62 percent neutral, 25 percent positive and only 13 percent negative.

The model is deliberately plain. TF-IDF, which weights words by how distinctive they are, turns each sentence into a list of weights for single words and two-word phrases, so net loss counts as one unit. Logistic regression, a classifier that learns one weight per term and label, then turns those into a probability for each of the three labels. The model learned from 1,011 sentences, and both models were scored on the remaining 253, which neither had seen.

[ Results · test set, 253 sentences ]

  • Always "neutral"Majority-class baseline
    61.7%Accuracy
    –Macro-F1
    0 of 33Negatives caught
  • VADERGeneral-purpose lexicon
    55.3%Accuracy
    0.458Macro-F1
    6 of 33Negatives caught
  • TF-IDF + logistic regressionTrained on 1,011 sentences
    83.0%Accuracy
    0.743Macro-F1
    14 of 33Negatives caught

Financial PhraseBank, FinanceMTEB split. Stratified random split, seed 42. Full figures in results.json in the repository.

Macro-F1 averages the score of each label equally, so a model cannot look good by getting only the common neutral sentences right. On that measure the gap holds: 0.46 for VADER against 0.74 for the trained model.


Each verdict comes down to a few lines of arithmetic

Neither model is a black box. VADER adds up word scores and squeezes the sum into a range from −1 to +1. The trained model weights each term, adds the weights for every label and turns the totals into probabilities. Below, the opening sentence runs through both, step by step.

VADER. VADER’s dictionary assigns a valence, a number between −4 and +4, to about 7,500 words and emoticons. Write the tokens of a sentence as u1,…,um, where m is the number of tokens. Let vi be the valence of token ui after VADER’s rules for negation, intensifiers and capital letters, with vi=0 if the token is not in the dictionary. The raw score s of the sentence is the sum of these valences:

s=∑i=1mvi.(1)

VADER also adds a small boost to s for exclamation and question marks; the opening sentence has none. The raw score has no fixed range: it grows with every scored word. VADER maps it into the range from −1 to +1 by dividing it by a quantity that grows with s itself. With a fixed constant α=15, the compound score is

compound=ss2+α.(2)

A sentence counts as positive if compound ≥ 0.05, as negative if compound ≤ −0.05, and as neutral in between. In the opening sentence only significant is in the dictionary, with a valence of 0.8. So s=0.8 and compound =0.8/0.64+15≈0.20: positive.

TF-IDF. TF-IDF is a standard method from information retrieval. Its second half, the inverse document frequency, goes back to Karen Spärck Jones in 1972, and common libraries such as scikit-learn ship the whole method ready to use. This project uses the scikit-learn implementation, and the equations below show what it computes.

The trained model cannot read words, only numbers. TF-IDF turns a sentence into numbers by asking, for every term, how much that term says about this particular sentence. Two observations drive the answer. A term that occurs several times in a sentence matters more for that sentence. A term that occurs in almost every sentence, such as the, says little about any one of them.

A term is a single word or a pair of adjacent words, so net loss counts as one term next to net and loss. The model’s vocabulary V holds the 5,000 terms that occur most often in the training sentences: 1,915 words and 3,085 pairs. For a sentence d and a term t in V, define

tft
the term frequency, the number of times t occurs in d;
n
the number of training sentences, here 1,011;
dft
the document frequency, the number of training sentences that contain t at least once.

The inverse document frequency turns dft into a weight that is high for rare terms and low for common ones:

idft=ln1+n1+dft+1.(3)

Here ln is the natural logarithm. The fraction is large when few sentences contain the term, and the logarithm compresses it so that a very rare term does not dominate everything else. The two 1s inside the fraction prevent a division by zero, and the 1 added at the end keeps a term that occurs in every sentence at a weight of 1 rather than 0. The occurs in 700 of the 1,011 training sentences, so its idf is ln(1,012/701) + 1 ≈ 1.37. Main occurs in 4, so its idf is ln(1,012/5) + 1 ≈ 6.31.

The weight wt of a term t that occurs in sentence d combines both parts. A second logarithm dampens repetition, so a term that occurs twice counts about 1.7 times as much as one that occurs once:

wt=(1+lntft)·idft.(4)

A term that does not occur in the sentence gets wt=0.

Collecting the weights of all terms in V gives a vector w with 5,000 entries, one entry wt per term, almost all of them zero. A long sentence would produce a larger vector than a short one, so the vector is divided by its length, the Euclidean norm

‖w‖2=∑t∈Vwt2,xt=wt‖w‖2.(5)

The result is the vector x, again with one entry xt per term. Its squared entries add up to 1, and x is all the classifier sees of the sentence.

Table 1 carries out these steps for the opening sentence. Its 9 tokens form 17 terms, 9 words and 8 pairs, and 9 of those terms are in V. Headwinds, faces and every pair that contains them are not, so they get no weight at all. Each remaining term occurs once, so tft=1, the logarithm in (4) is zero, and wt=idft.

[ Table 1 · vectors w and x, opening sentence ]

Term tdftidftwtxt
the7001.371.370.10
in4491.811.810.13
its973.333.330.24
group793.543.540.26
the group204.884.880.36
markets194.924.920.36
significant95.625.620.41
main46.316.310.46
in its46.316.310.46

Each row shows the entries of w and x for one term; the other 4,991 entries of both vectors are zero. n = 1,011 training sentences. ‖w‖₂ = √(1.37² + 1.81² + … + 6.31²) ≈ 13.71, so xt = wt / 13.71. The rarer a term, the larger its share of the sentence.

Logistic regression. Let K={negative,neutral,positive} be the set of labels and k one of them. For every label the model has an intercept bk, a baseline score that applies to every sentence, and a coefficient βk,t for every term t in V, which says how strongly that term speaks for label k. That makes 3 × 5,001 = 15,003 numbers in all. The score zk of label k adds the intercept and the weighted terms:

zk=bk+∑t∈Vβk,txt.(6)

Because xt=0 for every term that is not in the sentence, only the sentence’s own terms contribute. The softmax function then turns the three scores into probabilities. With e≈2.718, Euler’s number, the exponential makes every score positive, and dividing by the sum over all labels j in K makes the three numbers add up to 1:

P(y=k∣x)=ezk∑j∈Kezj,(7)

where y stands for the label of the sentence. The model predicts the label with the highest probability.

Training sets the intercepts and coefficients. Write the training data as pairs (xi,yi) for i=1,…,n, where xi is the vector of training sentence i from (5) and yi its label from the annotators. Let β collect all coefficients and ‖β‖22 be the sum of their squares. Training solves

minβ,b12‖β‖22−C∑i=1nlnP(y=yi∣xi).(8)

The sum is the log-loss. Each of its terms is close to zero when the model gives the correct label a probability near 1, and grows without limit as that probability falls toward 0. The first part penalizes large coefficients, which keeps the model from building its verdicts on a few rare terms. The constant C=1, the scikit-learn default, sets the balance between the two parts.

Table 2 applies (6) and (7) to the vector x from Table 1, whose nine nonzero entries are the only ones that enter the sum in (6).

[ Table 2 · scores and probabilities, opening sentence ]

NegativeNeutralPositive
Intercept bk−0.981.31−0.33
Nine terms, Σt βk,t xt−0.09−0.360.45
Score zk−1.070.950.12
Probability0.0850.6380.277

The nine terms pull toward positive, led by significant and its. The intercept, which reflects that most training sentences are neutral, outweighs them. The word that carries the meaning, headwinds, is not in the vocabulary and never enters equation (6).

Implementation. The code is written in Python. NLTK provides VADER through its SentimentIntensityAnalyzer. scikit-learn provides everything else: TfidfVectorizer implements equations (3) to (5), with ngram_range=(1, 2) for word pairs, max_features=5000 for the vocabulary and sublinear_tf=True for the logarithm in (4); LogisticRegression implements (6) to (8); and train_test_split draws the stratified split. pandas and pyarrow load the dataset from Hugging Face, NumPy handles the arrays and matplotlib draws the result dashboard. The figures in this article come from Python 3.10, scikit-learn 1.7.2 and NLTK 3.10.4. The demo below runs both models in JavaScript, and the export script checks that the trained model in the browser assigns the same label as scikit-learn to every test sentence. Code, data source and the full results are on GitHub: github.com/EmilOhligs/Financial-Sentiment-Analysis.


Both models run in your browser below

Type a sentence from an earnings release or pick one of the examples. The demo runs the same two models used for the numbers above, the trained one with its exact weights, and shows which words drove each verdict. Nothing you type leaves the page.

[ Live demo · two models, one sentence ]

VADER

General-purpose lexicon, no training
–
−1neutral ±0.05+1

Words it found in its dictionary

TF-IDF + logistic regression

Trained on labeled financial news
–

Terms that pushed it toward this label

Loading the models…
Private by design: both models run in your browser. Nothing you type is sent anywhere.
VADER lexicon: Hutto & Gilbert (2014), MIT license. Classifier trained on the Financial PhraseBank (Malo et al., 2014), CC BY-NC-SA 3.0. Code on GitHub


The trained model still misses more than half of the bad news

The headline numbers hide the weakest spot. Of the 33 negative sentences in the test set, the trained model caught 14. It labeled 15 of the rest neutral and four positive. When it did say negative, it was right every time, so its errors come from staying quiet. Negative sentences are the rarest in the data, 13 percent of the total, so the model sees few of them in training, and they are the ones that cost money when a screen misses them.

The opening sentence shows the limit. Faced with it, the trained model does better than VADER and still gets it wrong: it settles on neutral, at 64 percent. A guidance cut, "The bank cut its full-year guidance", also comes out neutral, and "Revenue was flat compared with last year" comes out positive. It reads "Net loss narrowed to EUR 2 million from EUR 9 million" correctly, as positive at 72 percent. A model that counts words and word pairs knows only which terms appeared next to which labels in its training data, so it is as good as the examples it happened to see.

Three limits of the evaluation matter as much. The sentences were split at random, so the model was tested on news from the same period it learned from, while a model in production meets text from the future. All figures come from a single split, without cross-validation, so they carry sampling noise. And this is a baseline. Dogu Araci's FinBERT, a language model adapted to financial text, reported 0.86 accuracy on the full PhraseBank and 0.97 on the sentences all annotators agreed on, under its own split and cross-validation. Those figures are not directly comparable with the ones here, but the distance is large enough to be real.

The failure mode is the part worth remembering. A tool that answers confidently on every input will produce a number for a sentence it cannot read, and the number looks the same as one it got right.

Before any sentiment score feeds a model, a memo or a screen, it is worth asking what text the tool was built on, and then testing it: label a hundred sentences from your own sources, run the tool, and look first at the label that would cost you most to miss. The open question is how much of this gap the newest general-purpose language models close without any training on financial text, and whether their errors are any easier to see.


Emil Ohligs and ThinkBeyondAI built this project as a comparison of two transparent baselines. Its code is released under the MIT license. The Financial PhraseBank is licensed CC BY-NC-SA 3.0, and the demo's trained weights inherit that license. Every number in this article comes from results.json in the repository, which one script regenerates.

If such a project sounds interesting to you, please get in touch with us. We are happy to improve and modify such models to your needs. This model is not an end result, but a research project.


Sources

  • Hutto, C. J., & Gilbert, E. (2014). VADER: A Parsimonious Rule-based Model for Sentiment Analysis of Social Media Text. ICWSM.
  • Loughran, T., & McDonald, B. (2011). When Is a Liability Not a Liability? Textual Analysis, Dictionaries, and 10-Ks. Journal of Finance, 66(1), 35–65. https://doi.org/10.1111/j.1540-6261.2010.01625.x
  • Malo, P., Sinha, A., Korhonen, P., Wallenius, J., & Takala, P. (2014). Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65(4). Dataset card: https://huggingface.co/datasets/financial_phrasebank
  • Spärck Jones, K. (1972). A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28(1), 11–21. https://doi.org/10.1108/eb026526
  • Araci, D. (2019). FinBERT: Financial Sentiment Analysis with Pre-trained Language Models. https://arxiv.org/abs/1908.10063
The link has been copied!