How we used a large language model to train compact sentiment analysis models specifically for literary texts, and what we learned from the results
Why do we need another sentiment analysis model?
Sentiment analysis has become a common instrument in computational literary studies and digital humanities. However, there is one important problem: most existing sentiment analysis models were not designed for literary texts.
Literary language differs from other linguistic domains not only in its formal characteristics but also in the ways emotions are expressed and perceived. Irony, ambiguity, narrative context, and indirect emotional expressions are common in fiction. We believe that these characteristics can affect how sentiment should be interpreted, and that models trained on other domains may not always capture them adequately.
Most widely used sentiment analysis models were trained on rather different types of data, like social media posts, movie reviews, and financial news. The quality of their annotations also varies considerably. Some datasets rely on several human annotators, while others derive sentiment labels indirectly from user reactions or other signals. Such annotations may be useful for their original purposes, especially if you create a fast reaction trading model percepting any minor change in financial news, but their applicability to literary studies remains questionable.
This raises a simple question: Can we create a sentiment analysis model specifically for literary texts without spending enormous resources on manually annotating a new dataset? - Digital Humanities researchers are not pepper sacks, unfortunately.
Recent developments in large language models (LLMs) suggest that we can.
Can large language models teach smaller ones?
Recent studies have demonstrated that LLMs can perform sentiment analysis directly through zero-shot or few-shot prompting. In zero-shot prompting, we simply instruct the model to evaluate sentiment without providing examples. Few-shot prompting additionally provides several examples to demonstrate the expected output.These approaches are flexible and allow researchers to define their own sentiment scales and annotation criteria. However, they also have an important disadvantage: computational cost.When analyzing hundreds of thousands or millions of sentences, repeatedly running a model with tens of billions of parameters becomes expensive and time-consuming. Meanwhile, compact encoder models, such as BERT or BART, can process the same corpus much more efficiently.But what if we could combine the advantages of both approaches?
There is a well-established machine learning technique called knowledge distillation, in which a larger teacher model transfers some of its learned behavior to a smaller student model. In our experiment, we used this idea to develop several compact sentiment analysis models for literary texts. In our previous research, we found that Gemma 4, a 31-billion-parameter model operating in its non-reasoning mode, demonstrated promising performance on literary sentiment analysis among the open-weight models we evaluated.
We therefore used this model to annotate approximately 10,000 randomly sampled sentences from Project Gutenberg. Each sentence received a sentiment score on a ten-point scale, from 1 (most negative) to 10 (most positive). Using this synthetic dataset, we fine-tuned several compact transformer models, ranging from approximately 10 million to 300 million parameters. Finally, we evaluated these models on a separate, human-annotated dataset based on Ernest Hemingway's The Old Man and the Sea. We compared their predictions with those of the teacher model and two widely used RoBERTa sentiment analysis models developed by Cardiff NLP.
The results were encouraging. In some evaluation settings, our smaller models even outperformed their much larger teacher, including in mean absolute error (MAE), which measures the average distance between predicted sentiment scores and human annotations.
Of course, this does not mean that a small model has become generally more capable than its teacher. Rather, it suggests that a compact model can successfully learn a specialized task from synthetic annotations and, under certain conditions, generalize better to an independent evaluation dataset.
But how should we measure sentiment in the first place?
Our experiment also addressed another question, which we consider particularly important: the architecture of sentiment prediction itself.
Many conventional sentiment analysis models treat sentiment as a classification problem. For example, a three-class model predicts whether a sentence is negative, neutral, or positive. A five-class model may additionally distinguish between very negative and very positive expressions. In conventional classification, each class is represented as a separate category, and the model learns to predict the probability of belonging to each category.
However, there is something peculiar about sentiment: its categories are not independent. They have a natural order, which follows from a very common sence. A very negative sentence is closer in sentiment to a moderately negative sentence than to a very positive one. Yet a conventional classification loss does not necessarily account for this distance. We can also approach this question from the perspective of human perception.
Consider a small experiment. Read two sentences and try to estimate their sentiment on a scale from 1 to 10. How do you arrive at your answer?
We suggest that people do not necessarily possess an absolute internal measure of sentiment. Instead, we often make relative judgments. Is the sentence positive or negative? Is it strongly negative or only slightly negative? Is it closer to neutrality? In other words, sentiment evaluation may be understood as a series of comparisons along an ordered scale.
This is why we became interested in ordinal regression (for sure, this idea is not our invention and it proved itself of sentiment analysis models for other domains).
Unlike conventional classification, ordinal regression explicitly considers the order of possible outcomes. Rather than treating all sentiment scores as unrelated categories, the model learns a series of ordered decisions. For example, on a ten-point scale, the model can learn to answer questions such as:
Is the sentiment greater than 1?
Is the sentiment greater than 2?
Is the sentiment greater than 3?
And so on, across the sentiment scale.
By combining these decisions, we can estimate the final sentiment score while preserving the ordinal structure of the task. We implemented this approach in our models, and the results suggest that it is a promising alternative to conventional sentiment classification. In particular, our models demonstrated strong performance on sentences with clearly positive or negative sentiment. Despite being trained on a relatively small synthetic dataset, they performed better in our evaluation than the selected general-purpose models trained on much larger datasets from other domains.
This does not establish that ordinal regression is universally superior. Nevertheless, our findings suggest that explicitly modeling sentiment as an ordered variable may be useful, especially when researchers need more detailed sentiment gradations than simple positive–negative classification.
What does this mean for digital humanities?
We see two important implications of this experiment.
First, the architecture of sentiment analysis matters. Sentiment is not simply a collection of independent categories. Treating it as an ordinal variable may better reflect the structure of the annotation task and possibly some aspects of how people evaluate emotional intensity. Our results provide empirical support for exploring this approach further, although more experiments are necessary to establish its advantages across different datasets and literary genres.
Second, domain-specific sentiment analysis may be much more accessible than previously assumed. Researchers do not necessarily need enormous manually annotated datasets or expensive computational infrastructure to develop specialized models. A capable LLM can generate synthetic annotations according to a researcher's own criteria, and these annotations can subsequently be used to fine-tune much smaller models.
The computational requirements are also relatively modest. Generating annotations with a large model may take one to several days on consumer-level hardware, depending on the model and available memory. Once the dataset has been prepared, fine-tuning a compact encoder model may require only minutes or hours.
This opens opportunities not only for literary sentiment analysis but also for other specialized tasks in digital humanities, where existing models do not adequately reflect the characteristics of the material under investigation. At the same time, synthetic annotations are not equivalent to human judgments. The student models inevitably inherit some of the teacher's assumptions, limitations, and possible biases. Independent evaluation on human-annotated texts therefore remains essential.
Our experiment was an attempt to investigate how far this approach can take us with a relatively small dataset and accessible computational resources. The results suggest that even compact models can be effective when their training data and prediction architecture are carefully adapted to the research task.
Try our models
We have made our models available on Hugging Face (scroll untill below of HF page to see all models), where you can try them, examine their evaluation metrics, and use them in your own research. We are also preparing a preprint describing the methodology, model architectures, experiments, and evaluation results in greater detail. We have tried to make the project as transparent and reproducible as possible, including the limitations of the synthetic dataset and the models themselves.
We hope that these resources will be useful for researchers interested in literary sentiment analysis, computational literary studies, and the broader application of compact language models in digital humanities.
Very finally, our experiment suggests a rather practical conclusion: we do not always need larger models to obtain better results. Sometimes, a smaller model trained for the right task is exactly what we need.