Can a 600M-Parameter LLM Learn Sentiment Analysis from a Larger Model?

Can a large LLM teach a much smaller LLM to perform the same task?

Teaching Qwen3-0.6B

We fine-tuned Qwen3-0.6B, a model with only around 600 million parameters, on approximately 14,000 sentences from Project Gutenberg. Sentiment labels and explanations were generated by a much larger 31B-parameter LLM.

The small model learned to return a sentiment score from 1 (very negative) to 10 (very positive) together with an explanation in JSON format. Even this relatively small experiment produced a noticeable change.

The most interesting result is Kendall's correlation. It increased from only 0.13 to 0.46, approaching RoBERTa's 0.50. This is important for sentiment arcs. If we want to follow how the emotional tone of a novel rises and falls, getting the relative ordering of sentences right may be almost as important as predicting their exact numerical scores.

What does the tiny model actually say?

Some outputs are remarkably reasonable.

For:

“The sail was patched with flour sacks and, furled, it looked like the flag of permanent defeat.”

the model gives 1/10, explaining that expressions such as “permanent defeat” create imagery of ruin and defeat.

For:

“Everything about him was old except his eyes and they were the same color as the sea and were cheerful and undefeated.”

it gives 10/10, correctly focusing on “cheerful” and “undefeated” and interpreting the sentence as strongly positive.

It also handles neutral narration reasonably well:

“I remember,” the old man said.

Score: 5/10. The explanation identifies it as a largely neutral descriptive statement.

But the failures are perhaps even more interesting.

For:

“No,” the old man said.

the model gives 1/10, despite simultaneously describing the sentence as essentially neutral. It then invents a strong sense of “disappointment or frustration” that is not really present in the sentence.

An even better example is:

“The brown blotches of the benevolent skin cancer the sun brings…”

The model gives 10/10, apparently being attracted by words such as “benevolent,” “sun,” and “tropic sea” while failing to properly balance them against “skin cancer.”

These examples reveal the main weakness of the current model: it still has a strong tendency toward extreme scores. Yet its explanations often show that it understands considerably more about the sentence than the numerical prediction suggests.

Why is this promising?

Before fine-tuning, Qwen3-0.6B was essentially unusable for our sentiment analysis. Its numerical predictions were unstable, and sometimes its score and its own explanation contradicted each other. After seeing only about 14,000 synthetic examples, the model became substantially more coherent. Its exact scores are still much less accurate than RoBERTa's, but the ordering of sentiment became surprisingly good.And unlike RoBERTa, the model can tell us why it made a prediction. This suggests an interesting middle ground between extremely fast classifiers and much more capable—but expensive—large LLMs. A tiny generative model will probably never be as fast as BERT, but it may eventually offer LLM-style interpretation at a much more manageable computational cost.

What next?

This is a very raw experiment, not a finished sentiment model. Our current training set is small, and extreme sentiments such as 1/10 and 10/10 are relatively rare. We now want to generate a substantially larger and better-balanced synthetic corpus and see whether the model's tendency toward extreme predictions can be corrected. Still, increasing Kendall's correlation from 0.13 to 0.46 with only ~14,000 examples is encouraging. It suggests that instead of running a huge LLM over every sentence in a corpus, we may be able to use the large model as a teacher—and then let a much smaller model do most of the actual work.

The model is still experimental, but if you would like to play with it, check the page.