I had Gemini train its own replacement for $9

What I set out to do

The plan had three steps. Have Gemini label a few thousand Reddit comments once, marking every brand, model and steel. Train GLiNER on those labels. Then run GLiNER on my own machine for every comment after that, and stop calling Gemini.

Gemini labeled 4,290 comments for $9, or $0.0021 a comment. That means the trained model pays for itself at roughly comment 4,291, as long as later comments are about the same length and it runs on a GPU I already own. The test of success was simple: on 225 comments the model had never seen, how often does it tag the same words Gemini tagged? One catch to keep in mind for every score in this article. Nobody checked Gemini's labels by hand, so the model is graded against Gemini, not against the truth. Where Gemini was wrong, the model gets marked right for copying the mistake and wrong for fixing it.

The approach

Gemini labeled the comments through OpenRouter at temperature 0 in 25 minutes. One prompt decision mattered more than the rest: I never ask the model for character offsets. It counts characters badly and returns spans off by two or three positions. The prompt asks for the exact substring and a label, and TypeScript finds the offsets. If the string is not in the comment, the entity is dropped and logged.

// The model returns strings. Code computes the offsets.
{ "entities": [
    { "text": "Benchmade", "label": "knife brand" },
    { "text": "940", "label": "knife model" },
    { "text": "S30V", "label": "knife steel" }
] }

Product names are full of punctuation a generic tokenizer splits, so a regex keeps VG-10, CPM-154 and 1.4116 whole and emits every other non-space character as its own token. Spans that still miss a token boundary are dropped rather than guessed. About 30% of the training set is comments that contain a known false-positive trigger (gyuto, carbon, handle, patina) and no product, labeled as empty. Before the second run I set aside 225 comments as a validation set and never touched them again. Training ran on a Tesla T4 on Modal with the HF Trainer.

per_device_train_batch_size = 2
gradient_accumulation_steps = 8
learning_rate = 1e-5
threshold = 0.45

What went wrong

For five runs the model learned nothing. The first three were configuration, and anyone using the HF Trainer with GLiNER will hit them in an afternoon.

Runs 4 and 5 were the expensive ones. GLiNER's tokenize_inputs crashed on broken Reddit emoji, so I had patched it, and the patch has to fill a tensor called words_mask. It sits next to attention_mask, has the same shape, and every attention mask I have ever built is ones for real tokens and zeros for padding. I built it that way. Nothing about the name or the shape says otherwise.

Training ran to completion. Loss started around 130, drifted to about 70 and stayed there. No crash, no warning, no NaN, gradients of ordinary size, checkpoints saved on schedule, eval F1 near zero. I blamed the label list first, because run 4 also had negatives with no labels set. Fixing that and rerunning gave the same flat loss. The only thing wrong with run 5 was a tensor I had never looked at.

How I fixed it

I read GLiNER's training loop instead of its docstrings. words_mask is not a mask. It is a word index: 0 for special, prompt and padding tokens, then 1, 2, 3 for the first sub-token of each real word. The span-scoring head uses it to pool sub-tokens back into words. Filled with ones it says the whole comment is a single word, so the model is asked to find brand and material spans inside one enormous token. It cannot, and the loss says so without saying why.

# what I wrote           # what GLiNER expects
words_mask = [1,1,1,1,1]  words_mask = [0,1,2,2,3]
                          #  [CLS] Mazaki wh ##ite #2

With the index fixed, run 6 learned on the first try. The rest was tuning against the locked set. The 209M medium model reached 0.800; the 459M large model, which fits a T4 only with gradient accumulation, reached 0.83. Ten times more adversarial negatives (510 instead of 51) dropped F1 to 0.799, so run 10 went back to 51. One threshold per class instead of a global cutoff took material recall from 0.787 to 0.911, because MagnaCut, S35VN and HAP40 need context, not string matching. Every large run bottoms out at epoch 2 and overfits after; with 2,000 examples that is a dataset-size problem, and early stopping is the fix.

What I learned

It worked. The model runs locally, matches Gemini's labels at 0.83 F1 on comments it never saw, and knows that "carbon steel" is a category rather than a steel and that PM2 sometimes means the Spyderco Paramilitary 2 and sometimes is just letters. One earlier run scored 0.879, but on a random split, and two of my early "regressions" turned out to be split noise, so I do not quote it.

On paper the project cost less than lunch: $9 of labels, $2.50 of GPU. The real budget line was the days spent on a tensor that passed every check the code had and was still wrong. I think that is the normal case. In small fine-tuning jobs the model and the data are rarely the problem; the plumbing between them fails in ways that look like a hard dataset. If I had to choose between a better label set and an assertion on every tensor I hand-build, I would take the assertion.

This model powers New Knife Day, which tracks what knife people on Reddit are buying and arguing about. The knife-side write-up there has the full run log. Both are linked below.

At a glance

Problem

Find the brand, model and material names in Reddit comments, and skip the generic words around them, without paying an LLM per comment

Approach

Have Gemini label the comments once, then fine-tune GLiNER large v2.5 (a DeBERTa-v3-large encoder) on those labels and run it locally

Result

0.83 F1 on a 225-comment validation set fixed before training; material recall 0.911 with a per-class threshold

Cost

$9 in Gemini labels plus about $2.50 of T4 time across ten runs

Stack

TypeScript and MongoDB for the scraper and labels, Python and PyTorch for training on Modal, FastAPI to serve the model