The tool worked. The number didn't.

01What it does, and what it refuses to keep

Attach an export from a group chat, paste something one of those people wrote, and the page tells you who. It reads the whole log, builds a model per person, scores your paste against each, and shows how sure it is.

Everything happens in the tab. The file is not uploaded, the pasted message is not sent anywhere, and the verdict is not stored. That is not a privacy paragraph bolted on afterwards — it constrains what I am able to learn about my own tool, and section 08 contains a question I cannot answer because of it. I still think the trade is right.

02The scorer

Naive Bayes over character 2- and 3-grams, additively smoothed:

Character n-grams rather than words because the signal in Korean group chat is in the texture — sentence endings, spacing habits, how many times someone types か, whether they put a space before a question mark. Word tokenisation throws most of that away.

It replaced a compression-based hybrid, and the replacement is the one conclusion in this write-up that has never been retracted. Over 100 real rounds in a five-person room:

Hold on to that 84%. Sections 04 and 07 take it apart.

A mitigation that died twice

When the hybrid first collapsed, it collapsed on a speaker whose messages were split across three accounts, so there was a comfortable reading available: it was not failing, it was confusing one person with themselves. Then two new speakers with one account each scored 26% and 5%, which killed that reading. Later I discovered I had also swapped two speakers' name labels in my notes, which killed it a second time — the original collapse was not even the account-split speaker. The numbers were all computed by corpus size and were never wrong. Only the labels were, and they were wrong for weeks.

03The ceiling is the room, not 100%

Accuracy has no meaning here without the number of suspects: in a two-person room, 50% is a coin. So the page carries a measured grid of what to expect, and refuses to show a number when the number would be a lie.

So the tool applies a rule rather than a special case for small rooms: if the expected edge over chance is under 10 points, it shows a warning instead of a figure. Twelve of sixty-four measured cells trip it, across every room size — a seven-person room with a 20-character paste is just as meaningless as a two-person room, and the old version would have printed 19% and let you believe it meant something.

Two details that cost me time and are worth stating:

• The baseline is computed from the suspects actually on screen, not from the grid key. Snapping a five-person room to the "4" row of the grid must not change the chance level the user is being compared against.
• One cell was a phantom. A single seed put the two-person / 2,000-char / 20-char cell at 39%, below the 50% coin — which, if real, would have been a genuinely interesting finding. A second seed gave 58%. Small rooms have enormous room-to-room variance; the published figure is now a weighted average over 350 rounds from two seeds. I nearly shipped a discovery that did not exist.

04The leak

Here is how the tool is actually used. Export the whole conversation. Then scroll up, find a message, and paste it. The query is inside the corpus. The model is being asked to identify a sentence it has already memorised.

I dismissed this early on. My reasoning was mass: one 50-character message inside a 3.5-million-character corpus is 0.001% of it, far too small to matter. The arithmetic is right and the reasoning is wrong, and the paired measurement says so:

The mechanism is coverage, not mass

Look at the scoring formula again. Every n-gram of the query contributes −log P(g|u), and an n-gram the model has never seen contributes −log(α/den) — a large penalty, because α is 0.1. If the sentence is in the corpus, the number of unseen n-grams is exactly zero, and the true author is the only person in the room paying no penalty at all. It is not about how much text you added. It is about whether anything is missing.

I re-measured this while writing this page, on a completely different Korean corpus — not the game's data, just to isolate the mechanism:

And this is the answer to my own mass argument. Between the first and last rows the query's share of the corpus falls tenfold, while the advantage it confers falls only 3.4-fold. The effect is not proportional to mass, which is why making the corpus bigger dilutes it far more slowly than intuition promises.

The second thing I predicted wrong, in the opposite direction

I then guessed that pasting several short fragments would be harmless, since each is too small to matter. It is the worst case. If all five fragments are in the record, each one collects the full coverage advantage and they accumulate — 100% accuracy even against a million-character corpus, worse than a single long sentence. That bundle of four to seven short messages is exactly the shape of a real paste.

05Removing it: from the owner, not from everyone

My first fix was to subtract the query's n-grams from every model in the room. That is wrong, and measurably so: it takes legitimate counts away from the innocent, which leaves the true author relatively better off. It barely moved — 84% against a 31% baseline.

The correct fix is available for free, and the file hands it over: if the pasted text matches a message in the export, the export says whose message it is. No inference required. Subtract only from that person.

Two honest residuals. Messages are joined with newlines when the model is built, so a handful of n-grams straddling a line boundary survive subtraction; they never changed a ranking in 480 rounds, but they are there. And I suspected the owner-lookup would over-remove, since short phrases are said by several people — measured, that is 3 cases in 500 (0.6%), and zero out of 120 for messages in the 20–60 character band. The suspicion was unfounded.

What does remain is a gap I do not want to paper over: held-out scoring lands at 83% where building the corpus without the message from the start gives 79%. Four to five points of optimism survive the fix, and every number measured with the new version should be read with that discount applied.

06The bench was not running the fix

I built the leak removal, shipped it, and left a note to confirm the local test bench was using it. It was not. The scoring path still read the untouched models, which means all 107 rounds of real-world testing I had accumulated were scored with the leak in place. They are discarded.

Fixing the bench surfaced two traps that would each have produced a quietly wrong comparison:

• Applying it to Naive Bayes only would have rigged the contest. The old hybrid has a compression axis, and a compressor benefits from a memorised string more than a Bayesian model does. Removing the leak from NB alone would have handicapped exactly the scorer I was arguing for. It goes on all five axes, and the cached preparation is rebuilt from the held-out models.
• Subtracting only n-grams breaks the other axes. Two of the five read the joined raw text, not the gram counts, so removing from one and not the other means the axes are looking at different corpora — which is the one thing a comparison harness may never do.

The log now records how many messages were held out. A count and nothing else — never the text, never a name.

And then I lost seven more rounds because the browser had an older copy of the bench open and I only noticed when the new field failed to appear. The file is now versioned in its own name with a version stamp printed under the title, and a test asserts the stamp is there. A tool that can silently be the wrong version is a tool that will be.

07The collapse that was not the scorer

With the fixed bench, thirty fresh rounds. Five-person room, chance 20%:

Confidently wrong, in the same direction, every time. That pattern is not incompetence, it is a systematic input error, but I did not think that first. I hypothesised that additive smoothing was biased by corpus size: the unseen-gram penalty depends on the corpus total, so a smaller corpus is penalised less, and this room ranged from 3.5M to 0.43M characters.

I built a bench with matched corpus imbalance and 100 rounds. No size bias — 80% accuracy with an even spread of choices, and three alternative smoothing schemes all within five points. The hypothesis died. Worse, the log had already contradicted it: the person NB kept choosing was not the smallest corpus in the room. Someone smaller was never chosen once. I could have known before writing the bench by reading my own log more carefully.

The answer was not in my data at all. I asked what had actually been pasted:

8/4/26 is a US-locale short date. The parser knew Korean 2025-12-05 and European 05.12.2025 and did not recognise this, so header extraction failed and the raw text went into the scorer — with the sender's name in it four to six times. The user's devices have different language settings, so the copy format changed depending on which phone the message came from.

The fix widened the date pattern instead of chasing locales one at a time, and handled a second format — the English KakaoTalk copy, which has no separator at all between the name and the message. There is no string rule that splits 4:57 PM Jb come to the coffee shop correctly, because Jb is a name and so is Kyung Min. So the extractor is handed the actual participant list read from the uploaded file and strips the longest match. It is a lookup, not a guess.

Guards, because a net this wide can swallow ordinary text: the format only applies when at least two lines start with a time and they are a majority; the bracket formats always win; and times inside a message are preserved, so a sentence about arriving at 5:36 pm survives intact.

Most importantly, the page now says something when it cannot read the format. Two red warnings appear in the query meter — one for text that looks like a header but parsed as nothing, one for a stripped time with a name that matched nobody in the room. There will be another locale I have not seen. Next time the screen has to speak. This time it said nothing at all, and thirty rounds of measurement went in the bin.

08One speaker really is harder, and I said otherwise first

One of the five sits at 67% where the others are 85–100%. At nineteen rounds I reported that the confidence intervals overlapped and there was therefore no evidence of a hole. That is a bad test, and overlapping intervals do not mean what I used them to mean. Testing it directly:

• 53/60 for the established speakers against 14/21 — Fisher exact p = 0.041
• Matched on query length, 23/25 against 9/15 — p = 0.036

And the obvious explanation is not available. This speaker has 450,000 characters; another has 430,000 and scores 89%. Same corpus size, twenty-two points apart. What is left is that some people's writing simply has less fingerprint — they speak in the phrasing the whole room shares. That is not a data-volume problem, and more logs will not fix it.

The fair reading is not that this scorer is weak here. Every other scorer gets 5–29% on the same speaker while Naive Bayes holds 67%, more than three times chance. It is a hard speaker for everyone, and this is the only method that survives them. The prescription is to accumulate evidence across several independent messages rather than to trust one.

And here is the cost of the privacy decision from section 01: I cannot investigate why. The log keeps numbers, never text, so there is no way to go back and look at what those messages had in common. Answering the question needs a fresh measurement designed for it. I knew that when I chose the design, and I would choose it again, but it is a real price and it should be written down rather than glossed.

09Everything I got wrong

Kept as a list, in order, because the pattern in it is more useful than any single entry.

Two patterns worth extracting.

Not one of these was caught by thinking about it. Eleven fell to a measurement, one to looking at a screenshot, one to asking a question. And the two I reached by pure reasoning — numbers 6 and 10, both of the form "this is too small to matter" — were both wrong, and one of them was wrong in the opposite direction from my prediction. I no longer trust myself to conclude that something is negligible without measuring it.

The last one is a different species. The bench could reject my hypothesis but could never produce the answer, because the answer — what the user had actually pasted — was not in my possession. I ran an experiment where I should have asked a question. When the missing information is not in your hands, measuring harder just gets you a more confident wrong answer.

10So what is the accuracy

I do not know yet, and that is the honest end of this.

The 84% figure came from 107 rounds scored with the leak in place; those are discarded. The next 30 were scored through a broken parser; those are discarded too. In the user's own recent sessions, 26 of 41 pasted messages (63%) were present in the uploaded record, so the leak was partial rather than total, which puts the honest estimate at 70–75% — and even that should be read as four to five points optimistic, per section 05.

The claim I retracted along with the number was more interesting than the number itself. I had concluded that real friend-group chat is easier to attribute than a crowdsourced dialogue benchmark, based on 83–88% against 43–52%. Two errors leaned the same way: the conditions were not matched (different corpus sizes, different query shapes) and the real-world side was leaking. Matched properly, with the leak removed, the benchmark scores 79% and the real room 70–75%. The conclusion was not merely unsupported, it pointed the wrong way.

What survives is the comparison between scorers. All five were run on the same rounds under the same conditions carrying the same leak, so the ranking is untouched even though every absolute figure moved. Relative claims survived what absolute claims did not, which is worth remembering next time I am tempted to publish a single headline number.

Roughly thirty clean rounds are needed before the page can state a figure again. Until then it states none.