Yoseph Eikelman
← Notes

ClinicLog · · 5 sources

I built a scribe to give therapists their evenings back. A model may add the commas, but code decides whether it may change a word.

A therapist who sees clients back to back writes the notes afterwards, often in the evening, from memory. ClinicLog gives that evening back. With the client’s consent, the therapist records the session on a phone. An hour in Hebrew, English or both becomes a transcript, then a draft note in the therapist’s own template. The therapist reads both, fixes what is wrong, and signs the note.

A speech recognizer plus a model told to tidy up fails twice here. Israelis switch language mid-sentence, and on the first day a recognizer set to Hebrew turned the English half into nonsense. And a tidied transcript is a falsified record: told not to tidy, a model still turned “not especially” into “especially”.

EXAMPLEDone · what the therapist reads

Two invented transcript lines in Hebrew with English in the middle. A model adds punctuation to both; code finds the same words and keeps it. A model then proposes repairs. On the first line it drops “not” from “not especially stressed”; code refuses it, and the line goes back to the words heard. On the second it changes the misheard “exorcise” to “exercise”; code keeps it, marked with the word that was heard.

  1. 1
    Punctuation. The model may add commas and full stops. Code strips punctuation and spacing from both versions and compares the words. If one word differs, the line goes back to what was heard.
  2. 2
    A lost negation. A repair that drops a word on the negation list is refused, and the line goes back to the words heard. The list holds English and Hebrew words: not, never, לא and more. Hebrew forms with a prefix letter, such as שלא, are not on it yet.
  3. 3
    A repair that passes. It is kept and marked, and the word that was heard stays attached to it.
Example. The lines and the model’s edits are invented; the verdicts are what ClinicLog’s checks return for these exact lines. The English under each line translates it.

The calls I made

Code, not the prompt, decides what the model may change. A stricter prompt was the easy answer, and I turned it down: the prompt already said not to tidy, and the model dropped the “not” anyway. So the model writes, and plain code with no model in it checks every line it returns. The punctuation pass may add marks but not change a word. The repair pass may fix a misheard word, but not drop a word on its negation list. A line that fails goes back to the words that were heard, on its own; the other lines keep their edits. A repair that passes stays marked, so the therapist can see what was heard.

I measured on real voices, and went against my own test data. On that first day I switched to a multilingual recognizer, because a Hebrew-only one cannot hear the English half. My synthetic Hebrew clips then said Hebrew-speaking practices would do better on the Hebrew-only model. Public recordings of real voices said the opposite: it was worse on three of four Hebrew clips and tied on the fourth. For English, the single-language model did cut word errors by about a third.11Measured 30 Aug 2026 on public recordings: English 9–23% word errors fell to 6–17% English-only; Hebrew-only was worse on 3 of 4 Hebrew clips, 5.7% to 8.7% at most So a practice that declares English gets the English model, and one that declares Hebrew keeps the multilingual one. Trusting the synthetic answer would have sent Hebrew practices to the worse recognizer.

More context, fewer guesses. The repair model works out what a garbled word meant from the lines around it. Given 20 turns at a time, it proposed seven repairs on the public test recordings and turned “compassmented” into “compartmentalized”. Given 60, it proposed two and wrote “compos mentis”, which is what was said. Less context made it guess more, not less, so repairs travel 60 turns at a time.22102 tests on the punctuation check, repairs, speaker roles and template split, passing 1 Nov 2026. Repair batch, 30 Aug: 7 edits with 20 turns of context, 2 with 60

What happened

Recording the session itself has been on in Israel since 24 August 2026, and stays off in the US; dictated notes came first, in November 2025.33Dictated notes since 9 Nov 2025; session recording on in Israel since 24 Aug 2026, off in the US. That day a mixed clip: 2.4% word errors multilingual, 26.8% Hebrew-only I am ClinicLog’s only engineer. Synthetic evals of the note and the speaker roles, on short invented exchanges in Hebrew and English, pass 14 of 14.44Synthetic evals, re-run 1 Nov 2026: 7 invented exchanges of 3 to 5 turns, Hebrew and English, each with name masking off and on: 14 of 14 passed, 1.2 to 3.5 s a call Those are synthetic and say nothing about clinical accuracy. That is what the therapist’s review and signature are for.

Under the hood

A language model makes four passes over a session, and code decides what each may produce: punctuation, repairs, speaker roles, and the note.

  1. In the room

    The phone records the session and uploads it in pieces while it runs.

  2. Speech service

    The voices are separated and Hebrew and English are transcribed in the same sentence. Our copy of the audio is deleted once the service accepts it.

  3. Model, then code

    A model adds punctuation. Code compares the words before and after, and a turn that changed goes back to what was heard.

  4. Model, then code

    A second pass may fix a misheard word. A change that drops a listed negation is refused; every change that passes is marked.

  5. Who is who

    A model reads the opening of the session and says which voice is the therapist. It answers with roles, never names.

  6. The draft

    The note is written from the text, into each section of the therapist's own template, in one call.

  7. The therapist

    They fix lines and voices, edit the draft, and approve. The note stays a draft until they finalize it.

How the pipeline survives a lost worker is its own note. This one is about the words.

The punctuation check

Both versions of a turn are split into words, everything Unicode classes as punctuation or spacing is removed, and the two word lists must match. A space inside a word fails; a new paragraph does not. The check, and its caller:

def is_insertion_only(original: str, revised: str) -> bool:
    """True when `revised` differs from `original` only in punctuation/spacing.

    One character of difference in the compared stream returns False, which
    the caller must treat as a rejection and fall back to the raw text.
    """
    return fidelity_tokens(original) == fidelity_tokens(revised)
        for text, candidate in zip(batch, candidates, strict=True):
            if is_insertion_only(text, candidate):
                refined.append(candidate)
            else:
                logger.warning("verbatim_readability_rejected")
                refined.append(text)
                rejections += 1

The first version ignored invisible formatting characters. A review before the pass went live wrapped a number in a Hebrew sentence in a direction override: the check passed it, and a browser shows the digits reversed. Models add direction marks of their own unasked, so invisible characters are compared now. And on the first real recording the model deleted the dashes that mark a measured pause; to Unicode a dash is punctuation. Pauses are now compared like words.

The repair guards

Negations, from a Hebrew and English list, are counted before and after and must match. A repair may change at most half of a turn’s words, or two in a short one, and never all of them. The changes are computed from the text, never self-reported, and each one reaches the therapist and the note model with the word that was heard.22102 tests on the punctuation check, repairs, speaker roles and template split, passing 1 Nov 2026. Repair batch, 30 Aug: 7 edits with 20 turns of context, 2 with 60

Language, roles and the template

On go-live day a synthetic mixed clip scored 2.4% word errors on the multilingual model against 26.8% Hebrew-only, and I switched that evening.33Dictated notes since 9 Nov 2025; session recording on in Israel since 24 Aug 2026, off in the US. That day a mixed clip: 2.4% word errors multilingual, 26.8% Hebrew-only Automatic language detection is no answer: it picks one language per file. On a two-person interview from the public set, multilingual scored 5.7% against 8.7% Hebrew-only.11Measured 30 Aug 2026 on public recordings: English 9–23% word errors fell to 6–17% English-only; Hebrew-only was worse on 3 of 4 Hebrew clips, 5.7% to 8.7% at most

The speech service separates voices without knowing whose they are. A model reads the first 40 turns, each cut to 240 characters, and may answer therapist, client or other, never a name.55Speaker roles read the first 40 turns, 240 characters each. Shipped 24 Aug 2026, never ran on two voices until the next evening's fix; a test now checks the schema Two therapists or two clients is hedging, and the answer is dropped. The therapist flips a wrong voice with a tap.

The note is written from the transcript, never the audio. Audio costs more than text, and when I gave the repair pass the audio too, it stopped repairing and started transcribing again: word errors on a public clip went up, not down. Until 21 August a structured template got the whole note in its first section. Now the response schema is built at run time with one property per section, described by the template’s own guidance for that section, so the plan lands in the plan section in the same call.

The roles failure that looked like a cautious model

Speaker roles shipped on 24 August. The next day the worker log showed they had never run. I had declared the answer as a mapping of voice to role, and the model provider’s library refuses that schema shape before sending anything. My code caught the error, as designed, returned nothing, and the screen showed “Voice 1” and “Voice 2”: exactly what a model that could not tell looks like. Every test faked the call.

The answer is now a list of pairs, and a test checks the schema for the shape the provider refuses. The working call then exposed two holes the broken one had hidden: the recognizer’s bucket for unattributed words could be labeled client, and the hedging check caught two therapists but not two clients. Both were closed that night.55Speaker roles read the first 40 turns, 240 characters each. Shipped 24 Aug 2026, never ran on two voices until the next evening's fix; a test now checks the schema

What transfers

  • Check the model’s output in code, item by item. A prompt is a request; a check runs every time, and a line that fails reverts alone.
  • Measure on real voices before trusting synthetic ones. The synthetic clips would have sent Hebrew practices to the worse recognizer.
  • A fallback that degrades to nothing needs a test of its own. Otherwise the failure looks like a model being careful.

Sources

5 sources behind the numbers above
  1. 1Measured 30 Aug 2026 on public recordings: English 9–23% word errors fell to 6–17% English-only; Hebrew-only was worse on 3 of 4 Hebrew clips, 5.7% to 8.7% at most
  2. 2102 tests on the punctuation check, repairs, speaker roles and template split, passing 1 Nov 2026. Repair batch, 30 Aug: 7 edits with 20 turns of context, 2 with 60
  3. 3Dictated notes since 9 Nov 2025; session recording on in Israel since 24 Aug 2026, off in the US. That day a mixed clip: 2.4% word errors multilingual, 26.8% Hebrew-only
  4. 4Synthetic evals, re-run 1 Nov 2026: 7 invented exchanges of 3 to 5 turns, Hebrew and English, each with name masking off and on: 14 of 14 passed, 1.2 to 3.5 s a call
  5. 5Speaker roles read the first 40 turns, 240 characters each. Shipped 24 Aug 2026, never ran on two voices until the next evening's fix; a test now checks the schema