Yoseph Eikelman
← Notes

ClinicLog · · 4 receipts

An LLM checks my production health four times a day. It gets counts, not log lines.

ClinicLog runs practice management for therapists in two regions, Israel and the US, and its privacy policy promises that error logs are stripped of personal data before any outside service sees them. A small team can’t read dashboards every morning, and plain threshold alerts either flood you or miss the slow problems: a disk that fills a little every day, an error count that doubles overnight.

So a model reads the health of both regions four times a day.11Runs at 01:00, 07:00, 13:00, 19:00 UTC; last run 2026-11-01 07:00: “All clear. No email sent.” The design question was never “can an LLM judge this”. It was what is it allowed to see, and when does it get overruled.

  1. 01:00 · 07:00 · 13:00 · 19:00 UTC

    About forty commands per host collect from both regions.

  2. Code first

    Plain threshold checks turn the numbers into findings, each with a severity.

  3. Privacy

    Log samples and quoted lines are dropped. They stay in a private file on our servers.

  4. The model

    It reads metrics, findings and recent runs, and answers PASS or ALERT.

  5. The last word

    Code overrules it: any critical finding means ALERT.

  6. The result

    An email goes out only on ALERT.

Code first, model second

About forty shell commands per host, over ssh for the remote region, collect disk, memory, containers, backups (local and offsite), certificate dates, failed logins, intrusion-detection events, database errors. Plain threshold checks turn those into findings, each with a severity.

Only then does the model get involved. It receives the metrics, the findings and recent history, and returns a fixed shape: PASS or ALERT, a summary, and findings with evidence.22scripts/daily-health-gate.py: 1,721 lines, 51 commits since 2026-03-22 The history matters most: thresholds can’t see “this has grown at each of the last seven runs”, a model reading a trend can.

The model never has the last word on a critical

Two rules live in code, not in the prompt:

  • If any deterministic finding is CRITICAL, the verdict is ALERT. If the model said PASS, the summary is prefixed [OVERRIDDEN] so I can see it happened.
  • If the model is down or misconfigured, the deterministic checks decide alone.

The alert goes out, marked [OVERRIDDEN].

The prompt says the same thing, but a prompt is a request. The barrier is an if statement, right after the model’s answer is parsed:

# SAFETY BARRIER: override verdict if deterministic checks found CRITICAL
has_critical = any(
    f["severity"] == "CRITICAL" for f in deterministic_findings
)
if has_critical and verdict.get("verdict") != "ALERT":
    verdict["verdict"] = "ALERT"
    verdict["summary"] = (
        f"[OVERRIDDEN] AI said PASS but deterministic checks found "
        f"CRITICAL issues. {verdict.get('summary', '')}"
    )

Below critical, the model does decide. It may downgrade a warning, and a PASS with warnings sends no email, only the routine chat message. That is the trade: code guarantees the floor, the model filters the noise above it.

It sees counts, not text

Log lines are where personal data hides: bound SQL parameters, exception messages, user names in auth logs. So no raw log line goes to the model. Any field ending in _samples is dropped, and a finding’s “first matching line” is cut before it is sent. That is a naming convention, not a detector: a new field that carries text has to follow it.

Example data
Invented names and documentation IP ranges.

The model learns that there were three SSH logins by users nobody expected. Which users, and the lines themselves, stay in a private history file on the server that runs the gate. The alert email points there instead of quoting them.

Five tests plant a patient name or an IP address in the input and assert it never comes out, including one that checks raw findings from history never reach the model.3314 tests in test_daily_health_gate_redaction.py; 5 plant a name or IP and assert it never reaches the model

The false alarm that taught me the most

On 22 July the gate reported a critical: the certificate had expired. Nothing was wrong with the certificate. One of the forty ssh connections the gate opened back to back was reset during the handshake, the certificate check came back empty, and the code stored “check failed” as minus one days. The model read minus one as expired. The threshold never fired: it only looked at 0 to 7 days, so it would have missed a certificate that really had expired.44False critical of 2026-07-22 and its fix, documented at the call site Replay it in the diagram above, under 2026-07-22.

Two fixes went in that day. The gate’s metric commands now share one multiplexed ssh connection. And a certificate check that fails is reported as unknown, a warning, never as minus one, while a negative number is now its own critical. In October the same rule reached the login and malware counts: a failed count of logins isn’t “no logins”, it’s “I couldn’t look”. I am still applying it check by check.

What I’d keep for any client

  • Let code set the floor; let the model read the trend. Thresholds for what must never be missed, a model for what moves slowly.
  • Send counts, keep text. Most of the judgment survives without a single raw line.
  • A failed measurement is not a measurement. “Unavailable” has to be a value the whole pipeline understands.
  • Silence must be earned. It emails only on ALERT; a green run is one message in a chat channel, and slow trends wait for one weekly report.

Receipts

  1. 1Runs at 01:00, 07:00, 13:00, 19:00 UTC; last run 2026-11-01 07:00: “All clear. No email sent.”
  2. 2scripts/daily-health-gate.py: 1,721 lines, 51 commits since 2026-03-22
  3. 314 tests in test_daily_health_gate_redaction.py; 5 plant a name or IP and assert it never reaches the model
  4. 4False critical of 2026-07-22 and its fix, documented at the call site