An AI read the same competitor page four times and counted 18 departures, 18, 18, then 1. Now each page it reads gets up to three readings, and it believes what two agree on.

My Trip is a group-travel company. Its operations staff need to know when a competitor adds a departure, sells one out or changes it, and nobody has time to open every competitor’s site twice a week. So a pipeline does it and mails the changes. An AI model does the reading, because most pages are free-form Hebrew, not tables. The line it must never get wrong is “sold out”: a false one says a competitor’s trip is full when it is still open. Trusting a single reading did not survive measurement: the same page, read four times, gave answers that did not all match.

EXAMPLEFive kept on two readings instead of three.↻

Competitor page: Fjords and Northern Lights, 8 days · invented

Sunday run

DateReading 1Reading 21 of 5Reading 3wrong rowReported

3 Febopennot foundopenopen 2 of 3

17 Febsold outnot foundsold outsold out 2 of 3

3 Maropennot foundopenopen 2 of 3

17 Maropenopensold outopen 2 of 3

31 Maropennot foundopenopen 2 of 3

Each departure is in at least two readings, so all five are kept. 17 Mar stays open, two readings to one.

Wednesday run

DateReading 1Reading 2identicalReading 3not boughtReported

3 Febopenopennot readopen 2 of 2

17 Febsold outsold outnot readsold out 2 of 2

3 Maropenopennot readopen 2 of 2

17 Maropenopennot readopen 2 of 2

31 Maropenopennot readopen 2 of 2

The first two readings match in every field, so the third could not change the vote and is not bought.

The calls I made

Measure the model before blaming the sites. On 8 August I captured each page once and had the model read the same bytes four times. One page read 18, 18, 18, then 1 departure, another 15, 15, 1, 1, and on a third a row’s status flipped to “waitlist” and back, which the feed counts as sold out.118 Aug 2026: four readings of byte-identical pages. Two pages read 18, 18, 18, 1 and 15, 15, 1, 1 departures; one row's status flipped to waitlist The bytes had not changed. The noise was the model’s.

No table parser in place of the model. Taking the model out was the obvious fix. I checked first: the two collapsing pages had no departures table at all. A day later a count across all sources found real tables on other sites, so a parser now reads those and checks the model against them. It does not replace the model, because on some of those sites the status is free text, not a table cell, and the status is the line that matters.

Pay three times, believe two. From that day each page was read three times, and a departure was reported only when two readings contained it; where they disagreed on a detail, the version most readings gave won. I did not wait for a better single reading, a sharper prompt or a structured read: that is the real fix, and it is still owed. The vote made the feed survivable in the meantime and roughly tripled the cost of a run.

Buy the third reading only when it can change the answer. Three days later: if the first two readings are identical, a third cannot change a two-of-three vote, so it is skipped.

What happened

The same 45 sources, same bytes, before and after:22Measured 8 Aug 2026, 45 sources, same bytes: varying sources 4 to 2, run totals 165/165/151/135 to 165/166/164, no false sold out, about 3x the cost

sources still varied on identical bytes, down from 4

2 of 45

false "sold out" lines that day, down from 1

0

the cost of a run, accepted

3×

Single readings totaled 165, 165, 151 and 135 departures; the vote gave 165, 166 and 164, where the healthy readings had landed. It absorbs noise rather than quietly reporting less. In the run of 1 November, 178 of the 383 pages it read stopped after two identical readings.33Runs twice a week. 1 Nov 2026: 383 pages read, 178 settled after two identical readings, 205 needed a third; 15–19% saved over three runs

Under the hood

The pipeline captures each page once, then reads it with the same model up to three times. A departure is identified by its dates, not its price, guide or status, because those are exactly the fields that wobble: keying on them would turn every wobble into a new departure and defeat the vote. Where a date holds several departures, which happens on catalog pages, its position among them is part of the key. The winning version is the one most readings produced, and ties go to the earliest reading, so the same readings always give the same answer.

The rule I care about most came from a failing test. If two of three readings fail outright, demanding two votes throws away everything the third read, and the page reports zero departures. Zero is also what a competitor with nothing on sale looks like. So the bar is capped at the readings that survived:

    # Never require more agreement than there were passes. With two of three
    # passes failing outright, demanding two votes drops every departure the
    # survivor read, and the page reports zero — indistinguishable from a
    # competitor who publishes nothing, which is the confusion 045 is about.
    # A bad day must not be able to impersonate a competitor's empty shelf.
    threshold = min(CONSENSUS_MIN, len(passes))

Readings only one pass saw are dropped, and the count goes on the page’s record beside the number of readings behind it. A departure backed by two of three readings is a different claim from one read once, and the record says which.

The 8 August failure

None of it was invented. Both status words are on the page with the flipped row: “registering” four times, “waitlist” once. The model attached the wrong row’s cell. One fault sat behind all three symptoms: on long lists it loses track of which row it is on, some of the time. The vote makes that survivable, not correct; a better reading is still owed.

Skipping the third reading

“Identical” uses the vote’s own idea of sameness: every field, in the same order. Two readings with the same dates at different prices do not agree, and the third reading breaks the tie. The guard also takes the number of readings left, so with five readings two matches would no longer end it early. My estimate was an upper bound of about a third of the reading cost. Production says less: on 1 November, 205 of 383 pages needed a third reading, so about 15% of readings were saved. The two runs before saved 18% and 19%.33Runs twice a week. 1 Nov 2026: 383 pages read, 178 settled after two identical readings, 205 needed a third; 15–19% saved over three runs The tests replay the measured collapse and the misread status, and four more cover the skip.4410 tests hold the vote and the skip rule, all passing on 1 Nov 2026. The skip was also checked against 4,000 random sets of three readings The more interesting number is the 205: on about half the pages, two readings of the same bytes differed somewhere, in a field or in the order of rows.

What two readings cannot catch

A vote catches a reading that wobbles, not a model that is wrong the same way twice. Nor does it vote on the status alone. The unit is the whole row, so if the two readings that say “open” also differ on a price, all three versions tie, and the first reading wins even when it is the one that says “waitlist”. On 11 August, building a test set for the reader of trip details, which runs the same believe-what-two-agree vote, turned up 14 records with a trip length the page never printed; several had been agreed by two readings.5511 Aug 2026: 14 records carried a trip length their page never printed, several agreed by two readings. A check now drops a length when the page text states none More readings were not the fix. A plain check now drops a trip length when the exact text the model was given states no length at all.

What transfers

• Measure the model against itself first. Reading identical input several times is cheap and tells you how much of the noise is yours.
• A failure must not look like an answer. Cap the vote at the readings that survived, so a bad day cannot read as an empty shelf.
• Agreement is not truth. Where a claim can be checked against the input, check it in code.