The critique, the replies and the replication record

The reply in one paragraph

In 1999 Statistical Science printed "Solving the Bible Code Puzzle" by Brendan McKay, Dror Bar-Natan, Maya Bar-Hillel and Gil Kalai. People call it MBBK. The authors did not dispute the arithmetic. They reran the test and got the same small number, even a smaller one. Their argument was about where that number came from. They said the result reflects the choices made in designing the experiment and collecting the data, and that nothing in Genesis is needed to explain it.

They asked two questions. Was there enough freedom in the experiment to reach a small number by using that freedom? And is there evidence that the freedom was used? Their paper answers yes to the first and offers circumstantial evidence for the second. They state clearly that they do not claim to know how any bias arose, and they call the unknown process tuning.

Objections to the test

The permutation test asks whether a man's names sit nearer to his own dates than to other men's. MBBK point out that the ranking is not evenly spread once the word list is fixed, because c values that share a word depend on each other. They give a sports comparison. In a long jump on a windy day every athlete has the same chance, but the wind may pick the winner. A low rank might say more about the list than about the text.

One concrete example: the true pairing produces more word pairs than about 98% of the scrambled pairings. That has nothing to do with Genesis. It comes from how many names and dates each man happens to have. MBBK add that they cannot show this explains the result. For the sake of argument they agree something interesting is going on, and they part ways on what it is.

They also report how much rests on a few items. P₂ and P₄ multiply small numbers, so the smallest values dominate.

• Remove the 4 rabbis out of 32 who contribute most, and 1 in 60,000 becomes 1 in 30.
• One appellation out of 102 contributes a factor of 10 alone.
• Remove the five most influential appellations and the result weakens by a factor of 860.

The Harvard statistician Persi Diaconis had proposed a different form of the test: a 32 by 32 table with one combined distance per pair of rabbis, permuted by column. MBBK report that WRR ran another version without his noticing, and that the two obvious readings of his design give results hundreds of times weaker.

Objections to the data

Most of the paper is about the list. The rule for choosing rabbis was mechanical, though MBBK found at least seven selection errors and agree those had little effect. The larger freedoms were elsewhere.

Dates. WRR wrote each date in three forms. Hebrew has at least five more. MBBK ran each form alone, and the result is striking.

The three forms WRR used all do well somewhere. The five they left out do not. Across all 255 combinations of the eight forms, WRR's choice was uniquely the best on two of the four measures, third best on one, and sixth best on another. MBBK caution that the forms are not used equally often, so no firm number should be drawn from that. WRR also included the old ways of writing the 15th and 16th. Dropping them weakens the four measures by factors of 8.7, 2.7, 33 and 5.2.

Names. This is what MBBK call the most serious problem. Rabbis are known by many names and acronyms, and WRR used far less than half of those known. The paper said Professor Havlin prepared the list from a computer search of the Responsa database, and many read that as a mechanical process. MBBK say it was not one: many Responsa names are not in the list, and many listed names are not in Responsa. Menachem Cohen of Bar-Ilan's Bible department studied the lists and said they had no scientific basis. Havlin himself wrote in 1996 that he prepared them alone, by his own scholarly judgment, and that he might decide differently here and there on a second try.

The War and Peace list

This is the best-known part of the reply. Dror Bar-Natan and Brendan McKay took WRR's second list and changed the names. They kept 83 names, removed 20 and added 29. Many changes swap one valid spelling for another. They used the same test, and the same 78,064-letter opening of the Hebrew War and Peace that WRR had used as a control.

The new list scores in Tolstoy about as well as WRR's list scores in Genesis, and it does poorly in Genesis. Menachem Cohen, asked to compare the two, said there was no essential difference between them.

Be careful about what this does and does not show. The list was built on purpose, with the result in view. Everyone agrees on that. MBBK offer it as an answer to their first question: the freedom was large enough. It does not by itself show that WRR's list was built the same way. Witztum replied that the changes break the rules that governed the original list. Writing in 2022, he calls the War and Peace list a declared manipulation aimed at the permutation test, and he reports that a different test shows it much weaker. MBBK answer that those rules were first written down in Havlin's 1996 letter, about ten years after the lists, and that WRR's own list breaks them too.

The study of variations

If WRR's many small choices were harmless, then changing them should help the result about as often as it hurts. MBBK tried about 100 small changes to the method: other distance formulas, other row lengths, other rules for minimal skips, other perturbations. They report that only a small fraction made WRR's result stronger, and then usually by a small amount.

WRR said the bound of 10 was only for saving computation. Yet 10 turns out to sit at a sharp peak. MBBK read the overall pattern as strong evidence for tuning, and they decline to put a number on it. They leave it for the reader to judge. You may object that any finely set test weakens when you disturb it. MBBK take that up: if the effect were real, they argue, there is no reason small changes should hurt more often than they help. Witztum and Beremez published a reply to the study of variations.

A smaller argument concerns results that look too neat. The P₂ values first reported for the two lists differed by a ratio of 1.12. In random splits of the same 66 rabbis the median ratio was about 700, and a ratio that small came up in fewer than 1 split in 100. MBBK note this was noticed after the fact and warn against concluding too much from it. They point to psychology research showing that people expect replications to match more closely than they should.

Attempts to repeat the result

MBBK say their strongest evidence is simple: they cannot find the effect. They hired Simcha Emanuel, a specialist in rabbinic history at Tel Aviv University, who had not seen WRR's lists, to prepare new ones. Here is what came back. A permutation rank near 0 would be striking. A rank near 0.5 is what chance gives.

They also looked at the other books of the Torah, using WRR's lists. Jewish tradition gives Genesis no special standing over the rest.

The figures for Exodus and Numbers on list 1 are low, and a reader should notice them. MBBK note that ranks across books are not independent, since the rank depends on the word list, so a low rank in Genesis may raise the chance of a low one elsewhere. List 2, the only list the 1994 paper tested, stays well above the level of its Genesis result in all four books.

The Hebrew University committee

The most careful test came from a committee, and it included people on both sides. Harold Gans had reported that the 66 rabbis' names stood close to the names of their communities. The result was withdrawn and later reissued. In 1996 a committee formed at the Hebrew University to test the idea. Its chair was Robert Aumann, later a Nobel laureate in economics. Hillel Furstenberg, Eliyahu Rips, Dror Bar-Natan and Isaak Lapides served with him.

They ran two experiments, with place lists gathered by outside experts chosen by Furstenberg. The "fresh" experiment was a new design. The "replicative" one followed Gans's rules as closely as it could.

Those ranks sit in the middle of the pack. Aumann and Furstenberg's report of 2004 says both tests failed to confirm the code. A companion paper has four separate analyses. Aumann and Furstenberg support the report. Lapides and Witztum do not. They argue that the outside experts' data had errors and that community prefixes were left out. Gans and his co-authors make the same case in a 2006 paper and report a strong result from a rule-based data list.

The problem of the text

A sequence depends on every letter between its first and last. Insert or delete one and it is gone. So which text is the right one? WRR used the Koren edition, published in 1962 and based on several sources. MBBK ran WRR's test on other editions of Genesis.

Koren comes out on top for both lists. Every other edition weakens the result, though most still give small numbers. In separate runs, deleting 10 random letters weakened the result by a factor of about 4,000 on average, and 50 random deletions removed it. MBBK explain the gap: the effect varies widely with which letters are hit, and real editions mix insertions with deletions, which partly cancel.

MBBK then press a harder point. The Talmud in Kiddushin 30a records that full knowledge of spellings had been lost. Fragments from the Dead Sea differ from today's text by anywhere from about 1 letter in 1,200 to 1 in 20. No two scrolls from before printing are known to be identical. A proponent can answer that the codes we see are the worn remains of a fuller pattern. MBBK tested this: chance damage over the centuries should have hit long-span sequences hardest, so the surviving effect should sit in short spans. It does not. For list 1, leaving out spans over 2,000 letters left all ranks above 0.15.

What proponents have said since

The research did not stop. Several papers on the subject appeared in the proceedings of the 18th International Conference on Pattern Recognition in 2006. A reader should know two things about that work: it comes from a connected group of researchers, and it holds the main parts of WRR's method fixed.

• Robert Haralick set out a general framework: measures of how tight a table is, a control he named ELS random placement, and a protocol for combining tests. He reports a result on WRR's rabbis under that protocol and says it was developed on the WRR data.
• Gans, Inbal and Bombach derive community names by an algorithmic linguistic protocol, to remove human choice from the list. They report ranks of 4 and 2 in a million for P₃ and P₄ and control runs that come out flat.
• Witztum reports a study of the birth dates of figures from Genesis, and Bombach and Gans report linked phrases and verses.
• Rips and Levitt publish a cluster and list their choices one at a time, then multiply by them. Our gallery entry shows the accounting.

As far as we can find, none of these has appeared in a statistics journal, and no statistician has answered them in one. They stand as stated by their authors.

The most interesting later move is Witztum's 2022 paper in the Bar-Ilan journal B.D.D. He writes that the permutation test was agreed with Persi Diaconis and Robert Aumann when comparing many shuffled texts was beyond the computers of the day. MBBK tell that history differently, as above. He argues the test measures the wrong thing. It asks whether a man's names sit closer to his own data than to another man's. What researchers want to know is the chance of so many close pairs arising at all. So he compares the real text with up to 50,000 copies in which the words of each verse are shuffled, a control WRR used in 1994.

On this test the committee's own fresh list looks strong, with an overall p of 0.00026 by his reckoning. Three things should stay in view. He reports cyclic-shift controls that came out not significant. The War and Peace list also stays low on his test, at 12 in 10,000. And the word list is held fixed in this test, so it cannot speak to how a list was built, which is the heart of MBBK's case. Any list fitted to the real text will tend to beat shuffled copies of that text. No independent group has repeated these runs, as far as we can find.

How to judge for yourself

We are not going to tell you what to conclude about the Torah. We can tell you where the published record stands. The rabbis result is a real number, computed correctly. Tests set up in advance by critics, and by a committee that included a proponent, have come back at or near what chance gives. Proponents dispute the data behind those tests and report strong results of their own. When you read any claim in this field, here are the questions we would ask.

1. Was the list fixed before anyone looked at the text, and is there a record of that?
2. Who chose the spellings? Would another expert have chosen the same ones?
3. How many other forms, skips, books or measures were available, and were they counted?
4. Does the result survive in other editions, and in a text nobody claims is special?
5. Has someone outside the group repeated it with data they gathered themselves?
6. Is the number a rank among controls, or a product of probabilities that assumes independence?

Torah Codex is built so you can ask those questions of your own searches. The laboratory names the edition, counts your tries, lets you lock a search first and ranks results among several kinds of control. Our position on the codes themselves is the one on every page of this site: we measure, and we say what chance would do.