Controls: what each one keeps and what it breaks

What a control is for

A control is a text, or a rearranged version of the text, where you run exactly the same search and compare. If your result is just as common there, the result does not point to anything about the original. If it is rarer in the original than in nearly all the controls, the result is unusual for that comparison. Unusual is as far as a control can go. It tells you how a number ranks, not why.

Every control keeps some features of the text and destroys others. That is the idea to hold on to. A control that keeps more of the text sets a harder, fairer test. A control that keeps very little makes almost any natural pattern look surprising.

Letter shuffle

This is the simplest control. Take every letter of the scope and put them in random order. The letter counts are exactly the same, so the expected count formula still applies, and the mean of many shuffles should sit close to it. It does. For תורה forward at skip 50 in Genesis the formula gives 2.215, and 1,000 shuffles averaged 2.347.

Each run uses a numbered seed, so anyone can repeat it. WRR used one shuffled text, called R, and a letter shuffle is what runs when you press "Run 25 shuffles" in the laboratory.

Letter shuffles overstate surprise at skips of 1 to 3. Real Hebrew has strong patterns between nearby letters, such as common prefixes and word endings. A shuffle wipes them out, so a real text can look "too good" at short skips for ordinary reasons of language. Always pair a shuffle with a text-level control there.

Word and verse shuffles

These keep more of the language. A word shuffle moves whole words around and leaves spelling intact. A within-verse word shuffle keeps every word in its own verse and changes only the order in the verse. WRR built this control in 1994 and called it U. It was suggested by a referee. A verse shuffle keeps each verse whole and moves the verses.

The within-verse shuffle has become the center of a real argument. In 2022 Witztum proposed it as the main test for word lists, in place of the permutation test. It keeps a text's short-range statistics almost entirely, which is its strength. Its limit is that the word list stays fixed. A list that was fitted to the real text, by any process, will beat shuffled copies of the real text. That control tests the text and leaves the list unexamined. See the critique for the numbers.

ELS random placement

Robert Haralick's ERP control does not touch the text. It takes the sequences you found and slides each one to a random start, keeping its skip, length and letters. Then it asks how tight the cluster is. If the real cluster is tighter than nearly all the slid ones, the sequences are closer than position alone would make them.

ERP is fast and it needs no search, which is why cluster studies use it. Its limit is what it takes as given. It treats the set of found sequences as fixed, so it says nothing about how the set was chosen. If you looked through a thousand words and kept the six that clustered, ERP sees only the six. We do not let ERP stand alone for a hand-picked cluster. The bench always shows a second family beside it.

Other books and other editions

A shuffled Genesis is not a natural text. A different book is. Other books keep the letter patterns of real Hebrew, and they test the idea that the effect belongs to one text. WRR used Isaiah and a Hebrew War and Peace of the same length as Genesis. In our lab, "Compare with a book" runs your search in a book of similar size. Good comparisons for Genesis are Numbers and Exodus: all three have between 63,000 and 79,000 letters.

Other editions test something else: whether a result depends on a handful of letters. MBBK's Table 4 runs the rabbis test on seven editions of Genesis, and the numbers move a lot. We search one edition, so we cannot run this for you, and every result names that edition. Treat a result as a statement about this text. One letter's difference inside a span removes the sequence.

Permuting the pairing

This is the control WRR used for their main test. The text and both word lists stay as they are. Only the matching of names to dates is scrambled. It is a sharp test of one question: do these names go with these dates better than with other dates from the same list? It can only be used when your terms come in natural pairs and you have a good number of them. MBBK's objection is that the rank depends on the list as well as the text, so a low rank may reflect the list.

Generated text

A Markov model builds a fake text from patterns of k letters in the real one. It is a reasonable extra check, but no paper in the published record uses it. If we show it, we label it as a supplement.

How to read a rank

Our lab reports a control as a count: how many runs out of how many matched or beat the real result. Here are five worked results, all from the gallery, each with 1,000 letter shuffles of its book. The seeds are 1 to 1,000.

Exodus stands out, with only 6 shuffles in 1,000 reaching 8. That looks strong until you remember how it was picked. The term, the skip and the book were all chosen because of a pattern someone had already noticed. Try enough combinations and one will land there.

• Name the family. "Rank 6 in 1,000" means nothing until you say among what.
• Note the resolution. With N runs you cannot see anything finer than about 1 in N. Zero of 1,000 means under about 1 in 1,000, and it does not mean impossible.
• Count your tries. A rank belongs to one locked question. Take the best of 20 and divide your surprise by about 20.
• Read it as a rank. "Rarer than 99% of shuffles" is an accurate statement. "99% chance this is real" is a different claim, and no control can back it.

One more thing. When two families disagree, ask what each one keeps. The disagreement usually tells you what the result depends on.