The Importance of Controlled Experiments and the Semmelweis ReflexRonny Kohavi

5/7/2008, updated 5/19/2008, updated 7/13/2008

I wanted to share some stories I collected in the last few months about the importance of controlled experiments vs. other experimental designs in establishing causality. These are summarized in the first part of this document.

The second part provides stories where people reject results because they conflict with their strong-held beliefs. Evidence (e.g., experimental results) that contradict beliefs will be questioned and attempts will be made to find any flaw with the design or the analysis. That’s one reason to build a trustworthy system and spend significant time on analysis and reliability.

Gary Loveman, the COO of Harrah’s, said that there were three ways to get fired at Harrah’s: steal, harass women, or institute a program or policy without first running an experiment (Hard Facts, p. 15). The culture at Microsoft is far away from the Loveman’s culture, but even when experiments will be executed here, there will sometimes be resistance to incorporating the results.

1. Importance of Randomized Controlled ExperimentsHere are two good examples of the importance of running randomized controlled experiments. The first example uses the term “placebo-controlled clinical trial,” which is the medical term for the randomized controlled experiment we use. The second example shows results from a quasi-experimental design, which, like it names implies, is not a real randomized design, but tries to imitate it.

1.1 Hormone-Replacement TherapyThe following story is from the NY Times, May 5, 2002. It’s a story told by Kevin Patterson, an internist.

When I started practicing medicine in the early 90's, one of my enthusiasms was hormone-replacement therapy. At that time, the observation had been made, repeatedly, that postmenopausal women who happened to take estrogen -- for osteoporosis or hot flashes, for instance -- were less likely to have heart attacks and strokes than women who didn't. I remember telling women in their 50's how premenopausal women were relatively immune to cardiovascular disease, at least compared with men, but that once they had been through menopause, this relative protection disappeared quickly. ''Take the estrogen,'' I suggested over and over. ''Preserve your youthful coronaries.''

This was in Manitoba, and these were pragmatic, sensible prairie women. I insisted to them that the recommendations and the evidence seemed clear. I remember my patients' brows knitting at the thought of menstrual cycles extending into their dotage, but ultimately the argument felt compelling. Certainly it did for me. I remembered being told in medical school that the underuse of estrogen was one of the great crimes of the medical patriarchy, itself an expression of latent misogyny. No misogynist I, off I went to work, my prescription pad leaping to hand at the sight of bifocals or pastel cardigans.

Page 1

Semmelweis Reflex

Then in 1998, the results of a formal, placebo-controlled clinical trial called the Heart and Estrogen/Progestin Replacement Study (HERS) were published. It showed that estrogen did not prevent heart attacks or strokes and, in fact, it made women more susceptible to blood clots. The net cardiovascular effect therefore was negative. This study astonished most doctors -- for me, it certainly felt like a betrayal. Betrayed by the recommendations, we had in turn betrayed many of the cardigan-clad women of our acquaintance.

A few months ago, in the emergency room of one of the hospitals I work in on Vancouver Island, I saw a woman in her mid-70's who was still taking Premarin, a common estrogen preparation. She had been having chest pain, and I was admitting her for observation, to make sure she wasn't having a heart attack.

''So, you take the Premarin because . . . ?'' I asked.

''My sisters all had heart attacks in their 50's,''she said. ''My doctor said the estrogen lowered my risk.''

''We now think it probably doesn't.''

''Really.''

''Yes.'' Me, nodding, smiling weakly.

''What changed?''

''Well, there were these studies that seemed to show that women who took estrogen had a relatively low incidence of heart attacks, but it turns out that really, it was the sort of woman who took estrogen who was less likely to have a heart attack. She was probably also less likely to smoke, more likely to seek regular medical attention -- she did something important different, anyway. When, just recently, they took a large group of women and randomly gave each woman either a placebo or estrogen, the ones taking estrogen didn't do at all better.''

''Well,'' she said. ''Isn't that something?''

My patient was not alone. The data from HERS were so surprising that many health-care providers seem not to believe them, even today. In 2001, Premarin was the third most-prescribed drug in the United States.

The key point: it was the woman who took estrogen who was less likely to have a heart attack – a correlation! In 2002, more than 6 million women were taking PremPro [similar to Premarin]. Statistically, that translates into (translation should be taken with a grain of salt since it’s from an attorney site hrt-attorneys.com ):

480,000 additional breast cancer cases. 420,000 more heart attacks. 480,000 more strokes. 480,000 more blood clot cases.

Experimentation Platform Page 2

Semmelweis Reflex

1.2 Twin StudiesThe following story comes from a Nov 2007 article by the Washington Post titled Study Debunks Theory On Teen Sex, Delinquency and the Twin Study article .

A deep study by Ohio State University early in the year found that youngsters who lose their virginity earlier than their peers are more likely to become juvenile delinquents. To reach this conclusion, the authors took into account (technically, controlled for) many variables that could affect the dependent variable (juvenile delinquency). This is commonly called a “quasi-experimental design.” If you believe no other causes other than those controlled for could cause juvenile delinquency, then you should believe the result (losing virginity earlier causes them to be more likely to be juvenile delinquents). This assumption is called Causal Sufficiency.

In the above study, the authors controlled for a range of variables, including:

1. Gender2. Race3. Receipt of public assistance4. Parental education5. Family structure6. Previous substance use and depression7. Importance of religion8. School GPA9. Relative pubertal status10. Virginity pledge status

With such an impressive list of causes under control, and data from a massive database called the National Longitudinal Study of Adolescent Health, who could question the result? The paper was accepted for publication.

But then someone did question the result. Paige Harden, a PhD student from the University of Virginia and her colleagues, used the same database and found 534 same-sex twins. Their study now controlled for genetic and environmental variables, and the result was reversed: earlier age at first sex predicted lower levels of delinquency in early adulthood. Since a twin study practically trumps all other quasi-experimental designs, the new result was published in the same journal. The first paper found a non-causal correlation!

While I shouldn’t make recommendations about early sex, this example is a good example of why controlled experiments are important. Causal Sufficiency is a strong assumption. While you may think you're controlling for all effects of time-of-day, day-of-week, season, geography, etc., there may be other factors that you are not taking into account that may reverse the trend, as was shown in this study.

That's why randomized experimental designs are the gold standard!

Experimentation Platform Page 3

Semmelweis Reflex

2. The Semmelweis Reflex: Rejecting Results that Contradict Strong-Held BeliefsThe following stories illustrate what happens when there is solid evidence that contradicts strongly-held beliefs.

2.1 Ignaz Semmelweis’s Childbed FeverThe story below is from the book Leadership and Self-Deception, the Encyclopedia Britannica, Childbed Fever: A Scientific Biography of Ignaz Semmelweis, and Wikipedia. I sent it to the authors of Hard Facts as a better example of something they discussed in the book. One of the authors blogged about it http://bobsutton.typepad.com/my_weblog/2008/05/the-semmelweis.html and correctly pointed out that controlled experiments are not always possible (e.g., the Yahoo/Microsoft merger). On the web and in services, they *are* possible, so let’s use this opportunity to its full extent.

Semmelweis was a European doctor, an obstetrician, in the mid 1800s. He worked at Vienna’s General Hospital, an important research hospital. The mortality rate in the ward where he practiced was one in 10 – one in every ten women giving birth there died! The reputation of Vienna General was so bad that women preferred to give birth on the street and then went to the hospital. In the book Childbed Fever, they estimated that 2,000 women died each year from childbed fever in Vienna alone, and that in nineteenth-century Europe, childbed fever killed more than a million women.

The collection of symptoms associated with these deaths was known as “childbed fever” or Puerperal fever. More than half the women who contracted the disease died within days. Patients begged to be moved to a second section of the maternity ward where the mortality rate was one in fifty – still horrific, but far better than one-in-ten in Semmelweis’s section.

Semmelweis became obsessed with the problem. He tried to control for all factors, including birthing positions, ventilation, diet, and even the way laundry was done. The one obvious difference between the sections was that Semmelweis’s section was attended by doctors, while the other section was attended by midwives.

After a four-month leave to visit another hospital, he discovered that the death rate had fallen significantly in his section of the ward in his absence. This, coupled with the death of his friend Jakob Kolletschka from an infection led to the breakthrough. Jakob’ contracted an infection after his finger was accidentally punctured with a knife while performing a postmortem examination and his autopsy showed a pathological situation similar to that of the women who were dying from childbed fever. Semmelweis proposed a connection between cadaveric contamination and childbed fever.

Experimentation Platform Page 4

Semmelweis Reflex

Yes, cadavers. Semmelweis spent far more time doing research on cadavers than other doctors.

Vienna General was a teaching and research hospital and many doctors split their time between research on cadavers and treatment of live patients. The doctors in his section performed autopsies each morning on women who had died the previous day, but the midwives were not required or allowed to perform such autopsies. They hadn’t seen any problem with that practice because there was as yet no understanding of germs.

Semmelweis concluded that ‘particles’ from cadavers and other diseased patients were being transmitted to healthy patients on the hands of the physicians. He experimented with various cleansing agents and instituted a policy requiring physicians to wash their hands thoroughly in a chlorine and lime solution before examining any patient. The death rate fell to one in a hundred!

After the initial success, where rates dropped, a new group of students was admitted and the students neglected the washings. Mortality rate increased and Semmelweis instituted stricter controls: the names of students were publicly displayed and assigned to each woman in labor, making it obvious who neglected the washings. Once again, the mortality rate fell (Childbed Fever, p. 53)

What is surprising about this story isn’t the discovery through attempts to control for factors, which led to the unthinkable conclusion (at the time) that there was something invisible that was transferred by the doctors. What is really shocking is how long it took the community of doctors to accept the results.

According to Encyclopedia Britannica, the mortality rate in Semmelweis’s division fell from 18.27% to 1.27% in 1848. That was not enough to generate sufficient recognition and in 1849 he was dropped from his post at the clinic and turned down for a teaching post. Semmelweis spent the next six years at a Hospital in Pest, Hungary, where he reduced mortality rate in the obstetrics department to 0.85% while in Prague and Vienna the rate was still about 10% to 15%.

According to Childbed Fever: A Scientific Biography of Ignaz Semmelweis (p. 69) an 1856 publication in a prominent Viennese medical periodical, Viennese Medical Weekly, by Jozsef Fleischer, a student of Semmelweis, showed success of chlorine washings. However, the editor for the periodical wrote at the end of the report “We believe that this chlorine-washing theory has long outlived its usefulness. The experiences and statistical results of most maternity institutions protest against the views presented above. It is time we are no longer to be deceived by this theory.”

Vienna continued to ignore his recommendations. In 1861, he published a book, but the community rejected his doctrine. In 1865 he suffered a nervous breakdown and was taken to a mental hospital, where he was beaten by asylum personnel and died. It took

Experimentation Platform Page 5

Semmelweis Reflex

another 14 years for the discovery to be accepted, after Louis Pasteur, in 1879, show ed the presence of Streptococcus in the blood of women with child fever. Semmelweis is now recognized as a pioneer of antiseptic policy.

More is available at Wikipedia’s Contemporary reaction to Ignaz Semmelweis.

A 2005 article called Simpson, Semmelweis, and Transformational Change by Grant etal. in Obstetrics and Gynecology claims that despite 150 years of evidence, recent research shows that hand-hygiene practices by healthcare workers remain unacceptably low. Inadequate hand washing is one of the prime contributors to the 2 million health-care-associated infections and 90,000 related deaths annually in the United States.

The Semmelweis Reflex is reflex-like rejection of new knowledge because it contradicts entrenched norms, beliefs or paradigms.

2.2 Bloodletting: The First Clinical TrialThe following story is from the NY Times, May 5, 2002, Childbed Fever, A Physician Looks at the Death of Washington, and from Wikipedia

Since the days of the ancient people, including Mesopotamians, the Egyptians, the Greeks, the Mayans, and the Aztecs, the prevailing conception of illness was that the sick were contaminated by some toxin or contagion. These conditions could be improved by opening a vein and letting the sickness run out – bloodletting.

Once the toxins were gone, the patient immediately felt different, and often better. As anyone who has given blood can tell you, losing a pint or two can make you feel transported, transformed. Intuitively, it was satisfying to doctors that the procedure left the patient feeling drained – physically, emotionally and into the sink.

The practice was continued by surgeons and barber-surgeons. Though the bloodletting was often recommended by physicians, it was carried out by barbers. This division of labor led to the distinction between physicians and surgeons. The red-and-white-striped pole of the barbershop, still in use today, is derived from this practice: the red represents the blood being drawn, the white represents the tourniquet used, and the pole itself represents the stick squeezed in the patient's hand to dilate the veins.

Bloodletting was used to treat almost every disease. One British medical text recommended bloodletting for acne, asthma, cancer, cholera, coma, convulsions, diabetes, epilepsy, gangrene, gout, herpes, indigestion,

Experimentation Platform Page 6

Figure 1: Breathing a Vein" in 1804

Semmelweis Reflex

insanity, jaundice, leprosy, ophthalmia, plague, pneumonia, scurvy, smallpox, stroke, tetanus, tuberculosis, and for some one hundred other diseases (Childbed Fever, p. 6). It was judged most effective to bleed patients while they were sitting upright or standing erect, and blood was often removed until the patient fainted.

Physicians often reported the simultaneous use of fifty or more leeches on a given patient. Through the 1830s the French imported about forty million leeches a year for medical purposes, and in the next decade, England imported six million leeches a year from France alone (Childbed Fever, p. 7).

On December 12, 1799, President George Washington, 68 years of age, rode his horse in heavy snowfall to inspect his plantation at Mount Vernon. It was about 30 degrees Fahrenheit, and he complained about a sore throat, yet rode again the day after. On December 14, he was in respiratory distress. Mr Albin Rawlins, the estate overseer, prepared a medicinal mixture of molasses, vinegar, and butter, and when Washington almost suffocated trying to swallow the concoction, decided on bloodletting and removed 12-14 ounces of blood. Dr. James Craik was brought in, and extracted another 20 ounces of blood, followed by yet another 20 ounces of blood. When a vinegar and hot water solution did not help, he extracted another 40 ounces of blood. In the afternoon, another doctor arrived, Dr. Dick, and he drew another 32 ounces of blood for a total of about 82 to 124 ounces, or 2.5 to 3.7 liters in ten hours. The total blood in George Washington’s body was estimated at 7 liters, so about 35% to over 50% was extracted, which inevitably led to preterminal anemia, hypovolemia, and hypotension. The fact that General Washington stopped struggling and appeared physically calm shortly before his death may have been due to profound hypotension and shock (A Physician Looks at the Death of Washington).

It is understood now that bloodletting only hastened the death of the ill.

We know that bloodletting is unhelpful because a Parisian doctor named Pierre Louis did an experiment in 1836 that is now recognized as one of the first clinical trials, or a randomized controlled experiment. He treated people with pneumonia either with early, aggressive bloodletting or less aggressive measures; at the end of the experiment, Dr. Louis counted the bodies. They were stacked higher over by the bloodletting sink.

Despite the result of the controlled experiment, it took years for bloodletting to be recognized as useful in very limited situations (e.g., in cases involving agitation, it has a sedative effect). Broussais, a well known French physician, continued to recommend leeches, fifty at a time. Since leeches were used repeatedly and in treatment of various diseases, it was possible for the leeches themselves to convey the disease.

Interestingly, Biopharm Leeches, originally established in 1812, is still alive and providing 50,000 leeches for modern surgery. Their tagline: The Biting Edge of Science.

2.3 Police Lineups, Technical Knockout of Morality vs. ScienceWhile the previous stories showed how it took time to learn, the following story (also available at http://jenk.livejournal.com/160588.html) shows that it may take a long time for results to sink.

Experimentation Platform Page 7

Semmelweis Reflex

Experimentation Platform Page 8