Evaluation of the Critical Factors in thePhylogenetic Analysis of Human andNeanderthal mtDNAChristopher R. Moradi ∗, and Alexander Schuster ∗

∗San Francisco State University Department of Mathematics, 1600 Holloway Avenue, San Francisco, CA 94132

Submitted to Proceedings of the National Academy of Sciences of the United States of America

We report the results of the first phylogenetic analysis of the re-lationship of humans and Neanderthals with the inclusion of theEl Sidron and Monte Lessini sequences. By performing multipleanalyses on several smaller data sets, we provide evidence of theeffects of the inclusion/exclusion of sequences, the substitutionmodel, the tree construction technique, and the branch supportmethod on the separation of these groups. We find no support forthe separation of humans and Neanderthals based on the HVRIand HVRI+II sequences presently available. In addition, we pro-vide evidence that the high support for separation found in pre-vious studies is due to deficiencies in the phylogenetic analysesperformed.

Neanderthal mtDNA | human evolution | phylogenetics

Introduction

The Neanderthals are an extinct group of archaic humans from theMiddle/Upper Pleistocene that have caused a great deal of con-

troversy, confounding researchers ever since their fossils were dis-covered in the early nineteenth century. The controversy deals withthe ultimate fate of the Neanderthals. Did they make a major contri-bution to the gene pool of anatomically modern humans, or did theysimply fade into oblivion without having any genetic effect on us atall, or does the true answer lie somewhere between these two ex-tremes? Was there evolutionary continuity between the Neanderthalsand modern European populations, or were the Neanderthals com-pletely replaced by the modern humans? This question has implica-tions for the more general debate over the origins of modern humans.The proponents of the Multiregional Continuity Model ([1, 2]) holdthat modern human populations evolved from the exodus of Homoerectus from Africa approximately one million years ago. On theother hand, those who favor the Out-of-Africa Model ([3, 4]) claim amuch more recent African origin for modern humans. They posit thatthese anatomically modern Africans spread all over the Old Worldand replaced the descendants of the earlier African emigrants, whichinclude the Neanderthals.

Researchers on both sides of the argument have claimed thatthe fossil evidence supports their respective position. Some claimthe existence of fossils that provide a bridge between Neanderthalsand anatomically modern humans, while others scoff at this position.Many anthropologists agree that the fossil evidence is inconclusive.(See [5] for a review of the fossil data.)

In 1997, a new type of evidence emerged that researchers hopedwould shed more light on the situation; Krings et al. ([6]) pro-duced the first hypervariable region one (HVRI) mitochondrial DNA(mtDNA) sequence from Neanderthal remains. Additional sequenceshave since been produced by Krings et al., as well as other researchgroups. As of this writing, four unique HVRI-only sequences withmore than 300 sites have been extracted from Neanderthal remains,and two sequences with both the hypervariable regions one and two(HVRI+II) data have been extracted (Table 1).

The early sequencing groups analyzed the available Neanderthalsequences in relation to human and chimpanzee sequences and con-cluded that there was a strongly supported separation between thesegroups. The papers that present the phylogenetic analysis of the

relationship of humans to Neanderthals based on the mtDNA dataare summarized in Supplemental Table 1. Comparisons of the av-erage pairwise differences ([6, 7, 9, 8, 10]) and pairwise distances([14]) between Neanderthal and human sequences were used to sup-port the separation of these groups. The authors of [6, 7, 9]) foundhigh support for separation using likelihood-mapping (LM, [16]) ofneighbor-joining (NJ, [15]) trees. In [8], the authors found similarsupport with bootstrapped ([17]) NJ and maximum parsimony (MP)trees. As a result of these findings, recent papers reporting new Ne-anderthal sequences ([12, 13, 18]) have limited their analyses to de-termining the placement of new Neanderthal sequences with respectto the other Neanderthal sequences and estimating the genetic di-versity of this population; phylogenetic analyses of these new Ne-anderthal sequences in relation to human sequences have not beendone by these recent sequencing groups. Although the sequencinggroups have not continued to analyze these data, [19] used Bayesiantechniques to estimate the phylogenetic relationship between thesegroups. Their analyses showed 100% posterior probability (PP) sup-port for the separation.

While consensus had rapidly formed that Neanderthals did notmake any genetic contribution to modern humans, supporting theOut-of-Africa model, several studies have sharply criticized thisconclusion. Questions have been raised about the authenticity ofthe sequences–primarily of FE1 ([6])–by several papers ([10, 19,20, 21, 22, 23]). In addition, opponents argue that the HVRI andHVRII regions are inadequate to resolve these phylogenetic ques-tions ([20, 24]). Guti«errez et al. ([22]) criticized the use of pairwisedifferences because of biases in the human sequence pool and theuse of LM because it would overestimate branch support levels. Aspart of their own analysis of the sequences, they found no support forseparating these groups based on bootstrapping NJ trees or using theinterior branch test ([25]) on NJ trees.

This research is directed towards reanalyzing the human and Ne-anderthal data with the inclusion of the new Neanderthal sequences(SI2 and MLS). In addition, we examine the critical factors for thephylogenetics analysis of these data to determine the stability of thebest-fit model selection, and the support for separating these groupswith neighbor-joining and Bayesian inference. We also evaluate theseparation by comparing the likelihood scores of maximum likeli-hood (ML) trees containing a human monophyly (HM) against thosethat do not. Finally, we provide evidence that the high support values

Reserved for Publication Footnotes

www.pnas.org — — PNAS Issue Date Volume Issue Number 1–6

for separation obtained in several studies are due to deficiencies inthe analyses performed, rather than high support given by the data.

Results and DiscussionModel Selection.The model selection process implemented withPAUP* [26] and MODELTEST [27] was unable to find a consistentbest-fit substitution model for these data across all data sets. Forall selection criteria used, the selected model varied in complexityfrom HKY+Γ ([28]), to the most complex model available in PAUP*,GTR+Γ+I ([29]). The results of the model selection tests are shownin Supplemental Table 2.

The differences in human sequences included in each data setled to the selection of different models. While the use of a Jukes-Cantor (JC, [48]) NJ base tree is typically considered adequate forestimating the parameter values and the likelihood scores for eval-uation in MODELTEST ([30]), this substitution model may be fartoo simplistic to estimate an adequate base tree for these sequences.Alternatively, the lack of data (short sequence lengths) likely con-tributes to the inability of this method to find a consistent best-fitmodel. We found that the Bayesian information criterion (BIC, [31])chose the simplest model in nearly all tests, which matches the find-ings of [32]; the Akaike information criterion (AIC, [33]) generallyselected the most complex model for the small data sets, while thehierarchical likelihood ratio test (hLRT) generally selected the mostcomplex model for the large data sets.

While we found no single best-fit model for the HVRI and HVRIIregions of human, Neanderthal, and chimpanzee mtDNA sequences,we did find parameters that should be present in the model. Firstly, allof the models selected over all criteria included the three parametersto specify unequal base frequencies. Secondly, all of the selectedmodels included the gamma shape parameter (with α < 1), andnearly all (152 of 160) included the proportion of invariable sites pa-rameter. This indicates that the sequences show a significant amountof rate heterogeneity by site that must be taken into account for anaccurate analysis of the data. Finally, all models selected includedat least two rate categories (e.g., transitions occur at a different ratefrom transversions).

Although the authors of [6, 7, 9] did not explicitly state the modelthat they used for tree construction or branch support, Guti«errez etal. ([22]) stated that the model was F84. This model would be insuf-ficient to determine an accurate tree as well as to find accurate branchsupport values, because it neglects the high levels of rate heterogene-ity exhibited in the HVRI and HVRII regions of human mtDNA.

Bootstrapped Neighbor-joining Analysis.The bootstrapped NJtrees constructed show that there is a lack of support for the sepa-ration of humans and Neanderthals based on the HVRI and HVRIIsequence data presently available. We did find 100% support fora chimpanzee monophyly (CM) with all data sets. In addition,we found high (above 90%) bootstrap proportion (BP) values for amonophyly of the Neanderthals; however, the placement of this cladewith respect to the human sequences was not well supported. Figure1 shows the BP values of the human monophyly (HM) when all siteswere included in the analysis (full), as well as for the data sets fromwhich sites with a majority of missing and ambiguous data were re-moved (trimmed). The support fell below 20% for nearly all data setscontaining all sites once rate heterogeneity was included in the sub-stitution model. Analysis of the trimmed data showed an increase inthe BP support over the BP values of the full data set. However, whenan adequate substitution model was used (e.g., F84+Γ+I [34, 35]),the support values for nearly all data sets were still below 30% and60% for the HVRI and HVRI+II sets, respectively.

Our results seem to match those found by Guti«errez et al. ([22]),but are much lower than those found by Ovchinnikov et al. ([8]).Guti«errez et al. reported BP support for an HM of 29% for theHVRI data and less than 32% for the HVRI+II data set. Ovchin-

nikov et al. reported 95% support for an HM for their HVRI-onlydata set. Guti«errez et al. attributed this difference to the inclusionof sequences with missing/ambiguous data and the use of correcteddistances instead of ML distances by the other researchers. Our re-sults indicate that the inclusion of missing/ambiguous data has theopposite effect on the NJ bootstrap analysis of these data–loweringthe support values instead of increasing them. Instead, the differencein BP support values found in these studies are likely the result ofa bug in PAUP* when corrected distance formulas are used in con-junction with rate heterogeneity parameters. When the distances arecalculated in PAUP* using the distance formulas, the gamma shapeand proportion of invariable sites parameters are silently ignored–reducing TN+Γ to TN. We verified this by repeating a bootstrappedNJ analysis with various gamma shape values using corrected dis-tances. All tests showed identical support values for an HM (41.6%)in the resulting consensus tree. However, when ML distances wereused, the BP support values of an HM decreased with a decrease inthe gamma shape parameter (41.6% to less than 5%).

Several human sequences were identified as divergent using oursimple analysis of the NJ bootstrap group frequencies. The followingare some of the sequences grouped with the chimpanzee sequencesto the exclusion of all Neanderthal sequences: GenBank accessionnos. AM711903, EF184590, AY632995, AY195777, and M76254.These are all African sequences. While these sequences were cho-sen because they showed the highest support for grouping with theCM, the support values were comparable to the support for an HM,including in some cases higher support than the support for an HM.

Finally, we repeated the NJ analyses on the data sets with thehighly ambiguous sequences removed. The BP support for an HMvaried slightly from those of the full data sets; however, there was noindication that the values tended to increase or decrease due to theremoval of the ambiguous sequences.

Bayesian Tree Inference.The Bayesian phylogeny constructedshowed no support for the placement of a Neanderthal clade outsideof a human monophyly. The results of our initial analysis showedhigh support for a Neanderthal monophyly (NM), but no support forthe HM. However, on closer inspection we found that a human se-quence that was basal to the Neanderthals and other humans in theconsensus tree had a large number of missing sites. This raised theconcern that sequences that contained a high percentage of missingsites were artificially “pulled” toward the root sequences. To removethis factor, we replaced four sequences that were highly ambigu-ous with four sequences that were divergent sequences (AM711903,EF184590, AY632995, and AY195777) from the NJ analyses. Thisnew data set was run under similar settings. Figure 2 shows the con-sensus tree from this analysis. While 100% PP support was foundfor NM and CM, no support (less than 5%) was found for an HM.Although the support of the NM, CM, and HM match those of theNJ analyses, the Bayesian inference found eight human sequencesthat grouped with the NM with 57.5% PP support. Six of thesesequences were available in GenBank with the following accessionnos.: AY714050, AF392241, AY633004, AY714040, AJ401501, andAJ401519. The other two sequences are available through HVR-Base++ with the sequence names CP-L9 ([36]) and Na12 ([37]).

In order to resolve the discrepancies between our results withthose found [19], the analysis was repeated twice more. In the firstof these the same set of sequences were used; however, the ambigu-ous sites excluded from the analysis. While no support for an HMwas found after excluding missing/ambiguous sites, the branch join-ing the Neanderthal sequences with the same eight human sequencesreduced in PP support to 45.2%. In the second analysis, the four di-vergent sequences were removed from the data set from which theambiguous sites had been excluded. Again, we were unable to findsupport for an HM, in conflict with the findings of [19]. The sameeight human sequences were grouped with the NM with 60% sup-port.

2 www.pnas.org — — Footline Author

Topology Tests. GARLI found trees with similar likelihood scoreswhen both a positively and a negatively human monophyletic con-straint was used. If there existed a strong phylogenetic signal thatshowed the separation of the Neanderthals and humans, we wouldexpect a significant improvement in the likelihood score of the treewith the positive HM constraint over the negatively constrained treefor all of our data sets. For all data sets analyzed, the Shimodaira-Hasegawa (SH) test as implemented in PAUP* showed that neithertree was significantly better than the other. While there may be aconcern that two runs we performed for each data set is not enoughto adequately exam the tree space, we estimate that the ratio of posi-tively constrained HM trees to the set of all trees is 2.6× 10−19 (seeSupplemental data). Therefore, the search is more likely to miss anoptimal tree without an HM than one showing an HM. Because therewere no data sets that showed a significant difference in the likelihoodscore of the two constrained trees, this reinforces the findings of ourother analyses–showing that the data available does not support theseparation of these groups.

Additional Comments on Prior WorkPairwise Differences. Several studies analyzing the relationship ofhuman and Neanderthal sequences ([6, 7, 9, 8, 10]) used the num-ber of pairwise differences (PWD) between the human and Nean-derthal sequences to show the separation of these groups. In additionto the concerns over bias in the sequence database ([22]), we wouldlike to point out additional concerns with the use of pairwise differ-ences comparisons. Firstly, this analysis technique fails to acknowl-edge that substitutions are not created equal; there are high levels ofrate heterogeneity in the HVRI and HVRII portions of the human D-loop. Some changes will be much more common than others, andtherefore add less to the divergence of these sequences. However, bysimply counting the number of differences between two sequences,we are assigning each change an equal weight. Secondly, we be-lieve the comparison of PWD is much more susceptible to errors inthe sequence than phylogenetic methods. Several papers have ques-tioned the authenticity of the FE1 sequence ([22, 20, 10, 21, 19]).In particular, [10] raised concerns over the four substitutions at posi-tions 16107-8 and 16111-2, that are unique to the FE1 sequence andnot present in the other Neanderthal sequences. Additionally, post-mortem artifacts are expected to occur more frequently at sites withhigher evolutionary rates ([38]). Hence, these errors will have lesseffect when distance-based or ML trees are constructed than changesin regions with low substitution rates. However, both of these typesof substitutions will have equal effect on the pairwise difference cal-culations. With phylogenetic analysis, sequence errors will affect thebranch lengths, but may not be enough to affect the placement of thesequence in the tree.

Likelihood Mapping. We believe that in [6, 7, 9], Krings et al. usedlikelihood mapping (LM, [16]) in an incorrect manner to establishbranch support for their NJ tree. Guti«errez et al. ([22]) criticizedtheir use of LM by showing that it was susceptible to long-branchattraction (LBA). We agree with this critique since Krings et al. re-ported their results using three basins of attraction for the mappingtriangle. However, TREE-PUZZLE ([39]) also provides results us-ing a mapping triangle divided into seven basins. This additionalpartitioning of the triangle helps to mitigate the LBA effects noted in[22]; the center triangle shows the percentage of sequence compar-isons with a star signal (all quartets have similar likelihood scores),while the three rectangular side sections show the sequence compar-isons with a network-like signal (two quartets show similar likelihoodscores). Figure 3 shows an LM analysis we performed on a dataset selected because the results approximate the results presented in[6] (only the FE1 sequence was included in the analysis). When weconsider the mapping triangle with three basins, there appears to be88.4% support for grouping the Neanderthal sequence with the chim-

panzee sequences. However, when we examine the seven basins out-put triangle, 15% of the quartets show an inconclusive signal fromthe sequences (at least two quartets show similar likelihood scores).In addition, 4% of the comparisons yield quartets that show a strongpairing of the Neanderthal sequence with one human sequence.

In addition to the problems with using the three basin mappingtriangle, we believe that Krings et al. ([6, 7, 9]) incorrectly groupedthe taxon to determine branch support levels. From the data pro-vided it appears that the support for the NM in Figure 7b of [6]was determined by selecting sequences from three groups: chim-panzees, the Neanderthal, and all humans. We believe that supportfor this branch should have been determined using sequences fromfour groups: chimpanzees, the Neanderthal, the four Africans near-est the Neanderthal, and the remaining humans. The authors did notelaborate on the method used for obtaining the other interior branchsupport values. It is unclear what the LM branch support values pre-sented in [6, 7, 9] indicate. They seem to show that the Neanderthalsequences are not deep within the human clade; however, this doesnot to seem to be an argument that anyone is purporting. Supple-mental Figure 1 shows the results of an analysis in which we usethe methodology of Krings et al. to compare a divergent Africansequence (GenBank accession no. AF346992) to chimpanzee andother randomly selected human sequences. There is 94.6% supportfor a strong grouping of this sequences with chimpanzee sequences;however, in the seven basin map we can see that 2.8% of the compar-isons reflect a strong grouping of this sequence with another human.While the support for separating this sequence from other human se-quences is strong using this methodology, this does not indicate thatthe African sequence is from a separate species.

In contrast, we used one of our small HVRI data sets to find thethe LM support for a branch separating the chimpanzee sequencesand African sequence AF346992 in this set from the Neanderthalsand 199 other human sequences. In order to determine the supportfor the tested branch in Figure 4, we used four groups for our LManalysis: chimpanzees, Neanderthals, the basal African, and the 199other human sequences. This analysis shows (Figure 4) that 84.2% ofthe quartets show a strong signal pairing the African sequence with achimpanzee sequence.

Divergence Time of Humans and Neanderthals. Many of the pro-ponents of the separation of humans and Neanderthals have used theestimation of the most recent common ancestor (MRCA) as evidenceagainst intermixing ([6, 7, 9, 40, 8]). However, we believe that in-adequate phylogenetic analyses invalidate these estimates. Firstly, akey to the MRCA calculation is that the correct phylogeny has beenestimated; our results as well as those of others ([22]) show that theseparation of these groups is unsupported. Secondly, the studies havefailed to use either a relative-rate test ([41]) or the likelihood ratio test([32]) to support the assumption that the data fits a molecular clock([42]). Thirdly, these studies ignore a main source of error in theestimation of the confidence interval for the divergence times whichis the stochastic variation in the clock rate ([43]). Finally, in [6] theauthors used a simpler substitution model (F84) for the estimation ofthe phylogeny and a much more complex model (TN+Γ+I, [50]) toestimate the distances used to calculate the MRCA.

Data and MethodsData Set Construction.Sequences were obtained through HVR-Base++ ([44]; http://www.hvrbase.de), which is an updated versionof the same repository used in many of the previous studies of hu-man and Neanderthal mtDNA. From this database, all available man-ually aligned HVRI and HVRII sequences were retrieved and sortedinto human, Neanderthal, and primate sequences. Additional Ne-anderthal and chimpanzee sequences were retrieved from GenBankand manually aligned to the other sequences. Sequences were com-pared (ignoring ambiguous and missing sites) and duplicates were

Footline Author PNAS Issue Date Volume Issue Number 3

discarded, keeping the sequence with the fewest missing/ambiguoussites. HVRI sequences with matching HVRII data were joined toform the HVRI+II sequences. The HVRI and HVRI+II pools con-tained 11,265 and 3,791 human sequences, respectively; after remov-ing the duplicate sequences these pools contained 4,471 and 2,597human sequences, respectively.

From each sequence pool, large and small data sets were cre-ated for use in our analyses. Thirty large data sets were created fromthe HVRI and HVRI+II sequence pools. For the HVRI pool, theseconsisted of 500 randomly selected human, 6 chimpanzee (three Panpaniscus and three Pan troglodyte), and 6 Neanderthal sequenceseach (FE1, FE2, MEZ, V75, MLS, and SI2); for the HVRI+II pool,these consisted of 500 human, 6 chimpanzee, and 2 Neanderthals se-quences each (FE1 and V75). In addition, 50 small data sets werecreated using the same Neanderthal and chimpanzee sequences, ex-cept that these contained only 200 human sequences each.

Because of the open questions concerning the influence of am-biguous data in phylogenetic analyses ([22, 45, 46, 47]), we re-peated experiments with the ambiguous sites excluded and highly-ambiguous sequences removed. For the “trimmed” data sets, the endsof the full data sets were removed from HVRI and HVRII regionsso that only the regions that contained data for the Neanderthal se-quences were included in the analysis. In addition, columns wereremoved where the majority of sequences showed either a gap or anambiguous site. Since this process was performed separately on eachdata set, some of the resulting sets included more sites than others.The minimum number of sites in the trimmed sets were 360 for all ofthe HVRI data sets, and 645 and 659 for the small and large HVRI+IIdata sets, respectively. Finally, we created a third group of data sets,which we will refer to as the “deleted-sequence” data sets, by exclud-ing all sequences with a large number of ambiguous sites from thefirst 10 data sets of each of the small and large HVRI and HVRI+IIfull sets.

Model Selection. The model selection process described in [32] wasapplied to each full data set (all sequences and sites), using PAUP*4.0b10 ([26]) and MODELTEST 3.7 ([27]). The best-fit model foreach data set was determined using three selection criteria: hLRT,AIC, and BIC. For BIC, 438 and 875 were used for the number ofcharacters parameter for the full HVRI and HVRI+II data sets, re-spectively. The defaults were used for all other MODELTEST set-tings.

Bootstrapped Neighbor-joining Analysis. In order to resolve theconflicting results of [8] and [22], we performed neighbor-joining(NJ) bootstrap analysis of our data sets for seven different substitu-tion models: JC ([48]), JC+Γ, F81 ([49]), F84 ([34, 35]), F84+Γ+I,TN+Γ+I ([50]), GTR+Γ+I ([29]). By using the multiple small andlarge data sets, we sought to determine the impact of the human se-quences included on the resulting bootstrap porportion (BP) supportvalues. While most phylogenetic analyses are done with a single sub-stitution model, we used seven different models to quantify the influ-ence that specific model parameters have on the support of a CM,NM and HM. For each data set, an NJ tree was constructed based onML distances with the model parameters fixed to pre-specified values(see Supplemental Table 2). Using this tree, the model parameterswere re-estimated and used to construct a final NJ tree. A bootstrapanalysis was performed on the final tree with 500 replicates, and thegroup frequencies were used to find the bootstrap proportion (BP) forthe desired branches. In all cases, ties were broken randomly in theneighbor-joining construction. For models with a gamma distributionspecifying rate heterogeneity, four rate categories were used. Identi-cal analyses were done on the trimmed and the deleted-sequence datasets. For all bootstrap analyses, a seed value of 12345 was used so

that comparisons could be made between tests while removing thevariability due to replicate construction.

In addition to collecting the BP support values for the threemonophyletic groups of interest, the group frequencies from the boot-strap analyses were used to identify divergent human sequences. Inthe first case, we searched for the highest grouping of human se-quences with the CM to the exclusion of Neanderthal sequences. Inthe second case, we looked for the highest grouping of human se-quences with the NM to the exclusion of the chimpanzee sequences.The human sequences that showed one of these pairings most fre-quently across all data sets and substitution models were considereddivergent sequences.

Bayesian Tree Inference. We used MrBayes 3.1.2 [51, 52] to inves-tigate the support for the separation of humans and Neanderthals un-der the Bayesian paradigm. We limited our analyses to one of thefifty small HVRI data sets that contained all sites and sequences. Wechose to analyze the HVRI set over an HVRI+II set because it al-lowed us to include all 6 Neanderthal sequences. In addition, while[19] concluded that their HVRII-only analysis showed strong supportfor the separation of humans and Neanderthals, the 63% PP supportthat they found for an NM is too low to support this conclusion. Since[19] found high support for a human monophyly using a larger set ofhuman sequences for their Bayesian inference, we selected a set thatcontained a divergent African sequence (GenBank no. AF346992).Similar to [19], we used the defaults of MrBayes and GTR+Γ+Ias the substitution model. The analysis was run for 25 million gen-erations, sampling every 1000 generations and discarding 25% forthe burn-in phase. We verified that the two runs converged based onthe average standard deviations of the split frequencies and PotentialScale Reduction Factors.

Topology Tests. Bootstrapped neighbor-joining has produced con-flicting results with regards to analysis of human and NeanderthalmtDNA ([8, 22]), which have led some to speculate ([19]) that thistechnique is inadequate for these data. In addition, Bayesian infer-ence is often criticized because of the priors chosen for parametervalues ([53]). Because of the concerns regarding these methods, wedecided to add a ML tree search to our test methods. Instead of usingbootstrap to determine branch support, we sought to examine the dif-ference in the likelihood scores of the best tree under the constraintthat the human sequences form a monophylic group to the exclusionof Neanderthal sequences against the best tree that did not containa human monophyly. GARLI 0.95 ([54]) was used for these testsbecause of its ability to rapidly find good ML trees under topologyconstraints. For each data set, we searched for the best ML tree underan HM constraint, and then for the best tree under the negative of thisconstraint (i.e., a tree that does not contain an HM). This was donefor only the small HVRI and HVRI+II data sets because of the timerequired to find each tree. Two runs were performed for each dataset, with the best tree kept for each type of constraint. F84+Γ+I wasused as the substitution model for all tests. A random starting treewas used. The default values were used for all other settings. Thefinal topologies were then reoptimized in PAUP*, as per the instruc-tions of GARLI’s author, and a final likelihood score was obtained.

The two trees found under the positive and negative constraint ofan HM were compared using the SH test. These likelihood scoreswere compared using the resampling of estimated log-likelihoods(RELL) SH test in PAUP* to see if the likehood score of the treewith the human monophyly was significantly better than the tree thatdid not contain this monophyly.

Although this comparison may be considered a misuse of the SHtest because the trees being compared were selected by ML search–violating the test’s assumptions–this is acceptable to show that thetrees are not significantly different. If however, we intended to showthat the one tree was optimal, our use of the SH test would not beacceptable. In that scenario, we would be required to compare the

4 www.pnas.org — — Footline Author

ML tree to all other possible optimal trees, and to show that our MLtree was significantly better than the set of other trees.

ACKNOWLEDGMENTS. This paper benefited from the careful review and com-ments provided by Niccolo Caldararo, Eric Routman, and Greg Spicer. The

authors thank the Dragutin Petkovic and Mike Wong of the Center for Computingin the Life Sciences (CCLS) for providing the computing power for this research.C.M. was partially supported by a CCLS Mini-grant. A.S. was partially supportedby NSF grant DMS 0601060.

1. Wolpoff MH, Thorne AG, Smith FH, Frayer DW, Pope GG (1994) in Origins of Anatom-ically Modern Humans, eds Nitecki MH, Nitecki DV (Plenum, New York), pp. 175-199.

2. Wolpoff MH, Wu X, Thorne A (1984) in The Origins of Modern Humans: A World Sur-vey of the Fossil Evidence, eds Smith F, Spencer F (Liss, New York), pp. 411-483.

3. Cann RL, Stoneking M, Wilson AC (1987) Mitochondrial DNA and human evolutionNature 325:31-36.

4. Stringer C, Andrews P (1988) Genetic and fossil evidence for the origin of modernhumans. Science 239:1263-1268.

5. Trinkaus E (2007) European early modern humans and the fate of the Neandertals.Proc Natl Acad Sci USA 104:7367-7372.

6. Krings M, Stone A, Schmitz RW, Krainitzki H, Stoneking M, Paabo S (1997) Nean-dertal DNA sequences and the origin of modern humans. Cell 90:19-30.

7. Krings M, Geisert H, Schmitz RW, Krainitzki H, Paabo S (1999) DNA sequence of themitochondrial hypervariable region II from the Neandertal type specimen. Proc NatlAcad Sci USA 96:5581-5585.

8. Ovchinnikov IV, Gotherstrom A, Romanova GP, Kharitonov VM, Liden K, Goodwin W(2000) Molecular analysis of Neanderhal DNA from the northern Caucasus. Nature404:490-493.

9. Krings M, Capelli C, Tschentscher F, Geisert H, Meyer S, von Haeseler A, Grosss-chmidt K, Possnert G, Paunovic M, Paabo S (2000) A view of Neandertal geneticdiversity. Nat Genet 26:144-146.

10. Schmitz RW, Serre D, Bonani G, Feine S, Hillgruber F, Krainitzki H, Paabo S, Smith FH(2002) The Neandertal type site revisited; interdisciplinary investigations of skeletalremains from the Neander Valley, Germany. Proc Natl Acad Sci USA 99:13342-13347.

11. Serre D, Langaney A, Chech M, Teschler-Nicola M, Paunovic M, Mennecier P, Hofre-iter M, Possnert G, Paabo S (2004) No evidence of Neandertal mtDNA contributionto early modern humans. PLoS Biol 2:313-317.

12. Lalueza-Fox C, Krause J, Caramelli D, Catalano G, Milani L, Sampietro ML, CalafellF, Martınez-Maza C, Bastir M, Garcıa-Tabernero A, de la Rasilla M, Fortea J, PaaboS (2006) Mitochondrial DNA of an Iberian Neandertal suggests a population affinitywith other European Neandertals. Curr Biol 16:R629-30.

13. Caramelli D, Lalueza-Fox C, Condemi S, Longo L, Milani L, Manfredini A, de SaintPierre M, Adoni F, Lari M, Giunti P, Ricci S, Casoli A, Calafell F, Mallegni F, Bertran-petit J, Stanyon R, Bertorelle G, Barbujani G (2006). Curr Biol 16, R630-2.

14. Caramelli D, Lalueza-Fox C, Vernesi C, Lari M, Casoli A, Mallegni F, Chiarelli B,Dupanloup I, Bertranpetit J, Barbujani G, Bertorelle G (2003) Evidence for a ge-netic discontinuity between Neandertals and 24,000-year-old anatomically modernEuropeans. Proc Natl Acad Sci U S A 100:6593-6597.

15. Saitou N, Nei M (1987) The neighbor-joining method: a new method for reconstruct-ing phylogenetic trees. Mol Biol Evol 4:406-425.

16. Strimmer K, von Haeseler A (1997) Likelihood-mapping: A simple method to vi-sualize phylogenetic content of a sequence alignment. Proc Natl Acad Sci USA94:6815-6819.

17. Felsenstein J (1985) Confidence limits on phylogenies: an approach using thebootstrap. Evolution 39:783-791.

18. Orlando L, Darlu P, Toussaint M, Bonjean D, Otte M, Hanni C (2006) 100,000 Year-old DNA Sequence Allows New Look At Neandertal’s Genetic Diversity Curr Biol16:R400-R402.

19. Hebsgaard MB, Wiuf C, Gilbert MTP, Glenner H, Willerslev E (2007) Evaluating Ne-andertal Genetics and Phylogeny. J Mol Evol 64:50-60.

20. Caldararo N, Gabow S (2000) Mitohondrial DNA analysis and the place of Neander-tals in Homo. Ancient Biomol 3:135-158.

21. Hansen AJ, Willerslev E, Wiuf C, Mourier T, Arctander P (2001) Statistical Evidencefor Miscoding Lesions in Ancient DNA Templates. Mol. Biol. Evol. 18:262-265.

22. Gutierrez G, Sanchez D, Marın A (2002) A Reanalysis of the Ancient MitochondrialDNA Sequences Recovered from Neandertal Bones. Mol Biol Evol 19:1359-1366.

23. Pusch CM, Bachmann L (2004) Spiking of Contemporary Human Template DNA withAncient DNA Extracts Induces Mutations Under PCR and Generates NonauthenticMitochondrial Sequences. Mol Biol Evol 21:957-964.

24. Ingman M, Kaesmann H, Paabo S, Gyllensten U (2000) Mitochondrial genome vari-ation and the origin of modern humans. Nature 408:708-713.

25. Sitnikova T, Rzhetsky A, Nei M (1995) Interior-branch and bootstrap tests of phylo-genetic trees. Mol Biol Evol 12:319-333.

26. Swofford DL (2003) PAUP*. Phylogenetic Analysis Using Parsimony (*and OtherMethods). Version 4. (Sinauer, Sunderland, MA).

27. Posada D, Crandall KA (1998) MODELTEST: testing the model of DNA substitution.Bioinformatics 14:817-818.

28. Hasegawa M, Kishino H, Yano T (1985) Dating of the human-ape splitting by amolecular clock of mitochondrial DNA. J Mol Evol 22:160-174.

29. Lanave C, Preparata G, Sacone C, Serio G (1984) A new method for calculatingevolutionary substitution rates. J Mol Evol 20:86-93.

30. Posada D, Crandall KA (2001) Selecting the best-fit model of nucleotide substitu-tion. Syst Biol 50:580-601.

31. Kass RE, Wasserman L (1995) A Reference Bayesian Test for Nested Hypothesesand its Relationship to the Schwarz Criterion. J Am Stat Assoc 90:928-934.

32. Posada D (2003) in The Phylogenetic Handbook, eds Salemi M, Vandamme A (Cam-bridge University Press, Cambridge) pp 256-282.

33. Akaike H (1974) A new look at the statistical model identification. IEEE Trans onAutomat Contr 16, 716-723.

34. Kishino H, Hasegawa M (1989) Evaluation of the maximum likelihood estimate ofthe evolutionary tree topologies from DNA sequence data, and the branching orderin Hominoidea. J Mol Evol 29:170-179.

35. Felsenstein J, Churchill GA (1996) A hidden Markov model approach to variationamong sites in rate of evolution. Mol Biol Evol 13:93-104.

36. Pereira, Cunha, Amorim (2004) Predicting sampling saturation of mtDNA haplo-types: an application to an enlarged Portuguese database Intl J Legal Med 118:132-136.

37. Brown MD, Hosseini SH, Torroni A, Bandelt HJ, Allen JC, Schurr TG, Scozzari R,Cruciani F, Wallace DC (1998) mtDNA Haplogroup X: An Ancient Link between Eu-rope/Western Asia and North America? Am J Hum Genet 63:1852-1861.

38. Gilbert MTP, Willerslev E, Hansen AJ, Barnes I, Rudbeck L, Lynnerup N, Cooper A(2003) Distribution patterns of postmortem damage in human mitochondrial DNA.Am J Hum Genet 72:32-47.

39. Schmidt HA, Strimmer K, Vingron M, and von Haeseler A (2002) TREE-PUZZLE:maximum likelihood phylogenetic analysis using quartets and parallel computing.Bioinformatics 18:502-504.

40. Nordborg, M (1998) On the probability of Neanderthal ancestry. Am. J. Hum. Genet.63:1237-1240.

41. Wu C, Li W (1985) Evidence for Higher Rates of Nucleotide Substitution in RodentsThan in Man. Proc Natl Acad Sci 82, 1741-1745.

42. Sarich VM, Wilson AC (1967) Immunological Time Scale for Hominid Evolution.Science 158:1200-1203.

43. Hillis DM, Mable B, Moritz C (1996) in Molecular Systematics, eds. Hillis DM, MoritzC, Mable B (Sinauer Assoc., Sunderland, Massachusetts) pp. 531-533.

44. Handt O, Meyer S, von Haeseler A (1998) Compilation of human mtDNA controlregion sequences. Nucleic Acids Res 26, 126-129.

45. Wiens JJ (1998) Does adding characters with missing data increase or decreasephylogenetic accuracy? Syst Biol 47:625-640.

46. Wiens JJ (2005) Can incomplete taxa rescue phylogenetic analyses from long-branch attraction? Syst Biol 54:731-742.

47. Wiens JJ (2006) Missing data and the design of phylogenetic analyses. J BiomedInform 39:34-42.

48. Jukes TH, Cantor CH (1969) in Mammalian protein metabolism , ed Munro HN (Aca-demic Press, New York) pp 21-132.

49. Felsenstein J (1981) Evolutionary trees from DNA sequences: a maximum likeli-hood approach. J Mol Evol 17:368-376.

50. Tamura K, Nei M (1993) Estimation of the number of nucleotide substitutions in thecontrol region of mitochondrial DNA in humans and chimpanzees. Mol Biol Evol10:512-526.

51. Huelsenbeck JP, Ronquist F, Nielsen R, Bollback JP (2001) Bayesian inference ofphylogeny and its impact on evolutionary biology. Science 294:2310-2314.

52. Ronquist F, Huelsenbeck JP (2003) MRBAYES 3: Bayesian phylogenetic inferenceunder mixed models. Bioinformatics 19:1572-1574.

53. Felsenstein J (2004) Inferring phylogenies (Sinauer Assoc., Sunderland, MA) pp301-306.

54. Zwickl, DJ (2006) Genetic algorithm approaches for the phylogenetic analysis oflarge biological sequence datasets under the maximum likelihood criterion. Ph.D.dissertation, The University of Texas at Austin.

Footline Author PNAS Issue Date Volume Issue Number 5

Table 1. List of Neanderthal mtDNA sequences with more than 300 sites.

Paper Location of remains Abbrev. Type Length[6] Feldhofer, Germany FE1 HVRI 379[7] Feldhofer, Germany FE1 HVRII 345[8] Mezmaiskaya Cave MEZ HVRI 345[9] Vindija, Croatia V75 HVRI 357[9] Vindija, Croatia V75 HVRII 288[10] Feldhofer, Germany FE2 HVRI 357[11] Vindija, Croatia V80∗ HVRI 378[12] El Sidron, Spain SI2 HVRI 303[13] Monte Lessini, Italy MLS HVRI 378∗Identical to V75

0

20

40

60

80

100

0

20

40

60

80

100HVRI − 50 data sets with 200 humans HVRI − 30 data sets with 500 humans

Full data setsTrimmed data sets

HVRI+II − 50 data sets with 200 humans

0

20

40

60

80

100

JC F81 F84 JC+Γ F84+Γ+I TN+Γ+I GTR+Γ+I

HVRI+II − 30 data sets with 500 humans

JC F81 F84 JC+Γ F84+Γ+I TN+Γ+I GTR+Γ+I

Fig. 1. NJ bootstrap support for an HM for full (all sites) and trimmed (excluding missing/ambiguous sites) data sets

Fig. 2. Consensus tree from Bayesian inference on HVRI data set containing: 6 chimpanzee, 6 Neanderthal, and 200 human sequences.

6 www.pnas.org — — Footline Author

(a,b)-(c,c)

(a,c)-(b,c) (a,c)-(b,c)

88.9%

5.6% 5.5%

81.5%

1.9% 1.9%0.7%

7.8%

3.1%

3.1%

Fig. 3. LM support with six chimpanzee (Group A), FE1 Neanderthal (Group B), 200 Human sequences (Group C). Human data set chosen to approximate resultsin [6].

(a,b)-(c,d)

(a,d)-(b,c) (a,c)-(b,d)

10.1%

0.9% 89.0%

7.9%

0.3% 84.1%0.7%

0.4%

6.1%

0.5%

Chimpanzees

Divergent African

NeanderthalsBranch Tested 199 Other Humans

Fig. 4. LM analysis of HVRI region with six chimpanzee (Group A), six Neanderthal (Group B), a divergent African human (Group C), and 199 randomly selectedhuman sequences (Group D). Based on TN+Γ+I with parameter values from [22].

Footline Author PNAS Issue Date Volume Issue Number 7