Data Structures&

Algorithm Analysisin C++

THIRD EDITION

Weiss C++ third pages 2005/9/19 11:27 p. i (front) Windfall Software, PCA ZzTEX v12

Weiss C++ third pages 2005/9/19 11:27 p. ii (front) Windfall Software, PCA ZzTEX v12

Data Structures&

Algorithm Analysisin C++

THIRD EDITION

Mark Al len WeissFlorida International University

Weiss C++ third pages 2005/9/19 11:27 p. iii (front) Windfall Software, PCA ZzTEX v12

Acquisitions Editor ???Project Editor ???Production Supervisor Marilyn LloydMarketing Manager ???Marketing Coordinator ???Project Management Windfall SoftwareComposition Windfall Software, using ZzTEXTechnical Illustration George NicholsProofreader MaryEllen N. OliverIndexer Ted LauxCover Design Manager ???Cover Designer ???Cover Image ???Prepress and Manufacturing Caroline FellPrinter ???

Access the latest information about Addison-Wesley titles from our World Wide Web site:http://www.aw-bc.com/computing

Many of the designations used by manufacturers and sellers to distinguish their products areclaimed as trademarks. Where those designations appear in this book, and Addison-Wesley wasaware of a trademark claim, the designations have been printed in initial caps or all caps.

Library of Congress Cataloging-in-Publication Data

Weiss, Mark Allen.Data structures and algorithm analysis in C++ / Mark Allen Weiss.—3rd ed.

p. cm.Includes bibliographical references and index.ISBN 0-321-37531-9 (alk. paper)

1. C++ (Computer program language) 2. Data structures (Computerscience) 3. Computer algorithms. I. Title.

QA76.73.C153W46 2005005.13′3—dc22

2005023534

Copyright © 2006 by Pearson Education, Inc.

For information on obtaining permission for use of material in this work, please submit a writtenrequest to Pearson Education, Inc., Rights and Contract Department, 75 Arlington Street, Suite300, Boston, MA 02116 or fax your request to (617) 848-7047.

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system,or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording,or any other media embodiments now known or hereafter to become known, without the priorwritten permission of the publisher. Printed in the United States of America.

ISBN 0-321-37531-91 2 3 4 5 6 7 8 9 10—CSR—08 07 06 05

Weiss C++ third pages 2005/9/19 11:27 p. iv (front) Windfall Software, PCA ZzTEX v12

???

Weiss C++ third pages 2005/9/19 11:27 p. v (front) Windfall Software, PCA ZzTEX v12

Weiss C++ third pages 2005/9/19 11:27 p. vi (front) Windfall Software, PCA ZzTEX v12

CONTENTS

Preface xv

Chapter 1 Introduction 11.1 What’s the Book About? 1

1.2 Mathematics Review 21.2.1 Exponents 31.2.2 Logarithms 31.2.3 Series 41.2.4 Modular Arithmetic 51.2.5 The P Word 6

1.3 A Brief Introduction to Recursion 7

1.4 C++ Classes 111.4.1 Basic class Syntax 121.4.2 Extra Constructor Syntax and Accessors 121.4.3 Separation of Interface and Implementation 151.4.4 vector and string 17

1.5 C++ Details 191.5.1 Pointers 191.5.2 Parameter Passing 211.5.3 Return Passing 221.5.4 Reference Variables 231.5.5 The Big Three: Destructor, Copy Constructor, operator= 231.5.6 C-style Arrays and Strings 26

1.6 Templates 291.6.1 Function Templates 291.6.2 Class Templates 301.6.3 Object, Comparable, and an Example 321.6.4 Function Objects 341.6.5 Separate Compilation of Class Templates 35

1.7 Using Matrices 371.7.1 The Data Members, Constructor, and Basic Accessors 371.7.2 operator[] 371.7.3 Destructor, Copy Assignment, Copy Constructor 39 vii

Weiss C++ third pages 2005/9/19 11:27 p. vii (front) Windfall Software, PCA ZzTEX v12

viii Contents

Summary 39

Exercises 39

References 41

Chapter 2 Algorithm Analysis 432.1 Mathematical Background 43

2.2 Model 46

2.3 What to Analyze 46

2.4 Running Time Calculations 492.4.1 A Simple Example 492.4.2 General Rules 502.4.3 Solutions for the Maximum Subsequence Sum Problem 522.4.4 Logarithms in the Running Time 582.4.5 Checking Your Analysis 622.4.6 A Grain of Salt 63

Summary 63

Exercises 64

References 69

Chapter 3 Lists, Stacks, and Queues 713.1 Abstract Data Types (ADTs) 71

3.2 The List ADT 723.2.1 Simple Array Implementation of Lists 723.2.2 Linked Lists 73

3.3 vector and list in the STL 743.3.1 Iterators 753.3.2 Example: Using erase on a List 773.3.3 const_iterators 77

3.4 Implementation of vector 79

3.5 Implementation of list 83

3.6 The Stack ADT 943.6.1 Stack Model 943.6.2 Implementation of Stacks 953.6.3 Applications 96

3.7 The Queue ADT 1043.7.1 Queue Model 1043.7.2 Array Implementation of Queues 1043.7.3 Applications of Queues 106

Summary 107

Weiss C++ third pages 2005/9/19 11:27 p. viii (front) Windfall Software, PCA ZzTEX v12

Contents ix

Exercises 108

Chapter 4 Trees 1134.1 Preliminaries 113

4.1.1 Implementation of Trees 1144.1.2 Tree Traversals with an Application 115

4.2 Binary Trees 1194.2.1 Implementation 1204.2.2 An Example: Expression Trees 121

4.3 The Search Tree ADT—Binary Search Trees 1244.3.1 contains 1254.3.2 findMin and findMax 1254.3.3 insert 1294.3.4 remove 1304.3.5 Destructor and Copy Assignment Operator 1324.3.6 Average-Case Analysis 133

4.4 AVL Trees 1364.4.1 Single Rotation 1394.4.2 Double Rotation 142

4.5 Splay Trees 1494.5.1 A Simple Idea (That Does Not Work) 1504.5.2 Splaying 152

4.6 Tree Traversals (Revisited) 158

4.7 B-Trees 159

4.8 Sets and Maps in the Standard Library 1654.8.1 Sets 1654.8.2 Maps 1664.8.3 Implementation of set and map 1674.8.4 An Example That Uses Several Maps 168

Summary 174

Exercises 174

References 181

Chapter 5 Hashing 1855.1 General Idea 185

5.2 Hash Function 186

5.3 Separate Chaining 188

5.4 Hash Tables Without Linked Lists 1925.4.1 Linear Probing 193

Weiss C++ third pages 2005/9/19 11:27 p. ix (front) Windfall Software, PCA ZzTEX v12

x Contents

5.4.2 Quadratic Probing 1955.4.3 Double Hashing 199

5.5 Rehashing 200

5.6 Hash Tables in the Standard Library 204

5.7 Extendible Hashing 204

Summary 207

Exercises 208

References 211

Chapter 6 Priority Queues (Heaps) 2136.1 Model 213

6.2 Simple Implementations 214

6.3 Binary Heap 2156.3.1 Structure Property 2156.3.2 Heap-Order Property 2166.3.3 Basic Heap Operations 2176.3.4 Other Heap Operations 220

6.4 Applications of Priority Queues 2256.4.1 The Selection Problem 2266.4.2 Event Simulation 227

6.5 d-Heaps 228

6.6 Leftist Heaps 2296.6.1 Leftist Heap Property 2296.6.2 Leftist Heap Operations 230

6.7 Skew Heaps 235

6.8 Binomial Queues 2396.8.1 Binomial Queue Structure 2406.8.2 Binomial Queue Operations 2416.8.3 Implementation of Binomial Queues 244

6.9 Priority Queues in the Standard Library 251

Summary 251

Exercises 251

References 257

Chapter 7 Sorting 2617.1 Preliminaries 261

7.2 Insertion Sort 2627.2.1 The Algorithm 262

Weiss C++ third pages 2005/9/19 11:27 p. x (front) Windfall Software, PCA ZzTEX v12

Contents xi

7.2.2 STL Implementation of Insertion Sort 2637.2.3 Analysis of Insertion Sort 264

7.3 A Lower Bound for Simple Sorting Algorithms 265

7.4 Shellsort 2667.4.1 Worst-Case Analysis of Shellsort 268

7.5 Heapsort 2707.5.1 Analysis of Heapsort 272

7.6 Mergesort 2747.6.1 Analysis of Mergesort 276

7.7 Quicksort 2797.7.1 Picking the Pivot 2807.7.2 Partitioning Strategy 2827.7.3 Small Arrays 2847.7.4 Actual Quicksort Routines 2847.7.5 Analysis of Quicksort 2867.7.6 A Linear-Expected-Time Algorithm for Selection 290

7.8 Indirect Sorting 2927.8.1 vector<Comparable *> Does Not Work 2957.8.2 Smart Pointer Class 2957.8.3 Overloading operator< 2957.8.4 Dereferencing a Pointer with * 2957.8.5 Overloading the Type Conversion Operator 2957.8.6 Implicit Type Conversions Are Everywhere 2967.8.7 Dual-Direction Implicit Conversions Can Cause Ambiguities 2967.8.8 Pointer Subtraction Is Legal 297

7.9 A General Lower Bound for Sorting 2977.9.1 Decision Trees 297

7.10 Bucket Sort 299

7.11 External Sorting 3007.11.1 Why We Need New Algorithms 3007.11.2 Model for External Sorting 3007.11.3 The Simple Algorithm 3017.11.4 Multiway Merge 3027.11.5 Polyphase Merge 3037.11.6 Replacement Selection 304

Summary 305

Exercises 306

References 311

Weiss C++ third pages 2005/9/19 11:27 p. xi (front) Windfall Software, PCA ZzTEX v12

xii Contents

Chapter 8 The Disjoint Set Class 3158.1 Equivalence Relations 315

8.2 The Dynamic Equivalence Problem 316

8.3 Basic Data Structure 317

8.4 Smart Union Algorithms 321

8.5 Path Compression 324

8.6 Worst Case for Union-by-Rank and Path Compression 3258.6.1 Analysis of the Union/Find Algorithm 326

8.7 An Application 331

Summary 334

Exercises 335

References 336

Chapter 9 Graph Algorithms 3399.1 Definitions 339

9.1.1 Representation of Graphs 340

9.2 Topological Sort 342

9.3 Shortest-Path Algorithms 3459.3.1 Unweighted Shortest Paths 3479.3.2 Dijkstra’s Algorithm 3519.3.3 Graphs with Negative Edge Costs 3609.3.4 Acyclic Graphs 3609.3.5 All-Pairs Shortest Path 3649.3.6 Shortest Path Example 365

9.4 Network Flow Problems 3679.4.1 A Simple Maximum-Flow Algorithm 367

9.5 Minimum Spanning Tree 3729.5.1 Prim’s Algorithm 3739.5.2 Kruskal’s Algorithm 376

9.6 Applications of Depth-First Search 3789.6.1 Undirected Graphs 3799.6.2 Biconnectivity 3819.6.3 Euler Circuits 3859.6.4 Directed Graphs 3889.6.5 Finding Strong Components 390

9.7 Introduction to NP-Completeness 3929.7.1 Easy vs. Hard 3929.7.2 The Class NP 3939.7.3 NP-Complete Problems 394

Weiss C++ third pages 2005/9/19 11:27 p. xii (front) Windfall Software, PCA ZzTEX v12

Contents xiii

Summary 396

Exercises 396

References 404

Chapter 10 Algorithm Design Techniques 40910.1 Greedy Algorithms 409

10.1.1 A Simple Scheduling Problem 41010.1.2 Huffman Codes 41310.1.3 Approximate Bin Packing 419

10.2 Divide and Conquer 42710.2.1 Running Time of Divide and Conquer Algorithms 42810.2.2 Closest-Points Problem 43010.2.3 The Selection Problem 43510.2.4 Theoretical Improvements for Arithmetic Problems 438

10.3 Dynamic Programming 44210.3.1 Using a Table Instead of Recursion 44210.3.2 Ordering Matrix Multiplications 44410.3.3 Optimal Binary Search Tree 44710.3.4 All-Pairs Shortest Path 451

10.4 Randomized Algorithms 45410.4.1 Random Number Generators 45510.4.2 Skip Lists 45910.4.3 Primality Testing 461

10.5 Backtracking Algorithms 46410.5.1 The Turnpike Reconstruction Problem 46510.5.2 Games 469

Summary 475

Exercises 475

References 485

Chapter 11 Amortized Analysis 49111.1 An Unrelated Puzzle 492

11.2 Binomial Queues 492

11.3 Skew Heaps 497

11.4 Fibonacci Heaps 49911.4.1 Cutting Nodes in Leftist Heaps 50011.4.2 Lazy Merging for Binomial Queues 50211.4.3 The Fibonacci Heap Operations 50611.4.4 Proof of the Time Bound 506

Weiss C++ third pages 2005/9/19 11:27 p. xiii (front) Windfall Software, PCA ZzTEX v12

xiv Contents

11.5 Splay Trees 509

Summary 513

Exercises 513

References 515

Chapter 12 Advanced Data Structures andImplementation 517

12.1 Top-Down Splay Trees 517

12.2 Red-Black Trees 52512.2.1 Bottom-Up Insertion 52612.2.2 Top-Down Red-Black Trees 52712.2.3 Top-Down Deletion 529

12.3 Deterministic Skip Lists 535

12.4 AA-Trees 541

12.5 Treaps 546

12.6 k-d Trees 551

12.7 Pairing Heaps 554

Summary 560

Exercises 560

References 563

Appendix A Separate Compilation Of Class Templates567

A.1 Everything in the Header 568

A.2 Explicit Instantiation 568

A.3 The export Directive 570

Weiss C++ third pages 2005/9/19 11:27 p. xiv (front) Windfall Software, PCA ZzTEX v12

PREFACE

Purpose/GoalsThe third edition of Data Structures and Algorithms Analysis in C++ describes data structures,methods of organizing large amounts of data, and algorithm analysis, the estimation of therunning time of algorithms. As computers become faster and faster, the need for programsthat can handle large amounts of input becomes more acute. Paradoxically, this requiresmore careful attention to efficiency, since inefficiencies in programs become most obviouswhen input sizes are large. By analyzing an algorithm before it is actually coded, studentscan decide if a particular solution will be feasible. For example, in this text students look atspecific problems and see how careful implementations can reduce the time constraint forlarge amounts of data from 16 years to less than a second. Therefore, no algorithm or datastructure is presented without an explanation of its running time. In some cases, minutedetails that affect the running time of the implementation are explored.

Once a solution method is determined, a program must still be written. As computershave become more powerful, the problems they must solve have become larger and morecomplex, requiring development of more intricate programs. The goal of this text is to teachstudents good programming and algorithm analysis skills simultaneously so that they candevelop such programs with the maximum amount of efficiency.

This book is suitable for either an advanced data structures (CS7) course or a first-year graduate course in algorithm analysis. Students should have some knowl- edge ofintermediate programming, including such topics as pointers, recursion, and object-basedprogramming, and some background in discrete math.

ApproachAlthough the material in this text is largely language independent, programming requiresthe use of a specific language. As the title implies, we have chosen C++ for this book.

C++ has emerged as the leading systems programming language. In addition to fixingmany of the syntactic flaws of C, C++ provides direct constructs (the class and template) toimplement generic data structures as abstract data types.

The most difficult part of writing the book was deciding on the amount of C++ toinclude. Use too many features of C++, and one gets an incomprehensible text; use too fewand you have little more than a C text that supports classes.

The approach we take is to present the material in an object-based approach. As such,unlike the first edition, there is almost no use of inheritance in the text. We use classtemplates to describe generic data structures. We generally avoid esoteric C++ features,and use the vector and string classes that are now part of the C++ standard. Using thesefirst-class versions, instead of the second-class counterparts that were used in the firstedition, simplifies much of the code. Previous editions have implemented class templates by xv

Weiss C++ third pages 2005/9/19 11:27 p. xv (front) Windfall Software, PCA ZzTEX v12

xvi Preface

separatig the class template interface from its implementation. Although this is undoubtedlythe preferred approach, it exposes compiler problems that have made it difficult for readersto actually use the code. As a result, in this edition, the online code represents class templatesas a single unit, with no separation of interface and implementation. Chapter 1 provides areview of the C++ features that are used throughout the text and describes our approachto class templates. Appendix A describes how the class templates could be rewritten to useseparate compilation.

Complete versions of the data structures, in both C++ and Java, are available on theInternet. We use similar coding conventions to make the parallels between the two languagesmore evident.

OverviewChapter 1 contains review material on discrete math and recursion. I believe the only wayto be comfortable with recursion is to see good uses over and over. Therefore, recursionis prevalent in this text, with examples in every chapter except Chapter 5. Chapter 1 alsoincludes material that serves as a review of basic C++. Included is a discussion of templatesand important constructs in C++ class design.

Chapter 2 deals with algorithm analysis. This chapter explains asymptotic analysis andits major weaknesses. Many examples are provided, including an in-depth explanation oflogarithmic running time. Simple recursive programs are analyzed by intuitively convertingthem into iterative programs. More complicated divide-and-conquer programs are intro-duced, but some of the analysis (solving recurrence relations) is implicitly delayed untilChapter 7, where it is performed in detail.

Chapter 3 covers lists, stacks, and queues. This chapter has been significantly revisedfrom prior editions. It now includes a discussion of the STL vector and list classes,including material on iterators, and it provides implementations of a significant subset ofthe STL vector and list classes.

Chapter 4 covers trees, with an emphasis on search trees, including external searchtrees (B-trees). The UNIX file system and expression trees are used as examples. AVL trees andsplay trees are introduced. More careful treatment of search tree implementation details isfound in Chapter 12. Additional coverage of trees, such as file compression and game trees,is deferred until Chapter 10. Data structures for an external medium are considered as thefinal topic in several chapters. New to this edition is a discussion of the STL set and mapclasses, including a significant example that illustrates the use of three separate maps toefficiently solve a problem.

Chapter 5 is a relatively short chapter concerning hash tables. Some analysis is per-formed, and extendible hashing is covered at the end of the chapter.

Chapter 6 is about priority queues. Binary heaps are covered, and there is additionalmaterial on some of the theoretically interesting implementations of priority queues. TheFibonacci heap is discussed in Chapter 11, and the pairing heap is discussed in Chapter12.

Chapter 7 covers sorting. It is very specific with respect to coding details and analysis.All the important general-purpose sorting algorithms are covered and compared. Fouralgorithms are analyzed in detail: insertion sort, Shellsort, heapsort, and quicksort. Externalsorting is covered at the end of the chapter.

Weiss C++ third pages 2005/9/19 11:27 p. xvi (front) Windfall Software, PCA ZzTEX v12

Preface xvii

Chapter 8 discusses the disjoint set algorithm with proof of the running time. This is ashort and specific chapter that can be skipped if Kruskal’s algorithm is not discussed.

Chapter 9 covers graph algorithms. Algorithms on graphs are interesting, not onlybecause they frequently occur in practice but also because their running time is so heavilydependent on the proper use of data structures. Virtually all of the standard algorithmsare presented along with appropriate data structures, pseudocode, and analysis of runningtime. To place these problems in a proper context, a short discussion on complexity theory(including NP-completeness and undecidability) is provided.

Chapter 10 covers algorithm design by examining common problem-solving tech-niques. This chapter is heavily fortified with examples. Pseudocode is used in these laterchapters so that the student’s appreciation of an example algorithm is not obscured byimplementation details.

Chapter 11 deals with amortized analysis. Three data structures from Chapters 4 and6 and the Fibonacci heap, introduced in this chapter, are analyzed.

Chapter 12 covers search tree algorithms, the k-d tree, and the pairing heap. This chapterdeparts from the rest of the text by providing complete and careful implementations for thesearch trees and pairing heap. The material is structured so that the instructor can integratesections into discussions from other chapters. For example, the top-down red-black tree inChapter 12 can be discussed under AVL trees (in Chapter 4).

Chapters 1–9 provide enough material for most one-semester data structures courses.If time permits, then Chapter 10 can be covered. A graduate course on algorithm analysiscould cover Chapters 7–11. The advanced data structures analyzed in Chapter 11 can easilybe referred to in the earlier chapters. The discussion of NP-completeness in Chapter 9 is fartoo brief to be used in such a course. Garey and Johnson’s book on NP-completeness canbe used to augment this text.

ExercisesExercises, provided at the end of each chapter, match the order in which material ispresented. The last exercises may address the chapter as a whole rather than a specificsection. Difficult exercises are marked with an asterisk, and more challenging exerciseshave two asterisks.

A solutions manual containing solutions to almost all the exercises is available onlineto instructors from the Addison Wesley Longman Publishing Company. Instructors shouldcontact their Addison-Wesley local sales representative for information on the manual’savailability.

ReferencesReferences are placed at the end of each chapter. Generally the references either are his-torical, representing the original source of the material, or they represent extensions andimprovements to the results given in the text. Some references represent solutions to exer-cises.

Code AvailabilityThe example program code in this book is available via anonymous ftp at ftp.awl.com. Itis also accessible through the World Wide Web; the URL is http://www.awl.com/cseng/(follow the links from there). The exact location of this material may change.

Weiss C++ third pages 2005/9/19 11:27 p. xvii (front) Windfall Software, PCA ZzTEX v12

xviii Preface

AcknowledgmentsMany, many people have helped me in the preparation of books in this series. Some arelisted in other versions of the book; thanks to all.

As usual, the writing process was made easier by the professionals at Addison-Wesley.I’d like to thank my editor, Susan Hartman; associate editor, Katherine Harutunian; andproduction editor, Marilyn Lloyd. I’d also like to thank Paul Anagnostopoulos and his staffat Windfall Software for their fine work putting the final pieces together.

Finally, I’d like to thank the numerous readers who have sent e-mail messages andpointed out errors or inconsistencies in earlier versions. My World Wide Web page http://www.cis.fiu.edu/~weiss will contain updated source code (in C++, C, and Java), an erratalist, and a link to submit bug reports.

M.A.W.Miami, Florida

Weiss C++ third pages 2005/9/19 11:27 p. xviii (front) Windfall Software, PCA ZzTEX v12

Data Structures&

Algorithm Analysisin C++

THIRD EDITION

Weiss C++ third pages 2005/9/19 11:27 p. xix (front) Windfall Software, PCA ZzTEX v12

Weiss C++ third pages 2005/9/19 11:27 p. xx (front) Windfall Software, PCA ZzTEX v12