Increasing device complexity has brought increased responsibilityboth for test engineers and design engineers: design engineers

must create testable designs; test engineers must reporttestability problems back to the designers.

The Effects of theMicroelectronicsRevolution onSystems andBoard TestEugene FoleyTeradyne, Inc.

Today more than ever, the luxury and increasingnecessity of sophisticated electronics is assuming alarger role in our daily lives. The microelectronicsrevolution has provided us with capability and per-formance in such small packaging that electronicswakes us in the morning, cooks our meals, controlsboth the environment in our automobiles and theengine which powers it, automates much of our work,provides recreation, and guards our homes while wesleep. But with growing capability and performancecomes increased complexity. Consumers initiallyshunned complexity, but with progressive familiari-ty and understanding through use, theynow demandit. This very complexity, however, is having anadverse effect on industry's ability to test modernelectronic products. And apparently the testing pro-blem can only become worse as devices such asmicroprocessors, minicomputers, and other large-scale and very-large-scale integration devices beginto populate nearly all electronics in ever-increasingnumbers.The testing problem is so severe that our ability (or

in some cases inability) to test is a major limitingprocess-step in the production line. Yet, withunderstanding and careful design, it is fully feasibleto test modern complex electronic products quicklyand economically.For some time I have been trying to teach elec-

tronic design engineers the guidelines for designingproducts that are testable. Rapid changes in the elec-tronics industry quickly render any rules or pro-cedures obsolete. Therefore, the only way to ensurethat products are actually testable is for the designerto clearly understand the testing process, includingits needs and capabilities. And he must be able to im-

plement state-of-the-art test techniques ashe designsstate-of-the-art products. To understand the effectsof LSI/VLSI on theproduction testenvironment, onemust first understand the production-line and testprocesses prior to the microelectronic revolution.Before the industrial revolution, a craftsman first

assembled and then tested his product. Usually theproduct functioned correctly, and what we currentlyknow as "systems test" or "final test" was the onlyinspection step. By modern standards the demandfor goods and services was small and products weresimple; thus there was no need for additionalassembly or inspection steps.When the evolution of mass production triggered

the industrial revolution, the concept ofeomponentsemerged, and with it the need for component-levelassembly-line processing with its own additional in-spection steps.,A producer found it necessary to in-spect components to ensure that they were dimen-sionally correctandwithin tolerance before their finalassembly as part of the final product. The motivationto inspect components was economic, for even in theearly days of mass production it was apparent thatthe earlier a defective component was found the lessit cost to find it.

Thus the basic elements of the modern produc-tion line begin to emerge, and an elementary process-.flow diagram can be constructed (See Figure 1).As product complexity steadily increased, the con-

cept of subassemblies evolved, further changing thebasic production flow shown in Figure 1. The singleassembly block required expansion to include sub-assembly production, subassembly inspection, andsystems assembly. The resulting production-flowdiagram clearly shows the basics ofa modern produc-

0018-9162/791000-0032SO0.75 0 1979 ItEE32 COMPUTER

tion line, alternating between inspection steps andprocess steps. When this production-flow diagram isapplied to the electronics industry, it is convenient torelabel the blocks to represent more accurately theprocess steps involved, as shown in Figure 2.This production-flow diagram indicates that the

product is a "system" comprising printed-circuitboards and that its manufacture calls for three in-spection steps. Board test, which -is our version ofsubassembly test, was introduced into the produc-tion flow for economic reasons. Again, the earlier adefective component/subassembly is found, the lessexpensive it is to find. The particular economics thatapply to the electronics industry show that the costto find a defective component increases ten-fold foreach inspection step that fails to find the defect. AsHotchkiss has observed,

"When considering the economics of testing, it isgenerally recognized throughout the industry thatthe cost of finding a fault increases dramatically asyou move to later and later stages of the manufactur-ing process, For example, the standard industry costfor finding a bad IC at incoming inspection is about30¢' (which includes the labor cost to handle and testthe partand the cost of the equipment amortized oversome time period) and assumes an average rejectionrate of 1% to 3%, the current rejection rate at incom-ing inspection for a standard TTL part. At board test,the cost of finding a bad IC increases, by a factor of10, to $3.00. This is due to the higher equipment costand labor cQst; the people doing the testing are bettertrained and therefore more expensive, and the testtimes are longer. At systems test, the cost to find afaculty IC increases, by another factor of 10, to$30.00. Again, this is due to the higher labor rate ofthe more skilled people who are performing thetesting and the longer testing times that result fromthe greater complexity of tests. At field test, the costis at least $300.00 and may be higher if you includecustomer down-time and customer ir will. "'

In the preceding statement, the fault is assumed tobe a defective integrated circuit of the transistor-transistor-logic family, which is generally of small-scale or medium-scale integration complexity. Cer-tainly as device-complexity levels increase, the costsof testingthem increase-a fact verified by observingincreased testing costs as we move further down theassembly line where complexity is always increasing.

This discussion of economics also brings to lightthe fact that our production-flow model is somewhatincomplete. Now we must include customer installa-tion and field testing as two additonal blocks so thatwe can more accurately describe the entire process,In adding the field testing steps to this diagram, itmust be understood that this inspection will be per-formed several timnes-once after installation to ver-ify that the system is in sound condition, and theneither as routine maintenance or when the systemfails. Thus, the true costs of field testing are enor-mous, and everything possible must be done to makethis task quick and economical.

To understand more clearly the effects of LSI andmicroprocessor components on the production pro-cess, let us examine the type of faults that populateSSI/MSI printed-circuit boards. Generally, onewould assume that assembling a printed-circuitboard of proven design with known good parts (allprocessed through incoming inspection) would yielda working board. Unfortunately, that's not the casesince the assembly procedure itself introduces faults.Usually, these faults are incorrectly inserted com-ponents, missing components, shorts and opens; jus-tifiably, they have earned the name of workmanshiperrors. Of course, some defective devices are notfound at incoming inspection (no inspection pro-cedure is 100 percent accurate), and a few devices

Figure 1. Basic elements of the modern production line.

Figure 2. Basic elements of the, modern electronicproduction-flow diagram.

October 1979 33

malfunction very early in their lives. These are knowncommonly as infant-mortality failures.With an understanding of the type of faults pres-

ent, itwas soon learned that the functional board-testsystem need not emulate the system environment inwhich the board would be used. In fact, almost noknowledge of the final system was necessary to findand correct these workmanship errors. For simplelogic boards, a test engineer generated test stimuli toexercise the board and, based on the types of failuresobserved at the board's output, determined what par-ticular corrective action to take. As board complexityincreased, this "footprint" method of diagnosis en-countered problems, since a high probability existedthat more than one fault could produce a particularobserved output failure. The introduction of the"guided probe" solved most of the problems ofdiagnosing faults, since test data were taken directlyfrom internal nodes of the board under test and com-pared to known-good nodal data. The only require-ment for guided-probe diagnosis was to locate adevice with good input data and bad'output data for a

Figure 3, The complete electronic production-flow diagram.

34

particular test step, and the falling node was iden-tified. In general, nodes of SSI/MSI boards are notvery complex (fewer than four devices) and easily ap-plied repair procedures or even trial-and-error tech-niques quickly repair the board.Additional supporting proof that system emula-

tion was not necessary to locate faults on printed-circuit boards evolved when a new type of assembly-inspection equipment was introduced for the solepurpose of finding shorts (solder shorts are the mostprevalent workmanship error found in mass-produced printed-circuit boards). Employing a "bed-of-nails" fixture which uses spring-loaded probesthat contact each node of the board-under-test, testengineers conducted simple tests to find soldershorts that populate boards after completion of thewave-solder production step. The new shorts-detection equipment no more resembled the finalsystem environment than would an ohmmeter; it is,in fact, an automated ohmmeter. However, thistechnique proved so useful that it has earned aperma-nent role in the asembly-flow process and must be ad-ded to our flow diagram.To be wholly correct, we must also include back-

plane test, printed-circuit board fabrication, andprinted-circuit board test in the production-flowdiagram. The completed flow diagram is shown inFigure 3, whichnow enables us to examine the effectsof LSI, VLSI, and microprocessors on the pro-duction-flow process. Close' inspection of Figure 3reveals, interestingly enough, that the left columnconsists largely of electrical component processing,the right column consists of physical/chemical pro-cessing, and the center column of electronic compo-nent processing where our main interest lies.The expanding flow diagram bears witness to the

fact that, up to this point in our discussion, complexi-ty has increased steadily. Not only has the complexi-ty of devices and components increased, but that ofthe printed-circuit board and the final system has in-creased as well. We have dealt with this complexityincrease by supplementing the production-flow pro-cess with additional inspection steps. But we've ar-rived at an interesting point: the mechanical size andcomplexity of final systems are no longer increasing,and therefore the size of printed-circuit boards is nolonger changing. With the size of a printed-circuitboard stabilized, the number of components on thatboard are obviously limited, and the only way to in-crease overall system complexity (capability or per-formance, if you wish) is to increase the functionalcomplexity of the components themselves. We allknow that this increase in complexity is in the form oflarge-scale and very-large-scale integration micropro-cessors and microcomputers. But what few of usknow is that, for the first time, we have an increase incomplexity that does not add process steps to ourflow diagram. Thus the existing inspection steps arebeing asked to deal directly with this increase in com-plexity.

It would be natural to attempt insertion of addi-tional inspection steps into the production-flow

COMPUTER

diagram, but economics again comes into play. Theincreasingly complex devices thatwe areusingdo notcost any more to purchase than their less-complexpredecessors, so there is a natural reluctance to in-crease their value by committing greater amounts oftest time to them. More significantly, productionvolumes are not affected directly by increasingdevice complexity: therefore, the maximum allow-able test times-which previously have been rigidlydefined-are not increased.Now that we understand the production and

testing process, and the reasons for its particularorganization, we can examine the changes caused byLSI/VLSI, microprocessors, and microcomputers ateach testing phase. From an intuitive viewpoint itmight seem natural to assume that the microelec-tronics revolution has had little or no effect on incom-ing inspection. Unfortunately, however, the effectsare profound. Device complexity has increased sorapidly that few incoming inspection facilities arestaffed with people trained to handle the problemsthey encounter. But this hurdle can be overcome bythe practical method of upgrading their technicaleducation.A more serious problem associated with LSINVLSI

is that the specifications which are supplied by themanufacturers with the devices are typically in-complete. Manufacturers no longer provide gate-level documentation as they did with SSI/MSI logicdevices. They no longer provide complete truthtables, nor do they specify device response under allinput-stimulus conditions. In some cases it is simplynot feasible to supply all the documentation that wewere accustomed to receive with SSI/MSI. In othercases, the manufacturers themselves just don't havethe information. The problem has been clearly statedby Dr. Kennth Parker of Hewlett-Packard:

"You can now buy more gates with less pages ofspecifications than at any other time in history."2

The upshot of this lack of specification data is thatincoming-inspection facilities don't know what teststo perform on new devices and are left in a guessingstate. However intelligent that guessing may be, it isstill merely guessing, and so the era of 100-percent in-coming inspection has passed.According to some arguments, total device testing

is impossible since equipment doesn't exist whichhas the capability to perform the required complextests. Knowledgeable people in the industry will dis-miss this notion out of hand. However, most of theavailable equipment is prohibitively expensive for allbut the largest manufacturers, andagain, economicsbecomes the main driving force.Aware of their inability to perform 100-percent

testing of some components, incoming-inspectionpeople are suggesting that devices be tested accord-ing to their functional end-use in particular products,instead of undergoing generalized testing. This ap-parently good solution is quickly discarded whendevices are used in several types of products or undervarying conditions-as would be the case where a mi-

croprocessor is employed to build a general-purposecomputer. Nevertheless, arguments for specializedtesting are still being presented to and are being ac-cepted by management, often resulting in a decisionto test the components according to their functionalend-use. Unfortunately, this usually means they arepart of a completed PC board, and due to the short-comings of incoming inspection, responsibilities arenecessarily shunted to the board-test phase becausethe board-test equipment is more sophisticated andbetter able to cope with the problem. This redistribu-tion of responsibility caused by device testing prob-lems seems to make sense when viewing the totaltesting picture, but the mission of functional board-test has changed, perhaps dramatically, and requirescareful scrutiny from both design engineers and testengineers. This change in responsibility is shown inthe process-flow diagram of Figure 4.

Another area where increased complexity has hada dramatic effect is that of systems test. After all, it isat systems test that the true complexity of new pro-

Figure 4. Redistribution of responsibility from incoming inspection tofunctional board test in the production-flow process.

October 1979 35

ducts is first seen. Many of the same argumentsabout staffing levels, allowable test 'times, andtechnical education which are heard in the incoming-inspection phase, are also heard in the systems-testphase. And again, management is asking the existingfacility to deal with the increased complexity.However, the increased complexity of final systemsmeans that problems are much more difficult tolocateand identify. Simple statistics show that as thenumber of potentially suspect components (devicesand boards) increase while the system is being con-figured, the probability of system failure increases.Also, there are two additional failure modes which wehave not yet. discussed that reveal themselves atsystems test: interactive failures and design inac-curacies.Th6 myriad problems that fall under the pseudo-

nym of "interactive failures" are largely dynamicperformance-related faults that directly result fromcomplexity. As system complexity grows, signal in-

Figure 5. Redistribution of responsibility from systems test to func-tional board test in the productlon-flow process.

tegrity between PC boards in a particular systemtends to decrease, and electrical noise on power sup-ply lines and other common lines tends to increase.The resulting system environmentmay differ greatlyfrom the board-test environment. There may be evensignificant environmental differences between likesystems. And certainly, as system operating speedincreases, more interactive failures occur. Thus, bydefault, systems test must shoulder the additionalresponsibility for locating dynamic performancefaults which are not identified at incoming inspectionor board test.

The second new failure mode comprises design in-accuracies. The term "design inaccuracies"' correctlyidentifies the problem while leaving the source (theerrant designer) unspecified: faulty design can orig-inate problems at the device level, board level, orsystem level. Certainly, systems emulation at boardtest should locate device- and system-type failures.But little if any systems emulation is being per-formed at board test, perhaps due to the lessonslearned while testing SSI/MSI boards where emula-tion wasn't beneficial, or due to the difficulty en-countered attempting to perform emulation.Remember too, that we designers are being asked

to accomplish bigger and better deeds with currentdevices and tools. This produces logic boards andsystems that run faster and faster, resulting in moreinteractive failures. The lack of complete devicespecifications means that at any design stage (device,board, system), design errors may occur that will af-fect some final designs but not others. More marginaltiming problems are being encountered along withdynamic problems caused by switching noise andcomponent variation. And software/firmware-con-trolled functions are becoming prevalent with theirattendant design oversights.The cry is heard clearly from systems test that

board test must test boards at their full operatingspeed and specifically attempt to find and removedynamic performance problems. Additionally, moreand more demands for system emulation at boardtest are being imposed-a circumstance that ischanging the complexion of the board-test phase.Remember that economics is again at work. By thetime we reach, systems test, the cost to find faultydevices has risen a hundred-fold over incoming-inspection cost, and ten-fold over board-test cost. Soaside from any technical argument to perform moreat board test, there are also strong economicarguments. The results of this redefinition of respon-sibility at board test are shown in Figure 5; board testis under a two-pronged attack.At this point in our discussion, we can begin to

detect an effect on the production process which hasbeen caused by the microelectronics revolution.While all inspection phases have undergone change,board test is caught in the middle and must bear thefull brunt of the attack. To counter this attackbrought on by the microelectronics revolution, boardtest has redefined its methods and procedures to im-prove both economics and throughput.

COMPUTER36

Since the manufacture of PC boards requires someprocessing that includes human skills (to somegreater or lesser degree depending on volumes andautomation), workmanship errors continue topopulate boards. We saw that the most prevalentworkmanship error (solder shorts) was dealt with byadding the assembly-inspection test step to theproduction-flow diagram. Under the urging anddirection of functional board-test production groups,manufacturers of assembly-inspection equipmenthave implemented design changes to include testingcapabilities which allow detection of most workman-ship errors.With its name changed to indicate the newly im-

plemented test-function differences, "in-ciricuit"assembly-inspection equipment not only makesresistance measurements, but also performs true im-pedance measurements and some limited functionaltesting. Most workmanship errors and some devicefailures canr be detected with this equipment,resulting in a significant reduction in the overallworkload imposed on the functional board-test level.In general, functional board testers require the sameamount of time to diagnose one fault as they do todiagnose any other fault. In-circuit test equipment,when used before the functional board-test phase, in-creases the throughput of board test by removingfaults that would otherwise require valuable time tolocate. The net result is that more time is available forboard test to accomplish new tasks that it is beingasked to perform.

To briefly summarize, board test is being asked toperform some device testing responsibilities when in-coming inspection is unable to provide this service.And systems test has requested that high-speed test-ing and system emulation be performed at the board-test phase. Unfortunately, these are not the only newresponsibilities created for the board-test productionphase by the microelectronics revolution. The mis-sion of board test is not only to test boards, but todiagnose the failure-ideally, to a particular failingdevice.

Test system manufacturers are nowtelling us that it is the design

engineer's responsibility to designtestable products.

A large percentage of LSI/VLSI components aredesigned to operate with microcomputers and micro-processors, and therefore are bus-oriented-posingyet another testing problem. With SSI/MSI printedcircuit boards, diagnosis beyond the failing node to aparticular failing device was not generally required,since easily applied repair techniques quicklyrepaired the board. However, the totally sequentialnature of modern boards and their extensive use ofbidirectional buses has so quickly rendered these pro-cedures obsolete that diagnosing a fault to the par-ticular failing device is mandatory.

October 1979

NEW AVAILAILE*from the

IEEE COMPUTER SOCIETY

MICROPOCSSORS

& MICROCOMPUTERS

MICROPROC£ESSORS& MICROCOMPUJTERSMICROPROESSORS

& MICROCOMPUJTERSMICROPROCESSORS

& MICROCOMPUTERS;MICROPROCESSORSOQ, Al 'rvxe^ Af ADlI lTUrn

IEEE COMPUTER SOCIETY INSTITUTE OF ELECTRICAL(OM ~~~~~~ANDELECTRONIC ENGINEERS

To respond to these new needs, manufacturers ofcircuit-board test systems have introduced newequipment and procedures. Naturally, equipment todiagnose failures to the faulty device is available,although some perform this function significantlybetter than others. And high-speed test systems thattest the board functionaly at its designed environ-mental speed are becoming increasingly available.But perhaps more significantly, test system manu-facturers are now telling us that it is the designengineer's responsibility to design testable products.Without the conscious effort to design testability in-to new designs, it is probable that some, if not all, of anew PC board wil not be testable. This is solely themost important effect of the microelectronics revolu-tion.

Certainly, test equipment manufacturers haven'twashed their hands of the testingproblemand are do-ing everything possible to make board testing easier.More sophisticated equipment is continualy avail-able along with better diagnostic procedures and pro-gramming methods. But if the responsibility to test acertain device at the board leveL instead of at incom-ing inspection, is placed on the board-test facility,and the designer has not provided means to gain ac-cess to or control over that device, the situation ishopeless. And so, there is a need for design engineersto understand the testing process and to coordinatetheir work with the test facility.Hardware designers should take the time to learn

what tests are performed by board test and whatfaults they find. Other than basic design rules,3 therehave been many recent articles and seminars dealingwith designing for testability, including a paper inthis issue of Computer. Software and firmwaredesigners have similar responsibilities. Programmedfunctions should include diagnostics and service aidsthat are as precise as time and space permit. It is asimportant for the software designer to designtestability into his program as it is for the hardwaredesigner to design it into his circuit. Understandingthe man-machine interaction during test and repairwill help the software engineer design diagnostics.Remember that these diagnostics will have theirgreatest effect at the system level, which means thatsystems test and field test will benefit. With the costof field testing so high, the software designer has aclear responsibility in this area.In a relatively short period, the functions of elec-

tronic production-line testing have grown to assumea major role, and have raised the relatively newdiscipline of test engineering to a prominent positionin the electronic engineering profession. A lack ofunderstanding and liaison between design and testengineering will produce the inevitable result of ex-cessive production and maintenance costs of finalsystems. There is a dual responsibility here. Testengineering must evaluate and report testabilityproblems back to design engineering for evaluationand consideration as to their applicability to currentrevisions as well as to future designs. And designengineering must review and update its philosophies

and methods to implement maximum testability as amatter of foresight, not hindsight.In addition to the practical matters of maintaining

an awareness of state-of-the-art technology, logicsimulators are playingan increasingly important roleas an aid forboth testanddesign engineers toachievetheir goals. Computer simulation of any particularlogic design can rapidly and precisely pinpoint suchproblems as lack of test points or external controlpoints, whichcan drasticaly affect testability. Anditcan also identify theoretical races or potentiallydangerous timing hazards which are then easilyevaluated and, if necessary, corrected by thedesigner.

When all is said and done, the microelectronicsrevolution has changed the production-flow processin many ways, but in general, everything has becomemore difficult. As a designer you have the opportuni-ty to have a great impact on that process and itsoveral economics. Al that is required is that youshoulder the responsibility given to you and thinkabout testing. *

References

1. J. Hotchkiss, "The Roles of In-Circuit and FunctionalBoard Test in the Manufacturing Process," ElectronicPackaging and Production, Vol. 19, No. 1, Jan. 1979,pp. 47-66

2. D. Wiseman, "To Simulate or Not to Simulate?" Elec-tronics Test; Vol. 2, No. 3, Mar. 1979, pp. 64-45.

3. E. Foley, "Designing MPU Boards for Testability,"Electronics Test, Vol. 2, No. 1, Jan. 1979, pp. 48-54

Eugene Foley is a design engineer in theautomatic functional board test equip-ment design group at Teradyne, Inc.Prior to joining Teradyne, he was a de-sign engineer for Bell Laboratories. Amember of the IEEE Computer Socie-ty, he serves as co-chairman of theSystems and Board Test Subcommit-tee of the Test Technology Committee,

£ / and as a member of the Instrumenta-tion and Measurement Group's Subcommittee on In-strument/Computer Interfaces. He has a BSEE from NewJersey Institute of Technology and an MSEE from North-eastern University.

COMPUTER38