Generating test inputs for embedded control systems ...krogh/papers/ZKH03.pdf · Generating Test Inputs for Embedded Control Systems ... Stateflow demonstration model of an automatic

Generating Test Inputsfor EmbeddedControl SystemsExisting simulation models, with carefullydesigned elements of a genetic algorithm,automatically create test inputs, eliminatingthe task of manual creation.

An essential aspect of ahigh-quality design pro-cess is the developmentof test patterns, or sets oftest inputs, that can beapplied to the final prod-

uct or intermediate instantiations to iden-tify faults and confirm correct systembehavior. In the design of embedded con-trol systems, in particular, the growing useof tools for computer-aided design and sim-ulation increases the prospects for per-forming extensive testing of the controllogic before it is realized in software and im-plemented on the target processor. Therapid growth of the complexity of embed-ded control systems and the demand forshort design cycles have increased the in-terest in effective methods for automatictest generation. This article presents a newmethod for leveraging existing simulationmodels for embedded control system de-signs to generate test inputs automatically,thereby eliminating the time-consumingtask of creating them manually.

Of the two principal ways testing isused to validate system designs, the mostcommon is postproduction to confirm thefunctionality of the final implementationand check for fabrication flaws. The re-sults of the tests are compared with those

August 2003 IEEE Control Systems Magazine 490272-1708/03/$17.00©2003IEEE

By Qianchuan Zhao, Bruce H. Krogh,and Paul Hubbard

©M

AS

TE

RS

ER

IES

Authorized licensed use limited to: Carnegie Mellon. Downloaded on October 24, 2008 at 16:26 from IEEE Xplore. Restrictions apply.

of a so-called golden model to identify defects. To ensurethat all possible defects are exposed, it is important thatthese tests drive the system through the complete range ofoperation where defects may impact behavior. This featureof a set of test inputs is referred to as coverage.

The second common use of testing occurs earlier in thedevelopment process when new instantiations of the sys-tem model are generated, e.g., by automatic code genera-tion or through incremental changes to the design. Duringdevelopment, testing is used in an exploratory manner toidentify overlooked implications of new designs or modifi-cations. Rather than a golden model, an implementation is

usually compared with an executable specification, which isa preliminary, possibly more abstract, design model. Again,coverage is important to provide high assurance that thesystem behavior satisfies the design specifications.

Automatic test generation for embedded control sys-tems is challenging for two major reasons. First, the prob-lem is complex due to the hybrid nature of the controllerand plant composition, which contains both logic (discretestates) and components with continuous dynamics (contin-uous states). Automatic test vector generation is well estab-lished for purely discrete-state systems, particularly fordigital integrated circuits [1]. For example, procedures existto generate test vectors that provide complete coverage todetect standard faults such as stuck-on conditions in digitalcircuits [2]. Analogous methods for mixed-signal integratedcircuits and systems have remained elusive and continue tobe an open and active field of research [1], [3]-[6]. Indeed,undecidability results for hybrid systems indicate that weshould not expect to discover completely algorithmic pro-cedures for test-vector generation for hybrid systems [7].Hence, global search methods that use domain-specificheuristics are used to produce test inputs for mixed-signalcircuits [1], [8], [9]. Similar methods are needed to generatetest vectors for embedded control systems.

The second challenge to developing automatic test inputgeneration tools for embedded control systems is practical-ity. To develop automatic methods, the design must first becaptured in a computer model that includes a representa-tion of the plant as well as an embedded control algorithm.With the growing use of simulation models for design and

evaluation of embedded control systems, computer modelsare available in many cases. These models are not amenableto formal analysis, however, making it impossible to usethem directly in recently proposed methods for test inputgeneration using counterexamples generated by modelcheckers [10]-[12]. In contrast to integrated circuits forwhich very structured models using standard languages aredeveloped during the design process, it is extremely diffi-cult, if not impossible, to extract formal models (such as au-tomata) from existing simulation models of embeddedcontrol systems.

We propose a method for test input generation that ad-dresses both of these challenges. Weformulate test generation as an opti-mization problem that can be solvedusing a genetic algorithm (GA) witheffective heuristics that meet thechallenge of computational com-plexity for hybrid systems. More-over, our approach uses existingsimulation models directly, withvery little modification, to evaluatethe objective functions (also knownas fitness functions) for given inputsignals in the GA implementation.

There is no need to construct new formal models of theembedded controller and the plant to apply our technique.

The use of GAs has been suggested in the context of dis-crete systems for the generation of test vectors and test in-put sequences [13], [14], but not in the context of test inputgeneration for mixed-signal systems. Testing of control sys-tems with GAs has also been studied in [15], but for a differ-ent purpose. In [15], the objective is to evaluate theperformance of controllers in the presence of faults whereasour purpose is to find system inputs that drive the system(without faults) through specified behavioral patterns. Our“golden model” is a given executable specification. Our con-tributions are 1) the formulation of a coverage problem forhybrid systems with discrete and continuous inputs andoutputs, 2) a pragmatic solution to the problem that uses aGA-based algorithm to produce a test input with the speci-fied coverage, and 3) implementation and demonstration ofthe method for industrial-sized MATLAB Simulink/Stateflowsimulation models.

The following section presents a formulation of the gen-eral test input generation problem addressed in this articleand describes the construction of a fitness function to evalu-ate the extent to which a given input sequence meets thecoverage criteria. We then present the GA approach tosearching for test inputs. We have implemented the proce-dure in MATLAB to generate test inputs based on simulationmodels of embedded control systems in MATLAB Simulink-Stateflow [16]. We illustrate the method for the Simulink-Stateflow demonstration model of an automatic transmis-sion and also present computational results for large indus-

50 IEEE Control Systems Magazine August 2003

We formulate test generation as anoptimization problem that can besolved using a genetic algorithmwith effective heuristics that meetthe challenge of computationalcomplexity for hybrid systems.


trial systems. The concluding sectionsummarizes the contributions of thiswork and discusses directions for fur-ther research.

Problem FormulationFigure 1 illustrates the elements of thegeneral problem of test input generation to satisfy a given cov-erage criterion. We discuss each aspect of the figure in turn.

Input SpaceWe consider the input space � = {( , )}u v , where u is a vectorof real-valued, bounded, and piece-wise constant inputs, i.e.,

u u t U t t tnk k k: : ( ) ,ℜ → ℜ = ≤ < + 1,

where U u u u u u ukl u

pl

pu l u∈ × × • •[ , ] [ , ], and ,1 1 L are lower and up-

per bounds of input variables, respectively. v is a dis-crete-valued input that remains constant over the sametime intervals as u, i.e.,

v v t t t tI k I k k: , ( ) ,ℜ → = ∈ ≤ < +Σ Σσ 1,

where Σ I is a finite set of values for v. Equivalently, the infor-mation in v can be captured in the set of pairs

{ }~ ( , ) , , , , ; ,v t j r t tj j j j j I= = > ∈+σ σ0 1 2 1K Σ .

The symbols in Σ I can each be finite-dimensional vectors ifthe system has multiple discrete-valued inputs.

Model Execution (P)A single execution (or simulation) of the model P with the in-put ( , )u v produces the output ( , )y z , which we write as( , ) ( , )y z P u v= .

Output SpaceWe consider an output space � = {( , )}y z , where y is a vectorof real-valued outputs y: ℜ → ℜ ℜ →m

Ozand : Σ is a dis-crete-valued output, where ΣO is an output alphabet. As withthe input v, we can define

{ }~ ( , ) , , , , ; , ,z t j p t t tj j j j j O= = = > ∈+σ σ0 1 2 00 1K Σ ,

where z can change value at arbitrary time instants. As themodel P is an executable simulation, it is assumed that anyintegration routine in P has a finite nonzero minimum inte-gration timestep and contains no algebraic loops, and so p isfinite and the output( , )y z is defined for all inputs( , )u v ∈�.The pairs ( , )u v and ( , )y z represent an extension to the no-tion of primary inputs and primary outputs defined in [13]and [17] to include discrete-valued inputs and outputs.

It may be necessary to instrument the simulation modelto bring the necessary switching values to an output value ifthese are needed in the coverage criterion. This could be

done by adding new transition labels as necessary to suffi-ciently identify the path through the internal discrete statespace and including these labels in the output alphabet. Inmany cases, this instrumentation can be accomplished au-tomatically, for instance with the Performance Tools [18]provided in MATLAB/Simulink, which provide an instru-mented simulation of Simulink models and a data objectwith information such as decision coverage (for Switchblocks and Stateflow states) or condition coverage (forLogic blocks, and Stateflow transitions).

Measure of Coveragef is a bounded map � �→ that provides an integer-valuedcoverage measure for each output( , )y z . The desired cover-age is defined by a set of regions R R k Nk= ={ , , , }1 K , whereeach region Rk is the union of a finite set of intervals inℜ ×m

OΣ , the range of ( , )y z .In particular, f is defined as

( )f y z d y z Rkk

( , ) ( , ),= −∑ ,(1)

where d y z Rk(( , ), ) = 0 if y ti( ), z t Ri k( ) ∈ for some t i, andd y z Rk(( , ), ) > 0 otherwise. This provides a broad range ofpossible coverage criteria. For example, the requirementthat a scalar output y must exceed a threshold c during agiven simulation run is specified by setting R c1 = ∞[ , ). Test-ing that an output ranges over values in each of the intervals[0,1],[2,3],. . . ,[9,10] can be specified by definingR R R1 2 100 1 2 3 9 10= = =[ , ], [ , ], , [ , ]K . With appropriate defini-tions of the regions Rk , this measure of coverage can also beused to ensure that all branches of switching logic are exer-cised (e.g., if-then cases in a Simulink/Stateflow chart orguard conditions in a hybrid automaton [19]). For instance,for a scalar output y and a binary output z, lettingR R1 21 0= ℜ × = ℜ ×{ }, { }will check that the output z takes onboth values during the course of the simulation.

Coverage SpecificationThe last block in Figure 1 defines a coverage criterion

S: { , } ( )� → 0 1 true, false ,

based on the measure of coverage (i.e., S u v f( , ) = ⇔1( ( , )) )P u v = 0 . An input ( , )u v is said to satisfy the specifica-tion S if S u v( , ) =1.

The test input generation problem can now be stated asfollows.

August 2003 IEEE Control Systems Magazine 51

Figure 1. Checking coverage for a given input.


Given a model P and a coverage criterion S, find an input( , )u v within the input space � such that S u v( , ) =1.

A GA Approach toTest Input GenerationThe coverage problem can be considered an optimizationproblem; that is, find ( , )u v ∈� so as to maximize the objec-tive function f. The objective function is constructed suchthat the maximum value of zero is reached if and only if all

the regions Rk are reached. Given the complexity of simula-tion models for embedded control systems, traditional gra-dient-based optimization algorithms cannot be used tosolve this optimization problem. Some type of global searchtechnique is needed.

Global optimization algorithms have received much at-tention in recent years. Several derivative-free algorithmshave been developed, including deterministic direct meth-ods and stochastic methods such as simulated annealingand GAs. The main advantage of the derivative-free algo-rithms is that no simple analytical form of the objectivefunctions is needed. We propose to apply a GA-based globalsearch engine to solve the test input coverage problem.

Figure 2 illustrates the GA approach to test input genera-tion. The top half of the figure represents the calculation ofthe fitness function by explicit simulation of the model fol-lowed by a measure of coverage. The bottom half shows theGA process varying the inputs to the simulation.

We now provide a short summary of GAs as needed forthe solution of the problem. The reader is directed to anystandard GA textbook (e.g., [20]) for a complete descrip-tion; a summary of applications in control systems engi-neering can be found in [21]. GAs work by encoding thearguments of a problem into a chromosome (sometimescalled an individual). A chromosome represents a point inthe solution space (typically a finite-dimensional Booleanspace or real number space). The basic idea of GAs is tomaintain a chromosome set (also know as population) that

evolves iteratively over generationsthrough a process of competitionand controlled variation. Each chro-mosome in the population has an as-sociated fitness that determines thechromosomes that are used to pro-duce the next generation (known asselection). The fitness is also the ob-jective function when solving optimi-zation problems. The goal is to find achromosome with the best fitness.

Chromosomes in the new generation are obtained from thechromosomes in the previous generation by mutation oper-ators, which produce a new chromosome from a single chro-mosome, and by crossover operators, which produce a newchromosome by combining elements of two or more chro-mosomes from the previous generation.

GA approaches are often successful because of the rela-tive ease with which the practitioner is able to bring domainknowledge from the application area (such as relevance ofparameters, sample times, or structural properties of themodel) into the encoding, selection, and crossover mecha-nisms. The relative lack of formal results or categorizations ofGAs has not limited their success in a wide range of problems(again, see [21] for control systems applications). GAs wereoriginally developed for the setting of binary coding ofcombinatorics problems and have been extended to real-val-ued variables [20], [22] called real-coded GAs (RCGA). Thechromosomes in RCGA are vectors of real numbers.

ChromosomesThe performance of GAsdepends greatly on theencoding method. In our im-plementation, the continu-ous inputs u are encoded asreal-coded chromosomesand the discrete-valued in-puts v are encoded with bi-nary coding. The modelinputs are captured withoutloss of information; that is,the input vector can be rec-reated completely from thechromosome (modulorounding errors). In the ex-


Figure 2. Direct simulation of the model is used to measure fitness in a GA approach to test inputgeneration.

To develop automatic methods,the design must first be captured in acomputer model that includes arepresentation of the plant as wellas an embedded control algorithm.


amples provided in the next section, the number of samplesfor each input is held constant, so the size of a single chromo-some remains constant. Although the input variables can bearranged in any order when they are encoded as chromo-somes, a careful selection of the sequence will make the de-sign of crossover operators more efficient. The initialpopulation of chromosomes is seeded with random inputs.

Fitness FunctionThe fitness of a chromosome (an input) is calculated by ex-plicit simulation of the model with the chromosome trans-lated to the input followed by ameasurement of the coverage [see(1)]. The evaluation of the fitnessfunction for a group of chromosomesprovides potentially useful knowl-edge about the problem.The selec-tion mechanism makes use of suchinformation to guide the search pro-cedure. Suppose x x N1 , ,K is a population of chromosomes.A common selection approach assigns a probability of se-lection to x i, P xs

I( ), as

P xf x

f xs

ii

j

j

N( )( )

( )=

=∑

1

.

Other selection mechanisms are reviewed in [23]. Theprobabilistic selection above is employed in the resultsshown in the following section.

Mutation and Crossover OperatorsCrossover and mutation operators should be designed totake advantage of the structure of the problem. Commonlyused crossover operators include simple crossover andarithmetic crossover (see [22] for details and other cross-overs). Given two real-coded chromosomes x x xn

111 1= ( , , )K

and x x xn2

12 2= ( , , )K , the simple crossover produces two

new chromosomes x 1′ and x 2′:

( )( )

x x x x x

x x x x x

i i n

i i n

111 1

12 2

212 2

11 1

′ =

′ =

+

+

, , , , ,

, , , , ,

K K

K K ,

where the position i n∈ −{ , , }1 1K is chosen randomly. Notethat if a binary encoding were used (i.e., each x j

i is a binarybit rather than a real value), then the choice of position iwould be restricted to boundaries between the encoding ofthe real values (see [13] for more discussion about alphabetsize). The arithmetic crossover generates two new chromo-somes, x 1′ ′ and x 2′ ′, as

x x x

x x x

1 1 2

2 2 1

1

1

′ ′ = + −

′′ = + −

λ λ

λ λ

( )

( ) ,

where λ is selected uniformly on a bounded interval.Simple crossover takes advantage of the time-invariant

nature of the hybrid systems in our applications. If one inputsequence performs well in the beginning and another inputsequence performs well at the end of the simulation, thencombining the first piece of the former and the last piece ofthe latter may give a better input sequence if the states ofthe system match at the connecting time instants. Evenwhen the states do not match, simple crossover may pro-duce an input sequence that can be mutated to cover themismatch in the state values. This crossover operator may

be expected to take advantage of the linear component ofthe dynamics to produce better offspring out of goodchromosomes.

A new idea in our work is another crossover operatorthat makes use of the property of superposition. Formally,for given chromosomes x 1 and x 2, the new crossover opera-tor produces two chromosomes,

( )( )( )( )

x x x x x

x x x

i i il

iu

i i

1 1 2

2 1 2

′ = +

′ = −

min max , ,

min max

and

( )( ), , ,x xil

iu

(2)

where i p K=1, , *K , and xil and xi

u are the lower and upperbound of the variable xi, respectively.

The mutation operator used in real-coded GA plays therole of a local search engine that takes advantage of localcorrelations in the problem space. Mutation operators forRCGA include uniform mutation and boundary mutation.For a given chromosome x, uniform mutation randomly se-lects one variable xi and sets it equal to random number si

uniformly distributed on the interval [xil , xi

u]. The result ofmutation is given by

x x x s x xi i i n′ = − +( , , , , , , )1 1 1K K .

The boundary mutation sets si to either the upper boundxi

u or the lower bound xil .

ApplicationsThe MATLAB Simulink demonstration set includes a modelof a drive train for an automobile, including a Stateflowchart for the logic for an automatic transmission [24]. To il-lustrate the GA approach to test input generation, we firstconsider the problem of generating acceleration profiles fortesting the logic of the automatic transmission. The test in-puts might be used to check the design model or could be


We propose to apply a geneticalgorithm-based global search engine to

solve the test input coverage problem


applied to the vehicle itself. In either case, it is desirable tohave a test input with high coverage of the logical switchingin the model.

Figure 3 shows the Simulink model diagram for the auto-matic transmission subsystem. An input (throttle schedule)is pulled from the workspace and the outputs (speed and en-gine rpm) are sent to the workspace, as illustrated in Figure4. The brake schedule has been fixed at the original values.The shift logic for the automatic transmission is shown inFigure 5.

The test generation problem is to find an input throttleschedule such that

a) the vehicle speed exceeds 120 km/hb) the engine speed reaches 4500 rpmc) all states are reached in the switching logic.To test criterion c), we require that the during function

for each state in the Stateflow chart executes at least onceduring the simulation (see [16] for more information aboutthe during function).

This objective is translated into a fitness function of theform described in the previous section. The objective is to

find an input throttle schedule such that f P u v( ( , )) = 0, where

( )f y z d y z Rkk

( , ) ( , ),= −=

∑1

9

,

with d y z Rk(( , ), ) = 0 if y ti( ), z t Ri k( ) ∈ for some t i andd y z Rk(( , ), ) =1otherwise and the regions R kk , , , ,=1 2 9K are

R

R

R j

O

O

j j O

1

2

120

4500

3

= ∞ × ℜ ×= ℜ × ∞ ×= ℜ × ℜ × ∈ =

[ , )

[ , )

{ }, ,

ΣΣ

Σσ K ,9

where ΣO is an alphabet that contains one symbol for eachstate in the Stateflow chart. In the Mathworks PerformanceTools [18], the coverage tool provides a method for per-forming an instrumented simulation that records executionof the individual Simulink blocks and records the executionof the during function for each Stateflow state. We assumethe execution of a state’s during function to be equivalent tothe appearance of the state’s symbol in ΣO at the output.

The sampling time for the throttleschedule is selected as 5 s, and the test isrun for a total time of 30 s. Each chromo-some is hence a six-element sequence,U U U= 1 6, ,K , and the continuous-timeinput throttle value is given as

u t U t t tk k kthrottle ( ) ,= ≤ < + 1,

where Uk ∈[ , ]0 100 is the bound on pos-sible throttle input values.


Figure 3. The automatic transmission example provided in the MATLAB Simulink demonstration package reduced to a subsystem with aninput (throttle schedule) and outputs (speed and engine RPM).

Figure 4. The automatic transmission example as a subsystem with input and outputsconnected to the workspace.


Typical crossovers, mutations, and selection mecha-nisms are used from the GAOT package [25], as well as thelinear combination operator and restricted simple cross-over described in the previous section.Each crossover (mutation) has equalopportunity to be applied when a cross-over (mutation) operation is requested.A population of 20 chromosomes wasused, and, in most cases, a chromo-some with fitness 0 was produced in fivegenerations or less. A typical inputthrottle schedule and output RPM andspeed are shown in Figure 6. The chro-mosome in this case is

U = 819 314 979 85 2 94 2 82 2. , . , . , . , . , . .

For this execution, all seven states inthe Stateflow chart became active atsome point during the execution.

Several industrial examples fromFord Motor Company and General Elec-tric Transportation Systems have beenrun, in addition to the example de-scribed above. The results are shown inTables 1 and 2.

For the small examples in Table 1, theGA produced inputs with the right cover-age in reasonable computation times

shown in the last column. Note that the number of executionsof the model does not necessarily equal the population sizemultiplied by the number of generations because the fitness


Figure 5. The shift logic in the stateflow block for the automatic transmission.

Figure 6. Inputs (throttle) and outputs (speed and engine RPM) for the generated testinput.


does not have to be reevaluated for chromosomes that are car-ried into the next generation without being altered.

We have applied our method to large-scale productionmodels from industry. The results in Table 2 are for the GELocomotive Traction Pilot model consisting of 1,478Simulink blocks, 50 Stateflow states, and 15 input signals.The sample time and duration of simulation time were 0.01 sand 0.1 s, respectively. Results for three sizes of populationof GA are shown in Table 2. For each population size, we ran20 independent experiments. The maximum number ofmodel evaluations for each experiment was set to 3,000. Ifthe GA failed to find a test vector with a fitness functionequal to zero (indicating it satisfies the desired coverage cri-teria) after 3,000 simulations, we terminate the experimentwith the assumption that either a solution does not exist(which was not the case for the system) or the chromosomepopulation has become “stuck” in a local maximum.

Table 2 shows that the success rate of the GA method isstill very high (90%), provided the population size is care-fully chosen. With a fixed limit on the number of simulations,the population size should not be too small or too large. Forthis example, a population size of 16 is better than four or 64.If the population size is too small, the power of the GA to ex-plore several possible locations in the search space in paral-lel is not fully exploited. If the population size is too large,the relative number of crossover and mutation operationsthat can be performed is limited because of the large num-ber of chromosomes. At the extreme case in which the pop-ulation size equals the model execution limit, GA willdegenerate into a pure random search.

We compared the GA approach to random search for theexamples in Tables 1 and 2 and found that for the first threesmall examples, random sampling of the input space could

eventually generate a solution, but with many more execu-tions of the simulation model than required by the GA ap-proach. For the GE Traction Pilot example in Table 2, a randomsearch never produced a suitable input despite very large sam-ple sizes (3,000). Comparisons with approaches based on for-mal methods are not possible because there is no apparentway to build the required abstract models from the given simu-lation models, even for the small examples.

In summary, the results of this section indicate that ourGA method can automate the generation of test inputs forreal-world problems. In most cases, test inputs can be gen-erated within a reasonable time even for complex controlsystems with hybrid dynamics or large-scale production-size models for which formal methods and random searchmethods cannot be used.

ConclusionsThis article presents a new technique for generating test in-puts for embedded control systems using existing simula-tion models. The power of the technique comes fromcarefully designed elements of a GA. The construction of thefitness function, the chromosomes, and the introduction ofspecial crossover and mutation operators all contribute toan algorithm that finds test inputs much faster than randomsearch. The complexity of the problem prohibits the use offormal methods or traditional gradient-based optimization.The key feature of the proposed method is that no analyticalmodel is required; tests are generated by nothing other thanrepeated execution of the simulation model. We have dem-onstrated the effectiveness of the method for large produc-tion-size models from industry.

There are several directions for future research. Thecurrent fitness function is simply the sum of the indicatorfunctions for sets of signals, representing the satisfactionof various criteria for acceptable test inputs. Fitness func-tions that provide better measures of the proximity ofcandidate test inputs to a solution might help speed upthe convergence of the GA. To make the method effectivefor designers of embedded control systems, a user inter-face should be developed that helps with the construc-tion of the fitness function for specific problems.Developing methods for exploiting more domain knowl-edge in the construction of crossover and mutation oper-ators is also of interest.


Example Number ofBlocks (SF States)

Number ofInputs

Sample Time(Sim Time)

GA Popula-tions (Genera-tions)

Model Executions(Compute Time)

Test drive (illustrative) 78 (7) 1 5.0 (30.0) 20 (5) 63 (45 s)

GE RTCembedded system

24 (4) 4 1.0 (10.0) 20 (2) 37 (30 s)

Ford drive train 232 (11) 6 1.0 (50.0) 40 (10) 372 (552 s)

GAPopulation

Number ofSuccessful Runs

Average Number ofModel Execution(Compute Time)

4 18/20 1044 (1,000 s)

16 18/20 518 (500 s)

64 10/20 299 (300 s)


AcknowledgmentsThe examples in this article were provided by Dr. SureshReddy of General Electric Transportation Division and Dr.Ken Butts and William Milam of Ford Scientific ResearchLabs; we gratefully acknowledge their support of this work.We would like thank the editor and the anonymous review-ers for helpful suggestions. This work was supported in partby Ford Motor Company, General Electric TransportationSystems, the National Science Foundation, Army ResearchOffice, and the Pennsylvania Infrastructure Technology Alli-ance, a partnership of Carnegie Mellon, Lehigh University,and the Commonwealth of Pennsylvania’s Department ofEconomic and Community Development. Q.C. Zhao wasalso supported by NSFC under Grants 60074012 and60274011, Ministry of Education of China and Tsinghua Uni-versity project.

References[1] B. Vinnakota, Analog and Mixed-Signal Test. Englewood Cliffs, NJ:Prentice-Hall, 2001.[2] M. Abramovici, M. Breuer, and A. Friedman, Digital Systems Testing andTestable Design. New York: IEEE Press, 1995.[3] M. Burns and G. Roberts, An Introduction to Mixed-Signal IC Test and Mea-surement. New York: Oxford Univ. Press, 2001.[4] M. Soma, S. Huynh, J. Zhang, S. Kim, and G. Devarayanadurg, “HierarchicalATPG for analog circuits and systems,” IEEE Des. Test Comput., vol. 18, no.1pp. 72-81, 2001.[5] A. Gupta, S. Malik, and P. Ashar, “Toward formalizing a validation method-ology using simulation coverage,” in Proc. 34th Design Automation Conf., Ana-heim, CA, 1997, pp. 740-745.[6] S. Huynh, J. Zhang, S. Kim, G. Devarayanadurg, and M. Soma, “Effcient testset design for analog and mixed-signal circuits and systems,” in Proc. 8thAsian Test Symp., Shanghai, China, 1999, pp. 239-244.[7] T. Henzinger, P.W. Kopke, A. Puri, and P. Varaiya, “What’s decidable abouthybrid automata,” J. Comput. Syst. Sci., vol. 57, no.1, pp. 94-124, 1998.[8] J. Tofte, C.K. Ong, J.L. Huang, and K.T. Cheng, “Characterization of apseudo-random testing technique for analog and mixed-signal built-in-self-test,”in Proc. 18th IEEE VLSI Test Symp., Montreal, Canada, 2000, pp. 237-246.[9] A. Gupta, S. Malik, and P. Ashar, “Toward formalizing a validation method-ology using simulation coverage,” in Proc. IEEE/ACM Int. Conf. Com-puter-Aided Design, San Jose, CA, 2001, pp. 286-292.[10] P. Amman, P.E. Black, and W. Majurski, “Using model checking to generatetests from specifications,” in Proc. 2nd IEEE Int. Conf. Formal EngineeringMethods (ICFEM’98), Brisbane, Australia, 1998 pp. 46-54.[11] P. Ammann and P.E. Black, “Abstracting formal specifications to generatesoftware tests via model checking,” in Proc. 18th Digital Avionics Systems Conf.(DASC99), St Louis, MO, 1999, vol. 2, pp. 10.A.6.[12] A. Gargantini, and C. Heitmeyer, “Using model checking to generate testsfrom requirements specifications,” in Proc. Joint 7th Eu. Software EngineeringConf. and 7th ACM SIGSOFT Int. Symp. Foundations of Software Engineering,Toulouse, France, 1999, pp. 146-162.[13] M. Rudnick, J. Patel, G. Greenstein, and T. Niemann, “A genetic frameworkfor test generation,” IEEE Trans. Computer-Aided Design, vol. 16, no. 9, pp.1034-1044, Sept. 1997.[14] M. Srinivas and L. Patnaik, “A simulation-based test generation schemeusing genetic algorithms,” in Proc. 6th Int. Conf. VLSI Design, Bombay, India,1993, pp. 132-135.[15] A. Schultz, J. Grefenstette, and K.D. Jong, “Learning to break things:Adaptive testing of intelligent controllers,” in The Handbook of EvolutionaryComputation, T. Bäck, D. Fogel, and Z. Michalewicz, Eds. New York: IOP Pub-lishing and Oxford Univ. Press, 1997, G3.4 .[16] The Mathworks: Developers of MATLAB and Simulink for technical com-puting [Online]. Available: http://www.mathworks.com

[17] M.K. Iyer and M.L. Bushnell, “Effect of noise on analog circuit testing,” inProc. 16th IEEE VLSI Test Symp, Monterey, CA, 1998, pp. 138-144.[18] The Mathworks-Simulink performance tools—Simulink model coverage[Online]. Available: http://www.mathworks.com/products/slperftools/[19] R. Alur, T. Henzinger, and P.H. Ho, “Automatic symbolic verification of em-bedded systems,” IEEE Trans. Software Eng., vol. 22, no. 3, pp. 181-201, 1996.[20] A. Wright, “Genetic algorithm for real parameter optimization,” in Foun-dations of Genetic Algorithms 1. San Mateo, CA: Morgan Kaufmann, 1991, pp.105-218.[21] P. Fleming and R. Purshouse, “Genetic algorithms in control systems en-gineering,” Dept. Automatic Control and Systems Engineering, Univ. of Shef-field, Res. Rep. 789, 2001.[22] F. Herrera, M. Lozano, and J. Verdegay, Tacking Real-Coded GAs: Opera-tors and Tools for Behavioral Analysis. New York: IEEE Press, 1996.[23] T. Back, F. Hoffmeister, and H.P. Schwefel, “Extended selection mecha-nisms in genetic algorithms,” in Proc. 4th Int. Conf. Genetic Algorithms, SanDiego, CA, 1991, pp. 92-99.[24] Mathworks (2002), Stateflow, car simulation demo. [Online]. Available:http://www.mathworks.com/products/demos/stateflow/sfcar.html[25] C. Houck, J. Joines, and M. Kay, Genetic algorithm optimization toolbox(GAOT) for MATLAB 5 [Online]. Available: http://www.ie.ncsu.edu/mi-rage/GAToolBox/gaot/

Qianchuan Zhao received the B.E. degree in automaticcontrol, the B.Sc. degree in applied mathematics, and thePh.D. in control theory and its applications in 1992, 1992,and 1996, respectively, from Tsinghua University. He is cur-rently an associate professor in the Department of Automa-tion, Tsinghua University. He also holds visiting positions atCarnegie Mellon University and Harvard University. He is anassociate editor for the Journal of Optimization Theory andApplications. His current research interests are DEDS theoryand the optimization of complex systems. He can bereached at the Department of Automation, Tsinghua Univer-sity, Beijing 100084, China, and the Department of Electricaland Computer Engineering, Carnegie Mellon University,5000 Forbes Ave, Pittsburgh, PA 15213, U.S.A,[email protected].

Bruce H. Krogh is professor of electrical and computer en-gineering at Carnegie Mellon University, Pittsburgh, Penn-sylvania. He was an associate editor of IEEE Transactions onAutomatic Control and Discrete Event Dynamic Systems: The-ory and Applications and founding editor-in-chief of IEEETransactions on Control Systems Technology. His current re-search interests include discrete event systems, hybrid dy-namic systems, and synthesis and verification of embeddedcontrol system designs.

Paul Hubbard received the B.Sc. and M.Sc. degrees, both inmathematics and engineering, from Queen’s University,Kingston, and the Ph.D. degrees in electrical engineering fromMcGill University, Montreal. He has worked previously withLockheed-Martin Canada on electronic counter-measures,with BBN Technologies on command and control systems, andat Carnegie Mellon University on automatic test generation.He is currently a defense scientist at Defense Research and De-velopment Canada, Ottawa, Ontario, where he works on theautonomous intelligent control of unmanned systems.



Documents

Generating test inputs for embedded control systems ...krogh/papers/ZKH03.pdf · Generating Test Inputs for Embedded Control Systems ... Stateflow demonstration model of an automatic