Desire Gettingstarted

Embed Size (px)

Citation preview

  • 8/3/2019 Desire Gettingstarted

    1/30

    David Scuse September, 20111

    DESIRE NEURAL NETWORK SYSTEM

    Introduction:

    The Desire system is a neural network modelling tool that allows the user to build models ofneural networks. Using the system, almost any neural network model can be developed,including those that require differential equations. This is in contrast to conventional neuralnetwork tools that contain a variety of pre-defined models; the user can modify a modelsparameters but can not define new models (i.e. the system is a black box). With the Desiresystem, the neural network is defined using statements in the Desire programming language sothe user can see how each portion of the neural network performs its processing and, if necessary,can modify the processing. The Desire language is a high-level language so the user does not getlost in the details (as happens with neural network toolkits written in C/C++).

    Running the DesireW (Windows) System

    The Desire system (currently version 15.0) runs under Windows and Linux. (The followinginstructions refer to the Windows version.) To begin, double-click the command fileDesireW.bat to launch the Desire system. This batch file opens a Desire editor window and theDesire command window.

    Figure 1: The Desire Editor Window and the Desire Command Window

  • 8/3/2019 Desire Gettingstarted

    2/30

    David Scuse September, 20112

    Load a file into the Editor window either by dragging and dropping it onto the editor windowor by using the editors Open command). You may store your files in the same folder as theDesire system or in any other convenient folder. Do not use Windows to associate Desire sourcefiles (.src and .lst) with the Desire Editor doing so causes the OK button in the Editorwindow to stop functioning correctly.

    Figure 2: Loading A Desire Source File

  • 8/3/2019 Desire Gettingstarted

    3/30

    David Scuse September, 20113

    Transfer the file to Desire by clicking on the OK button in the editor window (or using the

    shortcut alt-T O). Then type the command erun (or zz) into the Desire command window.This command causes the program to be executed.

    Figure 3: Transferring A Desire Source File to the Desire Command Window

  • 8/3/2019 Desire Gettingstarted

    4/30

  • 8/3/2019 Desire Gettingstarted

    5/30

    David Scuse September, 20115

    Figure 5: Graph Window Menu Items

    Similarly, the standard Windows facilities for copying and pasting the contents of the Desire

    command window are available by clicking the top left icon of the window.

    Figure 6: Command Window Menu Items

    By modifying the Properties of the command window, you can switch from the default white texton a black background to black text on a white background.

    Source Files

    Normally, Desire source files have the extension .src. Files with the .src extension do not

    contain line numbers while numbered source files have the extension .lst. If a source filealready contains line numbers, the line numbers can be removed by using the Desire commandwindow command keep 'xxx'to create an unnumbered version of the file on disk (the file willbe named xxx.src). Internally, Desire still keeps a file numbered so that it can generate errormessages and so that the programmer can refer to individual lines.

    In general, unnumbered source files are preferable to numbered source files in Desire.

  • 8/3/2019 Desire Gettingstarted

    6/30

    David Scuse September, 20116

    To view the line numbers used in the Desire command window for the current source file, typethe Desire command list+. This causes the file to be listed with line numbers in the commandwindow. To list a specific line, type list+ 2200 (or, for a range of lines, type list+ 2200-2300) in the Desire command window. To save a copy of the current file with line numbers, typelist+ 'filename.lst' This causes a copy of the file to be saved in the Desire directory (which

    is not necessarily the same directory as the original source file). If you want to save the .lstfilein a different directory, add the complete path to the file in the command; for example, list+'D:\MyFolder\filename.lst'

    A Quick Tour: Learning the OR Patterns

    The following sections briefly describe a simple single-layer, feedforward Desire neural network.The SLFF1 program in Figure 7 defines a neural network that attempts to learn the logical OR

    patterns ({0,0} {0}, {1,1} {1}, etc.). The SLFF1 program defines a 2-1 network ofthreshold units and uses the delta training rule to learn the patterns. The output of the SLFF1program is shown in the Graph Window in Figure 8. Each of the generated symbols ( )

    represents the value of the tss (the sum of squares of the error) at a particular point in time. Thescreen is organized in a graph format: the vertical axis represents the tss and ranges from -scale (represented by -) at the bottom to +scale (represented by +) at the top. The value of scaleis defined in the program and is displayed at the bottom of the screen. In this example, thevertical axis ranges from -4 at the bottom to +4 at the top. The horizontal axis represents thetime, t, and begins at 1 on the left and ranges to the total number of training iterations on theright. The number of iterations of the program is the number of training epochs times thenumber of patterns. A training epoch is the time period during which each pattern is presentedonce to the network. The number of training epochs is defined in the variable Nepochs. In theprogram below, the number of program iterations is 100 (25 * 4). The horizontal axis on thegraph displays the values of time, t, beginning at 1 and continuing for 100 iterations.

    ------------------------------------------------------ Single-Layer Feedforward Network (SLFF1)-- activation function: threshold-- learning rule: delta learning--------------------------------------------------Npat=4 | Ninp=2 | Nout=1ARRAY Layer1[Ninp],Layer2[Nout],Weights[Nout,Ninp]ARRAY Target[Nout],Error[Nout]ARRAY INPUT[Npat,Ninp],TARGET[Npat,Nout]---------------------------------------------------- Define the OR training patterns--------------------------------------------------data 0,0;1,0;0,1;1,1 | read INPUTdata 0;1;1;1 | read TARGET---------------------------------------------------- Initialize Run-Time Variables

    --------------------------------------------------scale=4 | display R | display N-16 | display W 362,0Lrate=0.2 | Nepochs=25---------------------------------------------------- Learn the weights--------------------------------------------------tss=0t=1 | NN=Nepochs*Npat | TMAX=NN-1drunSTOP--------------------------------------------------DYNAMIC

  • 8/3/2019 Desire Gettingstarted

    7/30

    David Scuse September, 20117

    ---------------------------------------------------- The following statements learn the weights-- The statements are executed once for each training pattern-- for a total of Nepochs * Npat learning repetitions--------------------------------------------------iRow=tVECTOR Layer1=INPUT#VECTOR Target=TARGET#VECTOR Layer2=swtch(Weights*Layer1)

    VECTOR Error=Target-Layer2DELTA Weights=Lrate*Error*Layer1DOT SSQ=Error*Errortss=tss+SSQdispt tss---------------------------------------------------- The following statements are executed once per training epoch-- (instead of once per training pattern)--------------------------------------------------SAMPLE Npattss=0 | -- reset tss to zero for next epoch

    Figure 7: SLFF1 OR Patterns Program

    Figure 8: SLFF1 OR Patterns Program Output

  • 8/3/2019 Desire Gettingstarted

    8/30

    David Scuse September, 20118

    In Figure 8, the value of the tss increases as each pattern is processed during an epoch and it isdifficult to determine the actual system error by observing the graph. To simplify the graph, thetss is normally stored in a different variable (such as TSS) which is not modified until the end ofeach epoch.

    ------------------------------------------------------ Single-Layer Feedforward Network (SLFF1b)-- activation function: threshold-- learning rule: delta learning--------------------------------------------------Npat=4 | Ninp=2 | Nout=1ARRAY Layer1[Ninp],Layer2[Nout],Weights[Nout,Ninp]ARRAY Target[Nout],Error[Nout]ARRAY INPUT[Npat,Ninp],TARGET[Npat,Nout]---------------------------------------------------- Define the OR training patterns--------------------------------------------------data 0,0;1,0;0,1;1,1 | read INPUTdata 0;1;1;1 | read TARGET---------------------------------------------------- Initialize Run-Time Variables--------------------------------------------------

    scale=4 | display R | display N-16 | display W 362,0Lrate=0.2 | Nepochs=25---------------------------------------------------- Learn the weights--------------------------------------------------

    tss=0 | TSS=scale

    t=1 | NN=Nepochs*Npat | TMAX=NN-1drunwrite TSSSTOP--------------------------------------------------DYNAMIC---------------------------------------------------- The following statements learn the weights and bias terms-- The statements are executed once for each training pattern-- for a total of Nepochs * Npat learning repetitions--------------------------------------------------iRow=t

    VECTOR Layer1=INPUT#VECTOR Target=TARGET#VECTOR Layer2=swtch(Weights*Layer1)VECTOR Error=Target-Layer2DELTA Weights=Lrate*Error*Layer1DOT SSQ=Error*Errortss=tss+SSQ

    dispt TSS

    ---------------------------------------------------- The following statements are executed once per training epoch-- (instead of once per training pattern)--------------------------------------------------SAMPLE Npat

    TSS=tss

    tss=0 | -- reset tss to zero for next epoch

    Figure 9: SLFF1b OR Patterns Program

    The program in Figure 9 displays the tss after the program has been modified to use the statementdispt TSS instead of dispt tss. Note that the name of the variable being graphed (tss inFigure 8 and TSS in Figure 10) appears at the bottom of the graph. As the number of epochsincreases, the graph of the TSS becomes smoother. The TSS is the value of the sum of the psssfor the previous epoch.

  • 8/3/2019 Desire Gettingstarted

    9/30

    David Scuse September, 20119

    Figure 10: SLFF1b OR Patterns Program Output

    As can be seen in Figure 10, the TSS converges to 0, meaning that the patterns can be learned bythis network.

    Learning the AND Patterns:

    We can modify the program to use the AND patterns as shown below:

    data 0; 0; 0; 1 | read TARGET | -- (AND patterns)

  • 8/3/2019 Desire Gettingstarted

    10/30

    David Scuse September, 201110

    If we attempt to learn the AND patterns, the following output is generated.

    Figure 11: SLFF1b AND Patterns Program Output

    For some reason, the AND patterns can not be learned with the same network.

    Adding an additional input to the output unit makes it possible to learn the AND patterns. Thisadditional term is often referred to as a bias term. The bias term always has an input of1 andcontains an adjustable weight.

    ------------------------------------------------------ Single-Layer Feedforward Network (SLFF1c)-- activation function: threshold-- learning rule: delta learning--------------------------------------------------Npat=4 | Ninp=2 | Nout=1

    ARRAY Layer1[Ninp],Layer2[Nout],Bias[Nout],Weights[Nout,Ninp]

    ARRAY Target[Nout],Error[Nout]ARRAY INPUT[Npat,Ninp],TARGET[Npat,Nout]---------------------------------------------------- Define the AND training patterns

    --------------------------------------------------data 0,0; 0,1; 1,0; 1,1 | read INPUTdata 0; 0; 0; 1 | read TARGET---------------------------------------------------- Initialize Run-Time Variables--------------------------------------------------scale=4 | display R | display N-12 | display W 362,0Lrate=0.2 | Nepochs=25---------------------------------------------------- Learn the weights--------------------------------------------------tss=0 | TSS=scalet=1 | NN=Nepochs*Npat | TMAX=NN-1

  • 8/3/2019 Desire Gettingstarted

    11/30

    David Scuse September, 201111

    drunwrite TSSSTOP

    --------------------------------------------------DYNAMIC---------------------------------------------------- The following statements learn the weights and bias terms-- The statements are executed once for each training pattern

    -- for a total of Nepochs * Npat learning repetitions--------------------------------------------------iRow=tVECTOR Layer1=INPUT#VECTOR Target=TARGET#

    VECTOR Layer2=swtch(Weights*Layer1+Bias)

    VECTOR Error=Target-Layer2DELTA Weights=Lrate*Error*Layer1

    Vectr delta Bias=Lrate*Error

    DOT SSQ=Error*Errortss=tss+SSQdispt TSS---------------------------------------------------- The following statements are executed once per training epoch-- (instead of once per training pattern)--------------------------------------------------SAMPLE NpatTSS=tss

    tss=0 | -- reset tss to zero for next epoch

    Figure 12: SLFF1c AND Patterns Program

    As can be seen in the output below, the AND patterns can now be learned correctly.

    Figure 13: SLFF1c AND Patterns Program Output

  • 8/3/2019 Desire Gettingstarted

    12/30

    David Scuse September, 201112

    The SLFF1 Program:

    In this section, we examine the commands used in the various SLFF1 programs above.

    The first statement in the program (excluding the initial comments) assigns initial values to thevariables Npat, Ninp, and Nout. These are simple variables (scalars) and the values assignedare simple numeric values. Note that all variable names are case sensitive and must be typedcorrectly.

    The next set of statements allocates storage for the vectors and matrices that will be used by thesystem.

    The data statement defines the values to be assigned to the INPUT matrix. Using the datastatement makes assigning values to a vector or matrix simpler than reading the values from afile or assigning values to the individual elements of a vector or matrix. The read statementcopies the values that are in the current data list into the named vector or matrix. Thus, theINPUT matrix contains 4 rows with 2 values per row (i.e. the 4 input patterns) and the TARGET

    matrix contains 4 rows with 1 value per row (the 4 output patterns).

    The display statement is used to set various Desire Graph Window display options. display Rgenerates thick dots on the screen while display Q generates thin dots. display N-x generatesblack lines on a white background; display Nx generates white lines on a black background.The value of x in these two statements identifies the starting colour for the graphs that aregenerated. The statement display W 362,0 defines the position of the top-left corner of thegraph window on the screen. This particular co-ordinate (362,0) works nicely with 1024x768resolution.

    The next several statements assign values to scalar variables in preparation for the actual learning

    process.

    The drun statement causes the statements in the dynamic segment to be compiled and thenexecuted NN times. The two variables that control the number of times that the dynamicsegment is executed must have been initialized in the interpreted segment. The initial timevariable, t, is normally initialized to 1. The number of times that the dynamic segment is to beexecuted is defined in the variable NN which must also be initialized by the programmer in theinterpreted segment. The time variable t is automatically incremented by Desire by 1 each timethat the statements in the dynamic segment are executed until t becomes greater than NN. Oncethe statements in the dynamic segment have been executed the specified number of times (NN),the statement that follows the drun statement in the interpreted segment is executed. This

    statement is normally a STOP statement.

    In this program, the statements in the dynamic segment are executed 100 times. Note that mostof the statements in the dynamic segment of the program are preceded by one of the keywords:VECTOR, Vectr, MATRIX, or DOT. This indicates that the operation is performed on allelements of the associated vector or matrix.

    The first statement in the dynamic segment, iRow=t, assigns the current iteration number to the

  • 8/3/2019 Desire Gettingstarted

    13/30

    David Scuse September, 201113

    variable iRow. This variable is used implicitly in the next statement to extract a training patternfrom the matrix of training patterns.

    The statement VECTOR layer1=INPUT# extracts pattern number iRow from the set of trainingpatterns stored in the matrix INPUT and stores the pattern in the vector layer1. The pattern that

    is extracted is pattern ((iRow-1 mod Npat)+1). In this example, the first time that the dynamicsegment is executed, the pattern in row 1 is extracted. The next time that the dynamic segment isexecuted, the pattern in row 2 is extracted. Executing the dynamic segment 4 times constitutesone training epoch, with each pattern begin processed once. Normally, NN (the number of timesthe dynamic segment is executed) is a multiple of the number of patterns (Npat).

    The statement VECTOR Layer2=swtch(Weights*Layer1+Bias) computes the result ofmultiplying the matrix Weights by the current input pattern (Layer1) and then adding the Biasterm. The resulting vector is sent through the threshold function (swtch is a built-in function)and the output is then assigned to the output vector (Layer2). In this program, the output layerconsists of a single unit, but in general, the output layer may consist of any number of units.

    The statement VECTOR Error=Target-Layer2 computes a vector (Error) which contains thedifference between the actual output (Layer2) and the desired output (Target).

    The statement DELTA Weights=Lrate*Error*Layer1modifies the matrix Weights based on thedifference between the actual output and the desired output.

    The statement Vectr delta Bias=Lrate*Error modifies the vector Bias based on thedifference between the actual output and the desired output. The statement DOTSSQ=Error*Errorcomputes the sum of the squares of the individual error terms.

    The statement dispt TSSgraphs the current value of the variable TSS.

    The statements that follow the SAMPLE Npat statement are executed only at the end of eachepoch (i.e. once every Npat times). In this program, the statements that follow the SAMPLEstatement first copy the current value of the tss into the variable TSS and then reset the variabletss back to zero for the next epoch.

    TSS=tsstss=0 | -- reset tss to zero for next epoch

    As can be seen in Figure 13, the TSS begins at scale, then is changed to 1, increases to 3,remains at 3 for another epoch, then decreases to 2, decreases to 1, and finally decreases to 0, andremains at 0. The TSS remains the same for 4 consecutive values of t (i.e., a training epoch)because the TSS variable is updated only at the end of each epoch.

    The > (displayed in the command window) is the Desire prompt character; this character isdisplayed after the system has completed its processing and is waiting for the users nextcommand. To terminate the Desire system, type bye.

  • 8/3/2019 Desire Gettingstarted

    14/30

    David Scuse September, 201114

    X-OR Patterns:

    We now attempt to learn the X-OR patterns. The only program modification required is:

    data 0; 1; 1; 0 | read TARGET | -- (X-OR patterns)

    The output is shown below.

    Figure 14: SLFF1d X-OR Patterns Program Output

    Even though we developed a more powerful network with the addition of bias terms, the X-ORpatterns can not be learned.

    Perceptron Neural Network:

    We now improve the neural network by initializing the weights and the bias terms to smallrandom values. Using random values may help the network reach a solution that it could notreach if the weights and the bias terms are initialized to 0. This neural network is referred to as aPerceptron.

  • 8/3/2019 Desire Gettingstarted

    15/30

    David Scuse September, 201115

    ------------------------------------------------------ Single-Layer Feedforward Network (SLFF1e)-- activation function: threshold-- learning rule: delta learning--------------------------------------------------Npat=4 | Ninp=2 | Nout=1ARRAY Layer1[Ninp],Layer2[Nout],Bias[Nout],Weights[Nout,Ninp]ARRAY Target[Nout],Error[Nout]

    ARRAY INPUT[Npat,Ninp],TARGET[Npat,Nout]---------------------------------------------------- Define the X-OR training patterns--------------------------------------------------data 0,0; 0,1; 1,0; 1,1 | read INPUTdata 0; 1; 1; 0 | read TARGET------------------------------------------------------------- Randomize the weight matrix-----------------------------------------------------------WFactor = 1for i=1 to Nout

    for j=1 to NinpWeights[i,j] = ran() * WFactornext

    Bias[i] = ran() *WFactornext

    ---------------------------------------------------- Initialize Run-Time Variables

    --------------------------------------------------scale=4 | display R | display N-16 | display W 362,0Lrate=0.1 | Nepochs=25---------------------------------------------------- Learn the weights--------------------------------------------------tss=0 | TSS=scale | t=1 | NN=Nepochs*Npat | TMAX=NN-1drunSTOP

    --------------------------------------------------DYNAMIC---------------------------------------------------- The following statements learn the weights and bias terms-- The statements are executed once for each training pattern-- for a total of Nepochs * Npat learning repetitions--------------------------------------------------iRow=t

    VECTOR Layer1=INPUT#VECTOR Target=TARGET#VECTOR Layer2=swtch(Weights*Layer1+Bias)VECTOR Error=Target-Layer2DELTA Weights=Lrate*Error*Layer1Vectr delta Bias=Lrate*ErrorDOT SSQ=Error*Errortss=tss+SSQdispt TSS---------------------------------------------------- The following statements are executed once per training epoch-- (instead of once per training pattern)--------------------------------------------------SAMPLE NpatTSS=tsstss=0 | -- reset tss to zero for next epoch

    Figure 15: SLFF1e X-OR PatternsPerceptron Program

  • 8/3/2019 Desire Gettingstarted

    16/30

    David Scuse September, 201116

    Figure 16: SLFF1e X-OR Patterns Perceptron Program Output

    Unfortunately, even with random weights, the X-OR patterns can not be learned.

    In the 1950s it was shown that a Perceptron network could not learn the X-OR patterns; this

    result caused neural network research to slow down significantly; it wasnt until several decadeslater that researchers solved the X-OR problem and research began moving forward morerapidly.

  • 8/3/2019 Desire Gettingstarted

    17/30

    David Scuse September, 201117

    Desire Programming Statements

    Program Segments:

    Each Desire program consists of two types of segments: an interpreted experiment-protocolsegment and a compiled dynamic program segment. The experiment-protocol segmentconsists of statements that define the structure of a model and the models parameters. Thesestatements are interpreted. The dynamic program segment contains the difference and/ordifferential equations that define the behaviour of the model over time. These statements arecompiled. The experiment-protocol program statements are placed first in the program; thedynamic program statements follow the interpreted statements; a DYNAMIC statement separatesthe two program segments.

    Some statements can be placed in either an interpreted segment or in a dynamic segment; otherstatements can be placed in one type of segment but not in the other type of segment. Forexample, array declarations can be placed only in an interpreted segment; assignment statementscan be placed in either type of segment; and most of the vector/matrix manipulation statementscan be placed only in dynamic program segments.

    Statement Syntax:

    The Desire syntax is straightforward. Statement lines are terminated by a carriage return. Forexample,

    Lrate=0.5

    Statement lines may contain several statements, using the | character as a statement separator.

    Lrate=0.5 | Yrate=1.3

    The characters -- indicate that a statement is a comment (note that the blank is required).

    -- this is a comment

    If a comment is defined on the same line as another statement, the preceding statement must beterminated using the | statement separator. For example,

    Lrate=0.5 | -- define the learning rate

    Desire is case sensitive; the two names Lrate and lrate refer to different variables. Some systemcommands (such as the STOP command) must be in upper case. Symbols must start with a letterand may have up to 20 upper-case and/or lower-case alphanumeric characters and $ signs.

  • 8/3/2019 Desire Gettingstarted

    18/30

    David Scuse September, 201118

    Interpreter Statements:

    Interpreter statements are translated line-by-line as they are executed. Interpreter statements maybe typed directly into the Desire system window by the user or they may be stored in a file forsubsequent execution. Interpreter statements that are typed directly into Desire are referred to as

    commands. Interpreter statements that are typed into a file are referred to as programmedstatements.

    The command-mode statement:

    > Lrate=0.5

    causes the variable Lrate to be assigned the value 0.5. This statement is executed (interpreted) assoon as the statement is typed. If the variable Lrate had been given a value earlier in the session,that value is replaced by this new value.

    Compiled Statements:

    Desire also contains statements that are compiled for quick execution. These statements have thesame syntax as interpreted statements but compiled statements can be entered only in a program.

    System Files:

    Desire uses several types of files; the file type is identified by the file extension. Files with anextension of .lstare ascii source files that include line numbers; files with an extension of .srcare ascii source files that do not include line numbers. The currently active source file isautomatically saved in syspic.lst.

    Screen Input/Output:

    The disptstatement is a compiled statement that can be used only in a dynamic segment. It isused to plot the values of one or more variables against the time variable, t.

    The write statement is an interpreter statement that is used to display the current value of one ormore variables. For example, the statement:

    write Weights | -- interpreter statement

    causes the current values of the Weights matrix to be displayed on the screen. This statement canbe used inside a program (in an interpreted statement) or outside of a program (in command

    mode). This statement is useful when testing built-in functions in command mode. For example,the following statement displays the result of evaluating the swtchfunction.

    write swtch(0.0)

    While the write statement can not be used in a dynamic segment, there is a correspondingstatement, the typestatement, that provides the same facility in dynamic segments.

  • 8/3/2019 Desire Gettingstarted

    19/30

    David Scuse September, 201119

    type X,Y,Z | -- dynamic statement

    File Input/Output:

    The interpreter write statement can also be used to write information to a disk file. For example,the interpreter statements:

    connect 'temp.dat' as output 4write #4,Lratedisconnect 4

    open a file for output, write the contents of the variable Lrateto the file, and then close the file.

    Drun and Go Statements

    Once a program has been run once, the programs dynamic statements can be executed againfrom where the previous execution left off by typing the drunstatement.

    If there are executable statements that follow the STOPstatement in the interpreted program, thesestatements can be executed by typing the gostatement.

    Interpreter Loops

    A loop may be added to the interpreter program using the for statement. For example, thefollowing statements:

    for count=1 to 10drunnext

    cause the drun statement to be executed 10 times (the screen is cleared at the beginning of eachdrun). To examine the screen before it is cleared, a STOPstatement can be added to the loop.

    for count=1 to 10drunSTOPnext

    The system stops when the STOP statement is encountered; however, the user can issue the gostatement to instruct the system to resume its processing.

    Display Statement

    The display statement is used to set various Graph window properties. By default, thebackground of the Graph window is black. It is usually a good idea to switch to a whitebackground (particularly if the graph window is to be printed).

    display N12 | -- use a black background and set initialcolour for graph to 12

    display N18 | -- use a black background and use whitefor all graphs

  • 8/3/2019 Desire Gettingstarted

    20/30

    David Scuse September, 201120

    display N-12 | -- use a white background and set initialcolour for graph to 12

    display N-18 | -- use a white background and use blackfor all graphs

    display C16 | -- set graph axes to blackdisplay R | -- generate thick dots on graphdisplay Q | -- generate thin dots on graph

    display W x,y | -- set position on screen of graph windowdisplay 2 | -- dont erase graph after drun (must be

    placed after drun statement)

    The display colour is used to identify each function that is graphed in the Graph window. In thelower portion of the graph window shown below, the Sig function is graphed using the firstcolour (the colour specified by x in the display Nx statement), the SigP function is graphedusing the next smaller colour (the colour x-1), the TanH function is graphed using the nextsmaller colour (the colour x-2), and so on. (This graph window is shown in its entirety in Figure18.)

    Programmer-Defined Functions

    The programmer may define functions at the beginning of the Desire program. For example, thefollowing function returns the minimum of two values:

    FUNCTION min(aaa,bbb)=aaa-lim(aaa-bbb)

    The following function returns a random integer between 1 and XX.

    FUNCTION Rand(XX)=round((((ran()+1.0)/2.0)*XX)+0.5)

    In the examples above, lim(x), ran(), and round(x) are Desire built-in functions. It isimportant that the parameters used in programmer-defined functions be variable names that areunique within the program (hence the choice of variables such as aaa and bbb).

    Labeled Dynamic Segments

    The first dynamic segment does not have a label; this segment is executed by invoking the drunstatement. Additional dynamic segments may also be defined; for example, the first dynamic

    segment could train a network; a second segment could test the ability of the network to recall aset of patterns. Labeled dynamic segments are invoked using the drun name statement. Alabeled dynamic segment is identified using a label name statement at the beginning of thesegment.

    CLEARN Statement

    The CLEARN statement is used during competitive learning.

  • 8/3/2019 Desire Gettingstarted

    21/30

    David Scuse September, 201121

    CLEARN Output = Weights(Input)Lrate,Crit

    where:Input is the input/pattern vectorWeights is the weight matrixLrate is the learning rate: if Lrate=0, there is no learning, just recallCrit is a flag: for Crit=0, a conscience mechanism is implemented;

    for Crit

  • 8/3/2019 Desire Gettingstarted

    22/30

    David Scuse September, 201122

    Desire Graph Window

    Graphing Functions

    The following graphprogram illustrates how Desire can be used to graph various functions. Inthe program, the sigmoid, sigmoid-prime (first derivative of the sigmoid function), tanh(hyperbolic tangent), and tanh-prime (first derivative of tanh) are graphed.

    ----------------------------------------------------- This program graphs a variety of functions (graph1.src)---------------------------------------------------display N-12 | display Rscale=1tmin=-4.0 | incr=0.001 | tmax=4.0t=tmin | TMAX=(tmax-tmin) | NN=(TMAX-tmin)/incr+1drunSTOP---------------------------------------------------DYNAMIC---------------------------------------------------

    Sig=sigmoid(t) | -- sigmoid functionSigP=Sig*(1-Sig) | -- first derivative of sigmoid functionTanh=tanh(t) | -- tanh functionTanhP=1-Tanh*Tanh | -- first derivative of tanh functiondispt Sig,SigP,Tanh,TanhP

    Figure 17: Graph Program

    Figure 18: Graph Program Output

  • 8/3/2019 Desire Gettingstarted

    23/30

    David Scuse September, 201123

    Offset Graphs

    At times, the graph window can become overloaded with functions that overlap, making itdifficult to determine which function is which (and it doesnt help that Desire only graphs the lastfunction if multiple function points occur at the same location on the graph). To make it easier to

    understand graphs, it is often helpful to offset one or more of the graphs.

    For example, the following program graphs 3 functions but the last function graphed erasesportions of the previous functions.

    ----------------------------------------------------- This program graphs the sigmoid function (graph2.src)---------------------------------------------------scale=1display N-6 | display Rtmin=-10.0 | incr=0.1 | tmax=10.0 | Shift=5t=tmin | TMAX=2*tmax | NN=(TMAX-t)/incr+1drunSTOP---------------------------------------------------DYNAMIC---------------------------------------------------sig1=sigmoid(t+Shift)sig2=sigmoid(t)sig3=sigmoid(t-Shift)dispt sig1,sig2,sig3

    Figure 19: Sigmoid Graph Program

    Figure 20: Sigmoid Graph Program

  • 8/3/2019 Desire Gettingstarted

    24/30

    David Scuse September, 201124

    The following program graphs the same functions but offsets two of the functions by a portion ofthe value ofscale. As a result, the graphs become much easier to read (as long as you rememberthat they are offset).

    ----------------------------------------------------- This program graphs the sigmoid function

    -- but offsets two of the graphs (graph3.src)---------------------------------------------------scale=1display N-12 | display Rtmin=-10.0 | incr=0.1 | tmax=10.0 | Shift=5t=tmin | TMAX=2*tmax | NN=(TMAX-t)/incr+1drunSTOP---------------------------------------------------DYNAMIC---------------------------------------------------sig1=sigmoid(t+Shift)sig2=sigmoid(t)-scale/2sig3=sigmoid(t-Shift)-scaledispt sig1,sig2,sig3

    Figure 21: Sigmoid Offset Graph Program

    Figure 22: Sigmoid Offset Graph Program Output

  • 8/3/2019 Desire Gettingstarted

    25/30

    David Scuse September, 201125

    Reducing the Learning Rate

    The following program illustrates how the learning rate can be gradually reduced. The top partof the graph displays the learning rate as it is reduced from its initial value of L0 (0.75) to itsfinal value of Ln (0.01). The function 1-sigmoid() (shown in the bottom half of the graph)

    defines the transition from the initial learning rate to the final learning rate. Other functionscould be used to define the transition (such as a simple linear function) but the sigmoid functionprovides more time at both extremes than linear functions. Changing the parameters of thesigmoid function (S0 and Sn) would change the rate of transition of the learning rate.

    ------------------------------------------------------------- Reduce the learning rate (Lrate.src)-----------------------------------------------------------display R | display N-13 | display W 362,0scale=1Nepochs=1000 | Npat=1LRate0=0.75 | LRateN=0.01S0=-6 | SN=6 | SDelta=(SN-S0)/(Nepochs*Npat) | S=S0t=1 | NN=Nepochs | TMAX=NN-1drun

    STOP-----------------------------------------------------------DYNAMIC-----------------------------------------------------------Sig=1-sigmoid(S)LRate=(LRate0-LRateN)*Sig+LRateNSig= Sig-scaleS=S+SDeltadispt LRate,Sig

    Figure 23: Reducing the Learning Rate Program

    Figure 24: Reducing the Learning Rate Program Output

  • 8/3/2019 Desire Gettingstarted

    26/30

    David Scuse September, 201126

    Clipping a Function

    The following program illustrates how a function may be clipped to ensure that it fits withinthe specified value ofscale. (This prevents the situation where a function may be outside of thebounds ofscale and thus not appear anywhere in the Graph window.)

    ------------------------------------------------------------- Clip a function so that it fits within the graph (clipping.src)-----------------------------------------------------------FUNCTION min(aaa,bbb)=aaa-lim(aaa-bbb)FUNCTION max(aaaa,bbbb)=aaaa+lim(bbbb-aaaa)FUNCTION clipMax(X1)=min(X1,scale)FUNCTION clipMin(Y1)=max(Y1,-scale)FUNCTION clip(Z1)=clipMin(clipMax(Z1))-----------------------------------------------------------display N-12 | display Rscale=1tmin=-2.0 | incr=0.001 | tmax=2.0t=tmin | TMAX=(tmax-tmin) | NN=(TMAX-tmin)/incr+1drunSTOP---------------------------------------------------DYNAMIC

    ---------------------------------------------------T=1.2*tanh(t)clipT=clip(T)dispt clipT

    Figure 25: Clipping a Function Program

    Figure 26: Clipping a Function Program Output

  • 8/3/2019 Desire Gettingstarted

    27/30

    David Scuse September, 201127

    Graphing the Sigmoid Function with Temperatures

    ----------------------------------------------------- This program graphs the sigmoid function-- for several temperatures (graph4.src)---------------------------------------------------display N-13 | scale=1

    tmin=-6.0 | incr=0.01 | tmax=6.0T1=4.0 | T2=0.25t=tmin | TMAX=2*tmax | NN=(TMAX-t)/incr+1drunSTOP---------------------------------------------------DYNAMIC---------------------------------------------------Sig=sigmoid(t)Sig4=sigmoid(t/T1)Sig14=sigmoid(t/T2)dispt Sig,Sig4,Sig14

    Figure 27: Sigmoid with Temperatures Program

    Figure 28: Sigmoid with Temperatures Program Output

  • 8/3/2019 Desire Gettingstarted

    28/30

    David Scuse September, 201128

    Graphing Gaussian Bumps

    ----------------------------------------------------- This program graphs Gaussian bumps (graph5.src)---------------------------------------------------display N-13 | display Rscale=1

    tmin=-4.0 | incr=0.01 | tmax=4.0s1=0.5 | s2=1.0 | s3=2.0t=tmin | TMAX=2*tmax | NN=(TMAX-t)/incr+1x=0.25drunSTOP---------------------------------------------------DYNAMIC---------------------------------------------------g05=exp(-(t*t)/(2*s1*s1))g10=exp(-(t*t)/(2*s2*s2))g20=exp(-(t*t)/(2*s3*s3))dispt g05,g10,g20

    Figure 29: Gaussian Bumps Program

    Figure 30: Gaussian Bumps Program Output

  • 8/3/2019 Desire Gettingstarted

    29/30

    David Scuse September, 201129

    Early Termination of a Dynamic Segment

    The Desire system normally executes each dynamic segment until completion. However, thereare times when it is useful to be able to stop a dynamic segment before all NN repetitions arecomplete. (For example, when the tss has reached an acceptable value.) The termstatement can

    be used in a dynamic segment for this purpose. As soon as the value of the expression in the termstatement becomes positive, processing of the dynamic segment is terminated.

    ----------------------------------------------------- This program terminates a dynamic segment early (term1.src)---------------------------------------------------display N-12 | display Rscale=1tmin=-4.0 | incr=0.01 | tmax=4.0t=tmin | TMAX=(tmax-tmin) | NN=TMAX/incr+1stop=2drunSTOP---------------------------------------------------DYNAMIC---------------------------------------------------term (t-stop)+0.0000001 | -- stop when this expression becomes positiveSig=sigmoid(t)

    dispt Sig-- type t

    Figure 31: Early Termination of a Dynamic Segment

    Figure 32: Early Termination of a Dynamic Segment Program Output

  • 8/3/2019 Desire Gettingstarted

    30/30

    id S S b 2011

    Differential Equations

    The DIFF program graphs the sigmoid and sigmoid-prime functions. It also illustrates the abilityof Desire to solve differential equations (numerically). A differential equation is introducedusing the keyword d/dt. (In addition to defining scalar differential equations, differential

    equations may also be defined using vectors and matrices.) In this example, the first derivativeof Y is defined as Sig*(1-Sig); thus, Y is the sigmoid function (which Desire calculatescorrectly). Note that the sigmoid and sigmoid derivative have been shifted so that they appear inthe bottom half of the graph.

    ------------------------------------------------------------------- Working with a scalar differential equation (diff.src)-----------------------------------------------------------------display N-12 | display C16 | display Rscale=1tmin=-8.0 | incr=0.001 | tmax=8.0t=tmin | TMAX=(tmax-tmin) | NN=(TMAX-tmin)/incr+1drunSTOP-----------------------------------------------------------------

    DYNAMIC-----------------------------------------------------------------Sig=sigmoid(t) | -- sigmoid functionSigP=Sig*(1-Sig) | -- first derivative of sigmoid functiond/dt Y = SigP | -- compute Y such that Y's first derivative is SigPerr = abs(Y) - abs(Sig)Sig=Sig-scaleSigP=SigP-scaledispt Sig,SigP,Y,err

    Figure 33: Differential Equation Program

    Figure 34: Differential Equation Program Output