8VLQJ 1HXUDO 1HWZRUN 0HWKRG FRPSXWHV 4XDGUDWLF 2SWLPL]DWLRQ 3UREOHPV

$, :X DQG 3�.�6� 7DP

'HSW� RI (OHFWURQLF DQG ,QIRUPDWLRQ (QJLQHHULQJ7KH +RQJ .RQJ 3RO\WHFKQLF 8QLYHUVLW\

(�PDLO� DLZX#HQ�SRO\X�HGX�KN

$EVWUDFW

$FFRUGLQJ WR WKH EDVLF RSWLPL]DWLRQ SULQFLSOH RI DUWLILFLDO

QHXUDO QHWZRUNV� D QRYHO NLQG RI QHXUDO QHWZRUN PRGHO IRU

VROYLQJ WKH TXDGUDWLF SURJUDPPLQJ SUREOHP LV SUHVHQWHG�

7KH PHWKRGRORJ\ LV EDVHG RQ WKH /DJUDQJH PXOWLSOLHU

WKHRU\ LQ RSWLPL]DWLRQ DQG VHHNV WR SURYLGH VROXWLRQVVDWLVI\LQJ WKH QHFHVVDU\ FRQGLWLRQV RI RSWLPDOLW\� 7KH

HTXLOLEULXP SRLQW RI WKH QHWZRUN VDWLVILHV WKH .XKQ�7XFNHU

FRQGLWLRQ IRU WKH SUREOHP� 7KH VWDELOLW\ DQG FRQYHUJHQF\

RI WKH QHXUDO QHWZRUN LV LQYHVWLJDWHG DQG WKH VWUDWHJ\ RI

WKH QHXUDO RSWLPL]DWLRQ LV GLVFXVVHG� 7KH IHDVLELOLW\ RI WKH

QHXUDO QHWZRUN PHWKRG LV YHULILHG ZLWK WKH FRPSXWDWLRQ

H[DPSOHV� 5HVXOWV RI WKH VLPXODWLRQ RI WKH QHXUDO QHWZRUN

WR VROYH RSWLPXP SUREOHPV DUH SUHVHQWHG WR LOOXVWUDWH WKH

FRPSXWDWLRQDO SRZHU RI WKH QHXUDO QHWZRUN PHWKRG�

�� ,QWURGXFWLRQ

+RSILHOG SURSRVHG DQ DQDORJ QHXUDO QHWZRUN >�@� ZKLFKFDQ VHHN IRU D PLQLPXP SRLQW RI D TXDGUDWLF HQHUJ\

IXQFWLRQ� 6LQFH WKHQ PDQ\ UHVHDUFKHUV ZHUH DWWUDFWHG WR

VWXG\ WKH DELOLW\ RI WKH RSWLPL]DWLRQ FRPSXWDWLRQ EDVHG RQ

WKH QHXUDO QHWZRUN PHWKRGRORJ\ DQG YDULRXV RSWLPXP

QHWZRUN PRGHOV DQG QHXUDO FRPSXWDWLRQDO WHFKQLTXHV KDYH

EHHQ UHSRUWHG >�@� >�@� >�@� >�@� 0RVW RI WKHP� KRZHYHU� DUH

RQO\ DEOH WR VROYH VRPH RSWLPXP SUREOHPV ZLWKRXW DQ\

FRQVWUDLQWV RU ZLWK VRPH VLPSOH OLQHDU FRQVWUDLQWV >�@� >�@�

>�@� 6� :� =KDQJ HW DO� SURSRVHG WKH /DJUDQJH

SURJUDPPLQJ QHXUDO QHWZRUNV �/311� EDVHG RQ WKH

/DJUDQJH PXOWLSOLHU WKHRU\ IRU FRQVWUDLQHG SURJUDPPLQJDQG E\ PHDQV RI VWUXFWXULQJ WKH /DJUDQJH IXQFWLRQV RI

RULJLQDO SUREOHPV >�@� 7KH /311 QHWZRUN PRGHO VROYHV

VXFFHVVIXOO\ WKH SUREOHPV RI FRQVWUDLQHG RSWLPL]DWLRQ LQ D

FHUWDLQ H[WHQW� %XW LW LV RQO\ OLPLWHG WR GHGXFH WKH

GLIIHUHQWLDO HTXDWLRQV DQG WKH HIILFLHQF\ RI SUDFWLFDO

DSSOLFDWLRQ LV QHJOHFWHG�

,Q WKH SDSHU� ZH DQDO\]H DQG V\QWKHVL]H VRPH RSWLPL]LQJ

QHWZRUNV DQG DFFRUGLQJ WR WKH EDVLF RSWLPL]DWLRQ SULQFLSOH

RI DUWLILFLDO QHXUDO QHWZRUNV� SUHVHQW WKH QHXUDO QHWZRUN

PRGHO DQG PHWKRG RI VROYLQJ WKH TXDGUDWLF SURJUDPPLQJ

SUREOHP EDVHG RQ WKH /DJUDQJH IXQFWLRQ� 7KH VWDELOLW\ DQG

FRQYHUJHQF\ RI WKH QHXUDO QHWZRUN LV LQYHVWLJDWHG DQG WKH

VWUDWHJ\ RI WKH QHXUDO RSWLPL]DWLRQ LV GLVFXVVHG� 7KH

PHWKRGRORJ\ LV EDVHG RQ WKH /DJUDQJH PXOWLSOLHU WKHRU\ LQ

RSWLPL]DWLRQ DQG VHHNV WR SURYLGH VROXWLRQV VDWLVI\LQJ WKH

QHFHVVDU\ FRQGLWLRQV RI RSWLPDOLW\�

7KH SDSHU LV RUJDQL]HG DV IROORZV� ,Q 6HFWLRQ ,, ZH

FRQVWUXFW D PRGHO RI WKH QHXUDO RSWLPL]DWLRQ QHWZRUN EDVHG

RQ WKH /DJUDQJH IXQFWLRQ� ,Q 6HFWLRQ ,,, ZH GLVFXVV WKH

VWDELOLW\ DQG FRQYHUJHQF\ RI WKH QHXUDO QHWZRUN DQG D

/\DSXQRY IXQFWLRQ LV VHW XS LQ WKH SURFHGXUH� 7KH QHXUDO

RSWLPL]DWLRQ VWUDWHJLHV DUH SURSRVHG LQ 6HFWLRQ ,9� ,Q6HFWLRQ 9 WKH UHVXOWV RI VRPH VLPXODWH SUREOHP H[DPSOHV

DUH SUHVHQWHG WR LOOXVWUDWH WKH FRPSXWDWLRQDO FDSDFLW\ RI WKH

QHXUDO QHWZRUN PHWKRG RI TXDGUDWLF SURJUDPPLQJ�

�� 0RGHOLQJ RI RSWLPL]DWLRQ QHXUDO QHWZRUN

1HXUDO QHWZRUN LV D ODUJH GLPHQVLRQDO QRQOLQHDU G\QDPLFV\VWHP WKDW LV FRPSRVHG RI QHXURQV� )URP WKH YLHZ SRLQW

RI G\QDPLFDO V\VWHP� WKH ILQDO EHKDYLRUV RI WKH V\VWHP DUH

IXOO\ GHWHUPLQHG E\ LWV DWWUDFWLRQ SRLQWV� LI WKH\ H[LVW� )RU D

VWHDG\ V\VWHP� LI D FHUWDLQ LQSXW LV JLYHQ� WKH V\VWHP ZLOO

UHDFK D VWHDG\ DWWUDFWLRQ SRLQW WKURXJK WKH V\VWHP

G\QDPLFV� ,I WKH ODVW VWHDG\ DWWUDFWLRQ SRLQW RI WKH QHWZRUN

V\VWHP LV VHHQ DV WKH RSWLPXP VROXWLRQ RI VRPH SUDFWLFDO

SUREOHPV� WKH HYROXWLRQ SURFHVV WKURXJK ZKLFK WKH QHXUDO

QHWZRUN UHDFK WKH VWHDG\ VWDWH IURP DQ\ LQLWLDO VWDWH LV MXVW

D SURFHVV RI VHHNLQJ IRU WKH RSWLPL]DWLRQ YDOXH RI WKH

REMHFWLYH IXQFWLRQ ZLWKLQ WKH DFWLQJ GRPDLQ� 7KHUHIRUH WKH

NH\ RI GHVLJQLQJ DQ RSWLPL]DWLRQ QHWZRUN LV KRZ VHW WKHFRUUHVSRQGHQW UHODWLRQVKLSV EHWZHHQ WKH SUREOHP DQG WKH

VWHDG\ DWWUDFWLRQ SRLQW RI WKH QHWZRUN� &RQVLGHULQJ WKH

IROORZLQJ TXDGUDWLF SURJUDPPLQJ SUREOHP ZLWK HTXDOLW\

FRQVWUDLQWV�

0LQLPL]H ;&*;;;I77

� �

���

6XEMHFW WR E$;

ZKHUH Q7

Q 5[[[; � ��������� ���

Q7

Q 5FFF& � ��������� ��

DQGP7

P 5EEEE � �����������

� $ LV D PuQ PDWUL[� * LV D

���

QuQ UHDO V\PPHWU\ SRVLWLYH GHILQLWH PDWUL[� )RU WKHSUREOHP ��� IRUPXODH� ZH GHILQHG WKH /DJUDQJH IXQFWLRQ

DV

������

� E$;;&*;;;/777

��� OO

ZKHUHP7

P 5� �����������

OOOO LV WKH /DJUDQJH

PXOWLSOLHU�

$FFRUGLQJ WKH FODVVLFDO OLPLW WKHRU\� WKH QHFHVVDU\

FRQGLWLRQ RI ZKLFK

; LV WKH RSWLPXP VROXWLRQV RI

HTXDWLRQ ��� LV WKDW WKHUH H[LVWV VXFK

O ZKLFK PDNHV

; WR VDWLVI\ WKH IROORZLQJ UHODWLRQ

����

�� � OO7

[ $&*;;/

����

� � E$;;/ OO

,Q RUGHU WR LPSURYH WKH FRQYHUJHQF\ RI WKH RSWLPL]DWLRQ

DOJRULWKP� ZH PD\ DGG D FRPSHQVDWLRQ WHUP�

��E$;

N� �

��!N WR WKH /DJUDQJH IXQFWLRQ ��� DQG XVH D YDULDEOH \

LQVWHDG RI WKH /DJUDQJH PXOWLSOLHU O � 6R� IRU WKH

TXDGUDWLF SURJUDPPLQJ SUREOHP RI HTXDWLRQ ���� WKHIROORZLQJ EDVHG RQ G\QDPLF QHXUDO QHWZRUN V\VWHP FDQ EH

LPSOHPHQWHG

PMYJ\

QLXJ[

E$;GW

GY

E$;N$\$&*;GW

GX

MM

LL

77

����������

�����������

��

�@�>

��

�����

ZKHUHQ

5; � LV WKH RXWSXW YDULDEOH RI WKH QHWZRUNFRUUHVSRQGLQJ WR WKH RSWLPL]DWLRQ YDULDEOH� E DQG & DUH WKH

LQSXW YDULDEOHV RI WKH QHWZRUN DQG \ LV D YDULDEOH

FRUUHVSRQGLQJ WR WKH /DJUDQJH PXOWLSOLHU O � $SSO\LQJ WKH

WHFKQLTXHV RI V\VWHP UHDOL]DWLRQ >��@� EORFN GLDJUDP

UHSUHVHQWDWLRQ RI D FLUFXLW IUDPH JUDSK RI WKH QHXUDOQHWZRUN FRUUHVSRQGLQJ WR HTXDWLRQ ��� LV VKRZQ LQ )LJXUH

��� 2EYLRXVO\� WKH FLUFXLW VWUXFWXUH RI WKH QHXUDO QHWZRUN LV

VLPSOH� 7KH QHWZRUN QHHGV RQO\ �Q�P� QHXURQV� 7KHUH DUH

Q��QP FRQQHFWLYH ZHLJKWV WR EH PRGLILHG LQ WKH QHWZRUN�

���J LQ HTXDWLRQ ��� LV D QHXURQ DFWLYH IXQFWLRQ ZKLFK PD\

EH D 6LJPRLG IXQFWLRQXH

XJ�

�

�

��� RU D K\SHU�OLQHDU

IXQFWLRQ �����

� XX

HHXJ�

� �

�� 6WDELOLW\ DQG FRQYHUJHQF\ RI WKH QHXUDO

QHWZRUN

7KH QHXUDO QHWZRUN GHVFULEHG E\ HTXDWLRQ ��� LV JOREDOO\

VWDEOH DQG FRQYHUJHV WR WKH RSWLPXP VROXWLRQ RI SUREOHP

��� HTXDWLRQ� 7KH HQHUJ\ IXQFWLRQ RI G\QDPLF V\VWHP ���

HTXDWLRQ LV GHILQHG DV

�

���

� ����� E$;E$;\;&*;;\[( N777�����

6R WKH GHULYDWLRQ UHODWLRQ RI HQHUJ\ IXQFWLRQ (�[� \� ZLWK

WLPH W LV GHVFULEHG DV

Figure 1. Block diagram representation of a circuit frame graph of the neural network

6 6

N

*

$7

³ ³6J��� J���E

&

X ;

$

QuP

Pu�

PuQ

PuP

Qu�

QuQ

Qu�

�

��

�

�

Y \

���

���

���

���

> @ > @

> @ > @

�

���

���������

������

����

�

�

��

��

d

��

��������

�������

��w

w���

w

w

��w

w���

w

w

�w

w��

w

w

¦¦

¦¦

¦¦

\[(

*\[(\[(*\[(

GW

GY*\[(

GW

GX*\[(

GW

GYYJ

\

(

GW

GXXJ

[

(

GW

GY

GY

G\

\

(

GW

GX

GX

G[

[

(

GW

G\

\

(

GW

G[

[

(

GW

G(

\

Y7

\[X7

[

Y7

\X7

[

M

M

P

M M

LL

Q

L L

MP

M M

M

M

L

L

LQ

L L

P

M

M

M

LQ

L L

ZKHUH ������� \[(\[(\[

�� UHVSHFWLYHO\ LV WKH SDUWLDO

GHULYDWLYH RI WKH HQHUJ\ IXQFWLRQ (�[� \� ZLWK UHVSHFW WR [

DQG \� $QG

�������� \[(GW

GY\[(

GW

GX\[ �� ��

� ���������������

�

�

QXXJXJXJGLDJ*

��������������

QLXJL

!

� � ����������������������������

�

�

PMYJYJYJYJGLDJ* MPY !

7KHUHIRUH WKH HQHUJ\ IXQFWLRQ (�[� \� RI WKH V\VWHP LV

UHGXFHG ZLWK LQFUHDVH LQ WLPH W XQWLO WKH VWHDG\ VWDWH RI WKH

V\VWHP LV UHDFKHG� 6R (�[� \� GHVFULEHG LQ HTXDWLRQ ��� LV

WKH /\DSXQRY IXQFWLRQ RI WKH V\VWHP DQG WKH G\QDPLF

V\VWHP LV JOREDOO\ VWDEOH� $FFRUGLQJ WR HTXDWLRQ ���� ZKHQ

DQG RQO\ ZKHQ �

GW

GXDQG �

GW

GY� ZH KDYH �

GW

G(�

6R� ZKHQ WKH G\QDPLF V\VWHP UHDFKHV LQ WKH VWHDG\ VWDWH�

WKH HQHUJ\ IXQFWLRQ (�[� \� LV QR ORQJHU FKDQJLQJ DQG (�[�

\� UHDFKHV WKH PLQLPXP�

7KH TXDGUDWLF SURJUDPPLQJ SUREOHP RI HTXDWLRQ ��� LV DFRQYH[ RSWLPL]DWLRQ SUREOHP� 7KH REMHFWLYH IXQFWLRQ

;&*;;;I77

� �

��� LV D ILUVW�RUGHU GLIIHUHQWLDEOH

IXQFWLRQ LQ WKH RSHQ FRQYH[ VHWQ

5; � � )RU HYHU\Q

5; � � +HVVLDQ PDWUL[ +�;� LV SRVLWLYH GHILQLWH DQG I�;�

LV D VWULFW FRQYH[ IXQFWLRQ� 7KHUHIRUH HYHU\ ORFDO VROXWLRQ

;

RI SUREOHP ��� HTXDWLRQ LV D JOREDO RSWLPXP VROXWLRQ

DQG WKH VHW 6 RI WKH JOREDO RSWLPXP VROXWLRQ LV WKH FRQYH[

VHW�

$FFRUGLQJ WR WKH .XKQ�7XFNHU RSWLPL]DWLRQ WKHRUHP� WKH

VXIILFLHQW DQG QHFHVVDU\ FRQGLWLRQ LV WKDW WKHUH H[LVWQ

5; � DQGP

5�O ZKLFK VDWLVI\ WKH IROORZLQJ HTXDWLRQ

E$;

$&*;7

�� �O

VR WKH VXIILFLHQW DQG QHFHVVDU\ FRQGLWLRQ RI ZKLFK ; LV WKH

JOREDO RSWLPXP VROXWLRQ RI SUREOHP ��� HTXDWLRQ LV WKDW

HTXDWLRQ ��� EH HVWDEOLVKHG� &RPSDULQJ HTXDWLRQ ��� ZLWK

HTXDWLRQ ���� ZKHQ � GW

GXDQG �

GW

GY� LQ RWKHU ZRUG

DIWHU WKH V\VWHP UHDFKHV VWHDG\ VWDWH� HTXDWLRQ ��� LV

HVWDEOLVKHG� 7KHQ QL[XJ[LLL

������������

DQG

PMYJ\ MLL ������������

O VDWLVI\ WKDW

;; LV WKH

VXIILFLHQW DQG QHFHVVDU\ FRQGLWLRQ RI WKH JOREDO RSWLPXP

VROXWLRQ RI SUREOHP ��� HTXDWLRQ� 7KHUHIRUH D VWHDG\

HTXLOLEULXP SRLQW RI WKH G\QDPLF V\VWHP� � GW

GXDQG

� GW

GXFRUUHVSRQGV WR WKH RSWLPXP VROXWLRQ

�������������

QLXJ;L

1R PDWWHU KRZ WKH LQLWLDO

VWDWH RI WKH G\QDPLF V\VWHP LV� WKH V\VWHP ZLOO WHQG WR WKH

HTXLOLEULXP VWDWH� ,Q RWKHU ZRUGV� QR PDWWHU ZKDW WKH LQLWLDO

YDOXHV RI WKH RSWLPL]DWLRQ YDULDEOHV DUH� WKH QHXUDO

QHWZRUN ZLOO PDNH WKH YDULDEOHV WR FRQYHUJH JOREDOO\ WR

WKH RSWLPXP VROXWLRQ HYHQWXDOO\�

�� 2SWLPL]DWLRQ VWUDWHJ\ RI WKH QHXUDO

QHWZRUN

,Q RUGHU WR VROYH SUREOHP ��� HTXDWLRQ� WKH REMHFWLYH

IXQFWLRQ DQG FRQVWUDLQW IXQFWLRQ LV PDSSHG DV D FORVH�ORRS

V\VWHP� :KHQ WKH FRQVWUDLQW FRQGLWLRQ LV QRW VDWLVILHG� WKH

YDULDEOH ZKLFK GRHVQ©W VDWLVI\ WKH FRQVWUDLQW FRQGLWLRQ PD\EH IHG EDFN WR DGMXVW DOO QHXURQ VWDWHV RI WKH QHXUDO

QHWZRUN DJDLQ� :LWK WKH HYROXWLRQ PRGHO� WKH HQHUJ\

IXQFWLRQ RI WKH QHXUDO QHWZRUN LV DOZD\V UHGXFHG XQWLO WKH

PLQLPXP VWDWH RI HQHUJ\� :KHQ WKH HQHUJ\ UHDFKHV WKH

PLQLPXP� WKH QHXURQ VWDWH MXVW LV WKH VROXWLRQ RI WKH

RULJLQDO SUREOHP�

(�[� \� DV WKH HQHUJ\ IXQFWLRQ RI WKH QHXUDO QHWZRUN LQ

HTXDWLRQ ��� KDV D WRWDO RI P�Q QHXURQV� ZKHUH Q LV WKH

GLPHQVLRQ RI WKH XQNQRZQ YHFWRU ; DQG P LV WKH

GLPHQVLRQ RI WKH /DJUDQJH PXOWLSOLHU� 7KH QHXURQ DFWLYH

IXQFWLRQ J��� DQG WKH IXQFWLRQ UHODWLRQ EHWZHHQ LWK QHXURQDQG MWK QHXURQ LV UHVSHFWLYHO\ GHVFULEHG LQ HTXDWLRQ ����:LWK WKH HYROXWLRQ RI WKH QHXUDO QHWZRUN�

QL[W[LL

������������

o � LW LV XQDYRLGDEOH WKDW D YLEUDWLRQ

WDNHV SODFH LQ WKH HYROXWLRQ SURFHVV� :H PD\ DGMXVW

VXLWDEO\ WKH /DJUDQJH PXOWLSOLHU WR UHGXFH WKH YLEUDWLRQ

WLPH RI HYROXWLRQ DQG WR LPSURYH WKH FRQYHUJHQFH VSHHG�

,I * � LQ WKH SUREOHP ��� HTXDWLRQ� WKH SUREOHP EHORQJV

WR D OLQHDU RSWLPL]DWLRQ SUREOHP VXFK WKDW J��� QHHGV WR EH

RQO\ D OLQHDU IXQFWLRQ� ,I * z � LQ SUREOHP ��� HTXDWLRQ� LW

LV VXLWDEOH WKDW J��� LV VHOHFWHG IURPX

H

XJ�

�

�

��� RU

���

���

�����

� XX

HHXJ�

� � ,W LV SRVVLEOH WR LPSURYH WKH VSHHG RI

WKH RSWLPL]DWLRQ FRPSXWDWLRQ E\ VHOHFWLQJ D VXLWDEOH

QHXURQ DFWLYH IXQFWLRQ DFFRUGLQJ WR WKH QRQOLQHDU OHYHO RI

WKH GLVFXVVHG SUREOHP� ,Q RWKHU ZRUGV� WKH HYROXWLRQ VSHHG

RI WKH QHXUDO QHWZRUN FDQ DOVR EH UDLVHG� )RU WKH RXWSXW

IXQFWLRQ �����

� XX

HHXJ�

� � LWV 7DLORU SRO\QRPLDO LQ X

LV

J�X� �X�X

� ��X�X

����X�X

����� ,W FDQ EH DSSUR[LPDWHG

DV J�X� X LQ D GRPDLQ QHDU X X

� 7KDW LV� ZKHQ WKH

V\VWHP DSSURDFKHV WKH HTXLOLEULXP SRLQW� WKH FRPSXWDWLRQ

VSHHG EHFRPH ORZHU� +RZHYHU� ZKHQ WKH LQLWLDO VWDWH LV IDU

IURP WKH HTXLOLEULXP SRLQW� WKH HIIHFW RI WKH KLJK�RUGHU

WHUP LV REYLRXV� 7KHUHIRUH WKH RXWSXW IXQFWLRQ

�����

� XX

HHXJ�

� KDV DQ DGDSWLYH DELOLW\ WR D FHUWDLQ

H[WHQW� ,WV IHDWXUH LV YHU\ VLPLODU WR WKH DFWXDO QHXURQ� 7KH

IXQFWLRQV KDYLQJ VLPLODU D IHDWXUH LQFOXGHV WKH 6�VKDSHG

IXQFWLRQV� )RU WKH QHXUDO QHWZRUN PRGHO ��� ZH KDYH

LQYHVWLJDWHG WKH HIIHFWV RI VRPH W\SHV RI J��� RQ WKH

FRPSXWDWLRQ WLPH WR VROYH WKH TXDGUDWLF SURJUDPPLQJ

SUREOHP RI ([DPSOH �� 7KH PHDVXUH UHVXOWV LQ WHUP RI WKH

QXPEHU RI WUDLQLQJ HSRFKV FRUUHVSRQGLQJ WR IRXU GLIIHUHQW

QHXURQ DFWLYH IXQFWLRQ J��� XQGHU WKH VDPH LQLWLDO FRQGLWLRQDQG FRQYHUJHQFH SUHFLVH DUH VKRZQ LQ 7DEOH ��

Table 1 � Epochs of training of different neuron activefunction for quadratic programming

�� �� ��� ��

$ FRPSHQVDWLRQ WHUP�

��E$;

N� DQG ��!N

FRUUHVSRQGLQJ WR WKH UHPDLQGHU GLIIHUHQFH RI WKH HTXDOLW\

FRQVWUDLQW FRQGLWLRQ LV DGGHG WR WKH /DJUDQJH IXQFWLRQ

VXFK WKDW WKH FRQYHUJHQFH RI WKH RSWLPL]DWLRQ DOJRULWKP

JHWV LPSURYHG� :KHQ * LV VHPL�SRVLWLYH GHILQLWH RU

XQOLPLWHG GHILQLWH� WKH DOJRULWKP FRQVWUXFWV WKHFRQYHUJHQFH PHWKRG RI WKH JOREDO GHYHORSLQJ� 7KH ILUVW

DQG VHFRQG SDUWLDO GHULYDWLYH RI ( �[� \� ZLWK UHVSHFW WR [ LV

UHVSHFWLYHO\ JLYHQ E\

$N$*\[(

E$;N$\$&*;\[(

7

[[

77

[

� �

���� �

���

�����

�

ZKHUH ������

[+\[([[

� LV D +HVVLDQ PDWUL[� ,W FDQ EH

VHHQ WKDW LI * LV SRVLWLYH GHILQLWH� WKHQ + LV DOO SRVLWLYH

GHILQLWH DQG LI * LV VHPL�SRVLWLYH GHILQLWH RU XQOLPLWHG

GHILQLWH� WKHQ D VXIILFLHQWO\ ODUJH FRPSHQVDWLRQ FRHIILFLHQWN!� FDQ PDNH + SRVLWLYH GHILQLWH� 7KH RSWLPL]DWLRQ

DOJRULWKP RI WKH QHXUDO QHWZRUN FDQ REWDLQ WKH JOREDO

RSWLPXP VROXWLRQ XQGHU DQ\ FRQGLWLRQ QR PDWWHU ZKHWKHU

* LV SRVLWLYH GHILQLWH� VHPL�SRVLWLYH GHILQLWH RU XQOLPLWHG

GHILQLWH�

,I WKH TXDGUDWLF SURJUDPPLQJ SUREOHP LQFOXGHV DQ

LQHTXDOLW\ FRQVWUDLQW FRQGLWLRQ� D ORRVH YDULDEOH PD\ EH

LQWURGXFHG WR FKDQJH WKH LQHTXDOLW\ FRQVWUDLQW FRQGLWLRQ

LQWR WKH HTXDOLW\ FRQVWUDLQW FRQGLWLRQ� 7KHQ WKH

RSWLPL]DWLRQ FRPSXWDWLRQ FDQ EH LPSOHPHQWHG ZLWK WKH

VDPH PRGHO DV VWDWHG DERYH�

�� 6LPXODWLRQ H[DPSOH

([DPSOH �

8VLQJ WKH QHXUDO QHWZRUN FRUUHVSRQGLQJ HTXDWLRQ ���

RSWLPL]HV WKH IROORZLQJ HTXDOLW\ FRQVWUDLQW SUREOHPV�

PLQ @�����>����

��

�

��

�

���

� [[[[[[;I �����

V� W��

���

��

���

�

���

[[

[[[

6ROYLQJ� 7R KDYH VHOHFWHG � RSWLPL]DWLRQ YDULDEOH QHXURQV

DQG �FRQVWUDLQ YDULDEOH QHXURQV DQG WKH FRPSHQVDWH

FRHIILFLHQW VR WKDW WKH FRUUHVSRQGLQJ G\QDPLF HTXDWLRQ LV

JLYHQ DV

¸̧¹

·¨̈©

§�

¸¸¸

¹

·

¨¨¨

©

§

¸̧¹

·¨̈©

§

��

��

¸̧¹

·¨̈©

§

¸¸¸

¹

·

¨¨¨

©

§

��¸¸¸

¹

·

¨¨¨

©

§

�¸¸¸

¹

·

¨¨¨

©

§

¸¸¸

¹

·

¨¨¨

©

§

�

�����

�

�

���

���

��

�

�

�

�

�

�

�

�

�

���

���

���

�@�>

�

�

�

�

�

�

�

�

[

[

[

E$;GW

GY

\

\

[

[

[

E$;$\$&*;GW

GX 77

VHOHFWLQJ WKH QHXURQ DFWLYH IXQFWLRQ �����

� XX

HHXJ�

�

DQG �����

� YY

HHYJ�

� � XVLQJ WKH LPSURYHG (XOHULDQ

IRUPXOD PD\ REWDLQ WKH HTXLOLEULXP VWDWH RI WKH QHXUDO

QHWZRUN7

X ���������������� �

7Y ������������ � �

WKH FRUUHVSRQGLQJ RSWLPXP VROXWLRQ7

[ ������������� �

7\ ��������� � DQG WKH REMHFWLYH IXQFWLRQ ��� �� [I �

7KHVH UHVXOWV DUH WRWDOO\ LGHQWLFDO ZLWK WKH WKHRUHWLFDO

VROXWLRQV�

���

X�

X� �H

X� H

�X���XNeuron Active

Function g(.)

Training Epochs

([DPSOH �

7R VHHN WKH TXDGUDWLF SURJUDPPLQJ SUREOHP ZKLFK KDV D

VROH RSWLPXP VROXWLRQ

PLQ �����

�

�

�

�

��

� [[[;I ��

V� W��

��

���

���

� ��

��

[[[

[[[

,W FDQ EH SURRIHG WKDW WKH SUREOHP KDV D VROH VROXWLRQ� 7KH

WKHRUHWLFDO VROXWLRQ LV77

[[[[ ��������������������������������

�

�

�� �

6ROYLQJ� 6LPLODU WR ([DPSOH �� WKH G\QDPLF HTXDWLRQV RI

HDFK QHXURQ KDV WKH IROORZLQJ IRUP

���

����

�����

�������

����

���

�

���

�

����

�

�����

�

����

�

����

����

������

������

�����

[[[

GW

GY

[[[GW

GY

\\[[GW

GX

\\[[[GW

GX

\\[[GW

GX

8VLQJ WKH VDPH QHXURQ DFWLYH IXQFWLRQ J��� DV ([DPSOH ����WKH HTXLOLEULXP VWDWH RI WKH QHXUDO QHWZRUN

7X ���������������������������� �

7Y ������������������� DQG WKH FRUUHVSRQGLQJ

RSWLPXP VROXWLRQ 7[ ���������������������������� � �

7\ ������������������� � FDQ EH REWDLQHG E\ PHDQV

RI XVLQJ WKH LPSURYHG (XOHULDQ IRUPXOD� 7KHVH

FRPSXWDWLRQ UHVXOWV DUH WRWDOO\ LGHQWLFDO ZLWK WKH

WKHRUHWLFDO VROXWLRQV�

�� &RQFOXVLRQ

$ QRYHO NLQG RI QHXUDO QHWZRUN PRGHO IRU VROYLQJ WKH

TXDGUDWLF SURJUDPPLQJ SUREOHP EDVHG RQ WKH /DJUDQJH

PXOWLSOLHU WKHRU\ LV SUHVHQWHG LQ WKLV SDSHU� 7KH

HTXLOLEULXP SRLQW RI WKH FLUFXLW LV VKRZQ WR FRUUHVSRQG WR

WKH /DJUDQJH VROXWLRQ RI WKH SUREOHP� $ /\DSXQRY

IXQFWLRQ LV HVWDEOLVKHG IRU WKH JOREDO VWDELOLW\ DQDO\VLV�

7KURXJK WKHRU\ DQDO\VLV DQG H[DPSOH FRPSXWDWLRQV� LW KDV

EHHQ VKRZQ WKDW WKH QHXUDO QHWZRUN LV DEOH WR JOREDOO\

VWDELOL]H DQG FRQYHUJH LQ WKH RSWLPXP VROXWLRQ� ,W LV DEOHWR UDLVH WKH VSHHG RI WKH RSWLPL]DWLRQ FRPSXWDWLRQ WR VHOHFW

VXLWDEO\ WKH QHXURQ DFWLYH IXQFWLRQ DFFRUGLQJ WR WKH

QRQOLQHDU OHYHO RI WKH GLVFXVVHG SUREOHP� 8VLQJ D VXLWDEOH

RSWLPL]DWLRQ VWUDWHJ\ FDQ UDLVH WKH FRPSXWDWLRQ VSHHG RI

WKH QHWZRUN� 7KH SURSRVDO QHXUDO QHWZRUN VWUXFWXUH LQ WKH

SDSHU LV VLPSOH DQG FDQ HDVLO\ EH LPSOHPHQWHG XVLQJ

DSSURSULDWH HOHFWURQLF FLUFXLWV�

5HIHUHQFHV

>�@ -� -� +RSILHOG DQG '� :� 7DQN� ¦1HXUDO FRPSXWDWLRQ RIGHFLVLRQ LQ RSWLPL]DWLRQ SUREOHP�§ %LRORJLFDO &\EHUQHWLFV�

����� ��� ��������>�@ &� <� 0DD DQG 0� $� 6KDQEODWW� ¦/LQHDU DQG TXDGUDWLF

SURJUDPPLQJ QHXUDO QHWZRUN DQDO\VLV�§ 1HXUDO 1HWZRUNV �

����� �� ��������>�@ '� :� 7DQN DQG -� -� +RSILHOG� ¦6LPSOH QHXUDO RSWLPL]DWLRQ

QHWZRUNV� $Q $�& FRQYHUWHU� VLJQDO GHFLVLRQ FLUFXLW� DQG DOLQHDU SURJUDPPLQJ�§ ,((( 7UDQV� RQ &LUFXLW DQG 6\VWHP������ ��� �� ��������

>�@ -� 0XUUH� ¦1HXUDO QHWZRUN IRU TXDGUDWLF RSWLPL]DWLRQ ZLWKERXQG FRQVWUDLQWV�§ 1HXUDO 1HWZRUNV� ����� �� ��������

>�@ 0� 3� .HQQHG\ DQG /� 2� &KXD� ¦1HXUDO QHWZRUNV IRUQRQOLQHDU SURJUDPPLQJ�§ ,((( 7UDQV� RQ &LUFXLW DQG6\VWHP� ����� ��� � �� ��������

>�@ $� .� 'KLQJUD DQG 6� 6� 5DR� ¦$ QHXUDO QHWZRUN EDVHGDSSURDFK WR PHFKDQLFDO GHVLJQ RSWLPL]DWLRQ�§ (QJLQHHULQJ

2SWLPL]DWLRQ� ����� ��� ��������

>�@ 6� :� =KDQJ DQG $� *� &RQVWDQWLQLGHV� ¦ /DJUDQJHSURJUDPPLQJ QHXUDO QHWZRUNV�§ ,((( 7UDQV� RQ &LUFXLW DQG6\VWHP� &$6�,,� ���� � ��� �� ��������

>�@ )QHJ 3HLHQ DQG 4LX 4LQJ \LQJ� ¦5HVHDUFK RQ PRGHOLQJ DQGFRQWURO VWUDWHJ\ RI FRQVWUDLQHG RSWLPL]DWLRQ QHXUDO QHWZRUN�§$FWD (OHFUWRQLFD 6LQLFD� ����� ��� �� ������

>�@ /� 2� &KXD DQG *� 1� /LQ� ¦1RQOLQHDU SURJUDPPLQJ ZLWKRXWFRPSXWDWLRQ�§ ,((( 7UDQV� RQ &LUFXLW DQG 6\VWHP � �����

&$6���� ��������

>��@ 7RVKL\DVX 7DNHIXML -XQ :DQJ� 1HXUDO FRPSXWLQJ IRURSWLPL]DWLRQ DQG FRPELQDWRULFV� 6LQJDSRUH� ZRUOG 6FLHQWLILF������

>��@ 5�NHQW QDJOH (GZDUG %�VDII� )XQGDPHQWDO RI GLIIHUHQWLDO(TXDWLRQV DQG %RXQGDU\ 9DOXH 3UREOHPV $GGLVRQ�:HVLH\3XEOLVKLQJ &RPSDQ\� ����

>��@ 1RUPDQ 6� 1LVH� &RQWURO 6\VWHPV (QJLQHHULQJ� 7KH

%HQMDPLQ�&XPPLQJV 3XEOLVKLQJ &RPSDQ\ ,1&� ����