US7711663B2  Multilayer development network having inplace learning  Google Patents
Multilayer development network having inplace learning Download PDFInfo
 Publication number
 US7711663B2 US7711663B2 US11/728,711 US72871107A US7711663B2 US 7711663 B2 US7711663 B2 US 7711663B2 US 72871107 A US72871107 A US 72871107A US 7711663 B2 US7711663 B2 US 7711663B2
 Authority
 US
 United States
 Prior art keywords
 neurons
 learning
 network
 layer
 input
 Prior art date
 Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
 Active, expires
Links
 230000018109 developmental process Effects 0.000 title description 11
 210000002569 neurons Anatomy 0.000 claims abstract description 163
 230000004044 response Effects 0.000 claims abstract description 44
 230000003109 amnesic Effects 0.000 claims abstract description 30
 210000004027 cells Anatomy 0.000 claims abstract description 22
 239000010410 layer Substances 0.000 claims description 115
 230000023886 lateral inhibition Effects 0.000 claims description 14
 230000000875 corresponding Effects 0.000 claims description 3
 230000001419 dependent Effects 0.000 claims description 2
 239000002365 multiple layer Substances 0.000 claims 1
 230000002401 inhibitory effect Effects 0.000 description 11
 230000001054 cortical Effects 0.000 description 10
 239000011159 matrix material Substances 0.000 description 10
 210000000225 Synapses Anatomy 0.000 description 7
 230000002964 excitative Effects 0.000 description 7
 238000000034 method Methods 0.000 description 7
 230000000946 synaptic Effects 0.000 description 7
 230000000694 effects Effects 0.000 description 6
 238000010304 firing Methods 0.000 description 6
 238000004458 analytical method Methods 0.000 description 5
 238000010586 diagram Methods 0.000 description 5
 230000004301 light adaptation Effects 0.000 description 5
 230000001537 neural Effects 0.000 description 5
 101700042008 TOP1 Proteins 0.000 description 4
 230000036748 firing rate Effects 0.000 description 4
 230000002787 reinforcement Effects 0.000 description 4
 230000005654 stationary process Effects 0.000 description 4
 238000000513 principal component analysis Methods 0.000 description 3
 230000001953 sensory Effects 0.000 description 3
 239000002356 single layer Substances 0.000 description 3
 230000004438 eyesight Effects 0.000 description 2
 230000002068 genetic Effects 0.000 description 2
 230000001939 inductive effect Effects 0.000 description 2
 230000000306 recurrent Effects 0.000 description 2
 210000004556 Brain Anatomy 0.000 description 1
 210000003855 Cell Nucleus Anatomy 0.000 description 1
 210000000887 Face Anatomy 0.000 description 1
 238000007476 Maximum Likelihood Methods 0.000 description 1
 210000000977 Primary visual cortex Anatomy 0.000 description 1
 210000003994 Retinal Ganglion Cells Anatomy 0.000 description 1
 210000003491 Skin Anatomy 0.000 description 1
 210000000239 Visual Pathways Anatomy 0.000 description 1
 230000003044 adaptive Effects 0.000 description 1
 229930002945 alltransretinaldehyde Natural products 0.000 description 1
 230000001174 ascending Effects 0.000 description 1
 238000005094 computer simulation Methods 0.000 description 1
 230000001143 conditioned Effects 0.000 description 1
 230000002596 correlated Effects 0.000 description 1
 238000001514 detection method Methods 0.000 description 1
 230000001973 epigenetic Effects 0.000 description 1
 230000001747 exhibiting Effects 0.000 description 1
 238000002474 experimental method Methods 0.000 description 1
 230000003993 interaction Effects 0.000 description 1
 239000012528 membrane Substances 0.000 description 1
 230000004048 modification Effects 0.000 description 1
 238000006011 modification reaction Methods 0.000 description 1
 238000005192 partition Methods 0.000 description 1
 230000002207 retinal Effects 0.000 description 1
 235000020945 retinal Nutrition 0.000 description 1
 239000011604 retinal Substances 0.000 description 1
 230000004256 retinal image Effects 0.000 description 1
 238000005070 sampling Methods 0.000 description 1
 230000031893 sensory processing Effects 0.000 description 1
 238000001228 spectrum Methods 0.000 description 1
 230000001629 suppression Effects 0.000 description 1
 230000002123 temporal effect Effects 0.000 description 1
 230000004400 visual pathway Effects 0.000 description 1
Images
Classifications

 G—PHYSICS
 G06—COMPUTING; CALCULATING; COUNTING
 G06N—COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS
 G06N3/00—Computer systems based on biological models
 G06N3/02—Computer systems based on biological models using neural network models
 G06N3/08—Learning methods

 G—PHYSICS
 G06—COMPUTING; CALCULATING; COUNTING
 G06K—RECOGNITION OF DATA; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
 G06K9/00—Methods or arrangements for reading or recognising printed or written characters or for recognising patterns, e.g. fingerprints
 G06K9/36—Image preprocessing, i.e. processing the image information without deciding about the identity of the image
 G06K9/46—Extraction of features or characteristics of the image
 G06K9/4604—Detecting partial patterns, e.g. edges or contours, or configurations, e.g. loops, corners, strokes, intersections
 G06K9/4609—Detecting partial patterns, e.g. edges or contours, or configurations, e.g. loops, corners, strokes, intersections by matching or filtering
 G06K9/4619—Biologicallyinspired filters, e.g. receptive fields
 G06K9/4623—Biologicallyinspired filters, e.g. receptive fields with interaction between the responses of different filters
Abstract
Description

 Type1 batch: A batch learning algorithm L_{1 }computes g and w using a batch of vector inputs B={x_{1}, x_{2}, . . . , x_{b}}, where b is the batch size.
(g,w)=L _{1}(B), (1)  where the argument B on the right is the input to the learner, L_{1 }and the right side is its output. The learning algorithm needs to store an entire batch to input vectors B before learning can take place. Since L_{1 }requires the additional storage of B, L_{1 }must be realized by a separate network L_{1 }and thus, the learning of the (learning) network L_{1 }is an open problem.
 Type2 blockincremental: A type2 learning algorithm, L_{2}, breaks a series of input vectors into blocks of certain size b (b>1) and computes updates incrementally between blocks. Within each block, the processing by L_{2 }is in a batch fashion.
 Type3 incremental: Each input vector must be used immediately for updating the learner's memory (which must not store all the input vectors) and then the input must be discarded before receiving the next input. Type3 is the extreme case of Type2 in the sense that block size b=1. A type2 algorithm, such as Infomax, becomes a Type3 algorithm by setting b=1, but the performance will further suffer.
 Type4 covariancefree incremental: A Type4 learning algorithm L_{4 }is a Type3 algorithm, but furthermore, it is not allowed to compute the 2^{nd }or higher order statistics of the input x. In other words, the learner's memory M^{(t) }cannot contain the second order (e.g., correlation or covariance) or higher order statistics of x. The CCI PCA algorithm is a covariancefree incremental learning algorithm for computing principal components as the weight vectors of neurons. Further information regarding this algorithm may be found in a paper by Weng et al entitled “Candid Covariancefree Incremental Principle Component Analysis” IEEE Trans. Pattern Analysis and Machine Intelligence, 25(8):10341040 (2003).
 Type5 inplace neuron learning: A Type5 learning algorithm L_{5 }is a Type4 algorithm, but further the learner L_{5 }must be implemented by the signal processing neuron N itself. A term “local learning” used by some researchers does not imply inplace. For example, for a neuron N, its signal processor model has two parts, the synaptic weight w^{(t) }and the sigmoidal function g^{(t)}, both updated up to t. A type5 learning algorithm L_{5 }must update them using the previously updated weight w^{(t−1) }and the sigmoidal function g^{(t−1)}, using the current input x_{t }while keeping its maturity indicated by t:
(w ^{(t)} ,g ^{(t)} ,t)=L _{5}(w ^{(t−1)} ,g ^{(t−1)} ,t−1,x _{t}). (2)  After the adaptation, the computation of the response is realized by the neuron N:
y _{t} =g ^{(t)}(w ^{(t)} ·x _{t}). (3)  An inplace learning algorithm must realize L_{5 }and the computation above by the same neuron N, for t=1, 2, . . . .
 Type1 batch: A batch learning algorithm L_{1 }computes g and w using a batch of vector inputs B={x_{1}, x_{2}, . . . , x_{b}}, where b is the batch size.
z _{1} =g(w·x−h·z) (4)
where w consists of nonnegative weights for excitatory input x, while h consists of nonnegative weights for nonnegative inhibitory input z but its ith component is zero so that the right side does not require z_{i}. For biological plausibility, we assume that all the components in x and z are nonnegative. The source of z is the response of neighboring neurons in the same layer.
where τ_{w }is a time constant (or step size) that controls the rate at which the weight w changes, and the operator E denotes expectation (i.e., average) over observations x and υ=w·x. In general, the Hebbian rule means that the learning rate of a neuron is closely related to the product of its current response and the current input, although the relation is typically nonlinear. Plug υ−w·x into Eq. (5), we get
The above expression gives rise to the need to compute the correlation matrix of x in many learning algorithms, resulting in Type1 and Type2 algorithms in unsupervised learning.
Y=R_{1}∪R_{2}∪ . . . ∪R_{c}, (6)
(where ∪ denotes the union of two spaces) as illustrated in
We can see that the best lobe component vector v_{i}, scaled by “energy estimate” eigenvalue λ_{i,1}, can be estimated by the average of the input vector y(t) weighted by then linearized (without g) response to y(t)·v_{i }whenever y(t) belongs to R_{i}. This average expression is very important in guiding the adaptation of v_{i }in the optimal statistical efficiency as explained below.
Then Eq. (7) states that the lobe component vector is estimated by the average:
in which the Fisher information matrix I(θ) is the covariance matrix of the score vector
and ƒ(x,θ) is the probability density of random vector x if the true parameter value is θ. The matrix I(θ)^{−1 }is called information bound since under some regularity constraints, any unbiased estimator {tilde over (θ)} of the parameter vector θ satisfies cov ({tilde over (θ)}−θ)≧I(θ)^{−1}/n.
where μ(t) is the amnesic function depending on t. If μ=0, the above gives the straight incremental mean. The way to compute a mean incrementally is not new but the way to use the amnesic function of n is new for computing a mean incrementally.
in which, e.g., c=2, r=10000. As can be seen above, μ(t) has three intervals. When t is small, straight incremental average is computed. Then μ(t) changes from 0 to 2 linearly in the second interval. Finally, t enters the third section where μ(t) increases at a rate of about 1/r, meaning the second weight (1+μ(t))/t in Eq. (11) approaches a constant 1/r, to slowly trace the slowly changing distribution.
where w_{t}(n) is the weight of data item x_{t }which entered at time t≦n in the amnesic mean
Since all the multiplicative factors above are nonnegative, we have w_{t}(n)≧0, t=1, 2, . . . , n. Using the induction on n, it can be proven that all the weights w_{t}(n) sum to one for any n≧1:
(When n=1, we require that μ(1)=0.) Suppose that the sample x_{t }are independently distributed (i.i.d.) with the same distribution as a random variable x. Then, the amnesic mean is an unbiased estimator of Ex:
Let cov (x) denote the covariance matrix of x. The expected mean square error of the amnesic mean
where we defined the error coefficient:
When μ(t)=0 for all n, the error coefficient becomes c(n)=1/n and Eq. (15) returns to the expected square error of the regular sample mean:
 a) Compute the membrane potential for all neurons: For all i with 1≦i≦c:
 The firing rate of the neuron i is r_{i}=g_{i}(z_{i}), where g is the sigmoidal function of neuron _{i}. In a multilayer network, the v_{i }includes three parts: bottomup weight, laternal weight, and topdown weight.
 b) (i) Topk response version: Simulating lateral inhibition, decide the topk winners the firing rates, where k>0.
 (ii) Dynamic response version: The lateral inhibition is computed directly using laternal inhibitory connections for every neuron and, thus, the total number of neurons that fire in each layer is dynamic.
 c) Update only the winner (or firing) neuron v_{j }using its temporally scheduled plasticity:
v _{j} ^{(t)} =w _{1} v _{j} ^{(t−1)} +w _{2} z _{j} y(t), where the scheduled plasticity is determined by its two agedependent weights:
 with w_{1}+w_{2}≡1. (That is, the bottomup, lateral and topdown weights are all updated for the firing neuron.) Update the number of hits (cell age) n(j)←+n(j)+1.
 d) All other neurons keep their ages and weight unchanged: For all 1≦i≦c,i≠j,v_{i} ^{(t)}=v_{i} ^{(t−1)}.
 The neuron winning mechanism corresponds to the well known mechanism called lateral inhibition (see, e.g., Kandel et al. Principles of Neural Science. McGrawHill, New York pg 4623 (2000)). The winner updating rule is a computer simulation of the Hebbian rule (see, e.g., Kandel et al at pg 1262). Assuming the plasticity scheduling by w_{1 }and w_{2 }are realized by the genetic and physiologic mechanisms of the cell, this algorithm is inplace.
y _{i} ^{(l)}(t)=g _{i} ^{(l)}(w _{b} ·y ^{(l−1)}(t)−w _{h} ·y ^{(l)}(t)+w _{t} ·a(t−1)). (18)
Note that the ith component in w_{h }is zero, meaning that neuron i does not use the response of itself to inhibit itself

 MILN Learning: For t=0, 1, 2, . . . do the following:
 1) Grab the current input frame.
 2) If the current desired output frame is available, set the output of the output layer L.
 3) For j=1, 2, . . . , L, run the singlelayer inplace learning algorithm on layer j, using Eq. (17) for computing response.
W _{i,j} =e ^{−d} ^{ i,j }, where d _{i,j} =i ^{2} +j ^{2}. (19)
This weight modifies w_{2 }in the CCI LCA algorithm through a multiplication, but the modified w_{1 }and w_{2 }still sum to one.
where l is the original response. All other neurons with i≧k are suppressed to have zero responses.
Claims (28)
v _{j} ^{(t)} =w _{1} v _{j} ^{(t−1)} +w _{2} z _{j} y(t),
Priority Applications (2)
Application Number  Priority Date  Filing Date  Title 

US78625706P true  20060327  20060327  
US11/728,711 US7711663B2 (en)  20060327  20070327  Multilayer development network having inplace learning 
Applications Claiming Priority (1)
Application Number  Priority Date  Filing Date  Title 

US11/728,711 US7711663B2 (en)  20060327  20070327  Multilayer development network having inplace learning 
Publications (2)
Publication Number  Publication Date 

US20080005048A1 US20080005048A1 (en)  20080103 
US7711663B2 true US7711663B2 (en)  20100504 
Family
ID=38877916
Family Applications (1)
Application Number  Title  Priority Date  Filing Date 

US11/728,711 Active 20281024 US7711663B2 (en)  20060327  20070327  Multilayer development network having inplace learning 
Country Status (1)
Country  Link 

US (1)  US7711663B2 (en) 
Cited By (10)
Publication number  Priority date  Publication date  Assignee  Title 

US20100312730A1 (en) *  20090529  20101209  Board Of Trustees Of Michigan State University  Neuromorphic spatiotemporal wherewhat machines 
US9262698B1 (en) *  20120515  20160216  Vicarious Fpc, Inc.  Method and apparatus for recognizing objects visually using a recursive cortical network 
US9373085B1 (en)  20120515  20160621  Vicarious Fpc, Inc.  System and method for a recursive cortical network 
US10275705B2 (en)  20140808  20190430  Vicarious Fpc, Inc.  Systems and methods for generating data explanations for neural networks and related systems 
US10343279B2 (en)  20150710  20190709  Board Of Trustees Of Michigan State University  Navigational control of robotic systems and other computerimplemented processes using developmental network with turing machine learning 
US10579922B2 (en)  20140408  20200303  Microsoft Technology Licensing, Llc  Deep learning using alternating direction method of multipliers 
US10635813B2 (en)  20171006  20200428  Sophos Limited  Methods and apparatus for using machine learning on multiple file fragments to identify malware 
WO2020095321A3 (en) *  20181106  20200625  Vishwajeet Singh Thakur  Dynamic structure neural machine for solving prediction problems with uses in machine learning 
US11003774B2 (en)  20180126  20210511  Sophos Limited  Methods and apparatus for detection of malicious documents using machine learning 
US11080617B1 (en) *  20171103  20210803  Paypal, Inc.  Preservation of causal information for machine learning 
Families Citing this family (11)
Publication number  Priority date  Publication date  Assignee  Title 

WO2006053050A2 (en) *  20041108  20060518  Commvault Systems, Inc.  System and method for performing auxiliary storage operations 
US8799342B2 (en) *  20070828  20140805  Honda Motor Co., Ltd.  Signal processing device 
US8843426B2 (en) *  20110525  20140923  Qualcomm Incorporated  Method and apparatus of primary visual cortex simple cell training and operation 
US8583577B2 (en) *  20110525  20131112  Qualcomm Incorporated  Method and apparatus for unsupervised training of input synapses of primary visual cortex simple cells and other neural circuits 
US9477926B2 (en) *  20121120  20161025  Qualcomm Incorporated  Piecewise linear neuron modeling 
EP3304437B1 (en) *  20150605  20210526  DeepMind Technologies Limited  Whitened neural network layers 
US10163022B1 (en) *  20170622  20181225  StradVision, Inc.  Method for learning text recognition, method for recognizing text using the same, and apparatus for learning text recognition, apparatus for recognizing text using the same 
EP3502973A1 (en) *  20171219  20190626  InterDigital CE Patent Holdings  Neural network and method for training the neural network, corresponding computer program products and devices 
CN108182454B (en) *  20180118  20210730  南栖仙策(南京)科技有限公司  Security check identification system and control method thereof 
CN111615705A (en) *  20180313  20200901  赫尔实验室有限公司  Sparse associative memory for object recognition 
CN108616837A (en) *  20180413  20181002  中南大学  A kind of Topk query method for the doubledeck Sensor Network 
Citations (4)
Publication number  Priority date  Publication date  Assignee  Title 

US6081273A (en) *  19960131  20000627  Michigan State University  Method and system for building threedimensional object models 
US6169981B1 (en) *  19960604  20010102  Paul J. Werbos  3brain architecture for an intelligent decision and control system 
US6353814B1 (en) *  19971008  20020305  Michigan State University  Developmental learning machine and method 
US6904335B2 (en) *  20020821  20050607  Neal Solomon  System, method and apparatus for organizing groups of selfconfigurable mobile robotic agents in a multirobotic system 

2007
 20070327 US US11/728,711 patent/US7711663B2/en active Active
Patent Citations (6)
Publication number  Priority date  Publication date  Assignee  Title 

US6081273A (en) *  19960131  20000627  Michigan State University  Method and system for building threedimensional object models 
US6169981B1 (en) *  19960604  20010102  Paul J. Werbos  3brain architecture for an intelligent decision and control system 
US6581048B1 (en) *  19960604  20030617  Paul J. Werbos  3brain architecture for an intelligent decision and control system 
US6353814B1 (en) *  19971008  20020305  Michigan State University  Developmental learning machine and method 
US6904335B2 (en) *  20020821  20050607  Neal Solomon  System, method and apparatus for organizing groups of selfconfigurable mobile robotic agents in a multirobotic system 
US7343222B2 (en) *  20020821  20080311  Solomon Research Llc  System, method and apparatus for organizing groups of selfconfigurable mobile robotic agents in a multirobotic system 
Cited By (13)
Publication number  Priority date  Publication date  Assignee  Title 

US20100312730A1 (en) *  20090529  20101209  Board Of Trustees Of Michigan State University  Neuromorphic spatiotemporal wherewhat machines 
US8694449B2 (en)  20090529  20140408  Board Of Trustees Of Michigan State University  Neuromorphic spatiotemporal wherewhat machines 
US9262698B1 (en) *  20120515  20160216  Vicarious Fpc, Inc.  Method and apparatus for recognizing objects visually using a recursive cortical network 
US9373085B1 (en)  20120515  20160621  Vicarious Fpc, Inc.  System and method for a recursive cortical network 
US9607262B2 (en)  20120515  20170328  Vicarious Fpc, Inc.  System and method for a recursive cortical network 
US9607263B2 (en)  20120515  20170328  Vicarious Fpc, Inc.  System and method for a recursive cortical network 
US10579922B2 (en)  20140408  20200303  Microsoft Technology Licensing, Llc  Deep learning using alternating direction method of multipliers 
US10275705B2 (en)  20140808  20190430  Vicarious Fpc, Inc.  Systems and methods for generating data explanations for neural networks and related systems 
US10343279B2 (en)  20150710  20190709  Board Of Trustees Of Michigan State University  Navigational control of robotic systems and other computerimplemented processes using developmental network with turing machine learning 
US10635813B2 (en)  20171006  20200428  Sophos Limited  Methods and apparatus for using machine learning on multiple file fragments to identify malware 
US11080617B1 (en) *  20171103  20210803  Paypal, Inc.  Preservation of causal information for machine learning 
US11003774B2 (en)  20180126  20210511  Sophos Limited  Methods and apparatus for detection of malicious documents using machine learning 
WO2020095321A3 (en) *  20181106  20200625  Vishwajeet Singh Thakur  Dynamic structure neural machine for solving prediction problems with uses in machine learning 
Also Published As
Publication number  Publication date 

US20080005048A1 (en)  20080103 
Similar Documents
Publication  Publication Date  Title 

US7711663B2 (en)  Multilayer development network having inplace learning  
PaugamMoisy et al.  Delay learning and polychronization for reservoir computing  
Yin  The selforganizing maps: background, theories, extensions and applications  
Nadikattu  The Supremacy of Artificial intelligence and Neural Networks  
US5822742A (en)  Dynamically stable associative learning neural network system  
Bala et al.  Classification using ANN: A review  
Becker  Implicit learning in 3D object recognition: The importance of temporal context  
Weng et al.  A multilayer inplace learning network for development of general invariances  
Pereira et al.  Machine learning fundamentals  
Shim et al.  Unsupervised learning in an ensemble of spiking neural networks mediated by itdp  
Weng et al.  Optimal inplace learning and the lobe component analysis  
Buscema et al.  Artificial adaptive systems using auto contractive maps: theory, applications and extensions  
Weng et al.  Brainlike emergent spatial processing  
Globerson et al.  Euclidean Embedding of CoOccurrence Data.  
De et al.  Hybrid soft computing for multilevel image and data segmentation  
Belciug et al.  Data MiningBased Intelligent Decision Support Systems  
Honkela et al.  Artificial Neural Networks and Machine LearningICANN 2011: 21st International Conference on Artificial Neural Networks, Espoo, Finland, June 1417, 2011, Proceedings, Part II  
Weng et al.  Optimal inplace selforganization for cortical development: Limited cells, sparse coding and cortical topography  
Yang et al.  Statistical inference: learning in artificial neural networks  
Ikram  A benchmark for evaluating Deep Learning based Image Analytics  
Weng et al.  Inplace learning for positional and scale invariance  
Khikmatullaev et al.  Capsule neural networks for text classification  
Schwabe et al.  Modeling the adaptive visual system: a survey of principled approaches  
Veličković et al.  ’Traffic Sign Classification Using Convolutional Neural Network ‘  
Huyck et al.  A spiking halfcognitive model for classification 
Legal Events
Date  Code  Title  Description 

AS  Assignment 
Owner name: BOARD OF TRUSTEES OF MICHIGAN STATE UNIVERSITY, MI Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:WENG, JUYANG;REEL/FRAME:019855/0598 Effective date: 20070911 Owner name: BOARD OF TRUSTEES OF MICHIGAN STATE UNIVERSITY,MIC Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:WENG, JUYANG;REEL/FRAME:019855/0598 Effective date: 20070911 

STCF  Information on status: patent grant 
Free format text: PATENTED CASE 

CC  Certificate of correction  
FPAY  Fee payment 
Year of fee payment: 4 

MAFP  Maintenance fee payment 
Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YR, SMALL ENTITY (ORIGINAL EVENT CODE: M2552) Year of fee payment: 8 