Notifications
Clear all

What is an Artificial Neural Network or ANN?

11 Posts
3 Users
0 Reactions
3,481 Views
Robo Pi
(@robo-pi)
Robotics Engineer
Joined: 5 years ago
Posts: 1669
Topic starter  

A Neural Network is man's attempt to try to imitate the biological neurons of a human brain.   Please note that no one fully understands how the neural networks within a human brain actually work.   And ANNs (Artificial Neural Networks) are known to not work in the same way that a human brain works.  These are just man's ultimate beginning toward trying to understand this concept.

A second thing to note is that Neural Networks are made up of many individual Neurons.    Artificial Neurons are typically referred to as "Perceptions" to distinguish them from the biological neurons they were designed to try to mimic.    Another thing to note here is that within the human brain there are actually several different types of neurons that behave in different ways.   It is also true that there are different types of perceptions.  This is to say that there are different ways that perceptrons can be designed to behave in different ways as well.    It might also be noteworthy to recognize that perceptrons are just man's attempt to model biological neurons, and therefore perceptrons may not achieve this goal as well as we might like.

Having said all of the above the following video addresses a rather simple ANN that actually contains over a thousand perceptrons.   To truly understand how ANNs work it would be wise to study and fully understand how individual perceptrons work before attempting to deal with complex ANNs.   Unfortunately due to impatience most people jump straight into working with complex ANNs immediately without having ever taken the time to fully understand how individual perceptrons actually work.  But that's a topic for another thread. 

Here is a video that explains a fairly "simple"  ANN consisting of over a thousand perceptions.   This first video explains the basic structure and function of the ANN.

Note in the above video at time =  12:20 to 12:40

He explains that "Learning" simply means:  "Setting all the weights and biases for a desired output based on known training data input."

It's important to understand that this is what is meant when we say that an ANN is "learning".    It's actually being programmed by a computer algorithm.  The ANN itself isn't doing any "learning" at all.  It could be argued that to say that the ANN is "learning"  is basically a misnomer.   Just a bad term to use.   It is being "trained" however.  And it is being trained by another computer program that goes through thousands, if not millions, of mathematical iterations in order to determine what the weights should be.

In the next video he discusses the mathematics used by this external computer program that is not really part of the ANN at all.

Again, he's talking about how machine "learn", but if you pay close attention the ANN isn't "learning", instead it's simply being designed via another program that determines what the weights of the ANN must  be to produce the desired outputs.  The term should definitely be "training" instead of "learning".

In fact, event the term "training" is a bit misleading because all that's really happening here is that the computer is "programing" the weights of the ANN based on an iterative mathematical algorithm.

In this second video (time = 0.10) he speaks of "Gradient Descent" and says that this is how ANNs learn.

However, in truth, Gradient Descent does not equal how neural networks learn but rather how they are programmed by a separate computer algorithm.   There is nothing in the ANN itself that is capable of programming its own weights.

Untrained Neural Networks produce trash.   In other words, an untrained ANN is basically useless.

Neural Networks are "trained" or "programmed" using a Cost Function (video time = 3:00 to 5:20)

Note that neither this Cost Function nor the mathematical algorithm that is used to "train" the ANN is part of the ANN itself.   The ANN itself has absolutely no way of "learning" anything on its own.

How to learn how Neural Networks are trained (see video Time 16:37 to 17:30)

You can download a free book that explains precisely how this particular ANN is "trained".  The real work is done by the computer algorithm that actually calculates what the weights of the ANN should be.   And this book explains how that process is accomplished.

Once the ANN has been trained it then becomes useful.   But the ANN itself is incapable of learning anything on its own.  At best it can only have its weights updated by an external computer program which isn't actually part of the ANN itself.

Keep in mind that the ANN = Artificial Neural Network.   This does not include any external computer programs that might be used to calculate what the weights of the ANN should be.   The ANN itself has no way of improving or modifying its own performance.

Just thought I'd  share this information for whatever it might be worth for anyone who his interested in learning about ANNs.

If you have comments or different views on any of this please do share them.

DroneBot Workshop Robotics Engineer
James


   
Quote
robotBuilder
(@robotbuilder)
Member
Joined: 5 years ago
Posts: 2230
 

An "artificial neural network" may refer to an end product like "a full adder network of NOR gates".  However during training it is part of a learning system. If you rip out the "artificial neural network" part of that learning system then of course that part cannot learn by itself.  The training program is really just wiring up a network to perform some function which if you were smart enough you could just wire up yourself as you might with a a network of NOR gates in a full adder.

 


   
ReplyQuote
Robo Pi
(@robo-pi)
Robotics Engineer
Joined: 5 years ago
Posts: 1669
Topic starter  
Posted by: @casey

An "artificial neural network" may refer to an end product like "a full adder network of NOR gates".  However during training it is part of a learning system. If you rip out the "artificial neural network" part of that learning system then of course that part cannot learn by itself.

Exactly my observation as well.   This is why I say that an ANN alone cannot learn anything.   It requires a very complex system of computer programs and mathematics to "train" the ANN.   So it's those computer programs that represent the learning process, not the ANNs. 

Posted by: @casey

The training program is really just wiring up a network to perform some function which if you were smart enough you could just wire up yourself as you might with a a network of NOR gates in a full adder.

Again exactly my observation as well.   This is why I'm excited about building hardware ANNs.   A hardware ANN is more efficient than running a computer simulation of an ANN for two reasons.

First a hardware ANN would produce the answer much faster than any simulation could ever hope to achieve.

Secondly, placing the ANN in hardware frees up the computer memory and resources to move on to bigger and better things (like designing new ANNs to do other things for example).  Why bog down the computer with software ANN simulations when it's not required?

I mean, I can see why large companies might not care since they have extremely fast computers and computing space and power isn't a major problem for them.   But for a hobbyist converting ANNs to hardware modules would be highly advantageous on many levels.

DroneBot Workshop Robotics Engineer
James


   
ReplyQuote
ricknl
(@ricknl)
Member
Joined: 5 years ago
Posts: 15
 

It’s not just mathematics that’s required for a ANN to learn (or be trained, depending on where you draw the border). An ANN adapts to new experiences it has, so without those experiences, nothing will happen. The benefit of an ANN is that for problems so complex we can’t wrap our mind around it, we might be able to get sufficient experience for our ANN to figure out how it’s working, and either accept it works to some degree or try and figure out what it has learned and see if we can formulate that simpler or more general.

 

I’ve recently finished a project at work where among other things, insufficient examples (experiences) were available for the Convolutional Neural Net (CNN) to generalize well. The model was good enough to have an error rate of 1 in 90 on the training set, whereas the validation set still had a 1 in 11 error rate, given a 1 in 5 error rate for always guessing “no”. If I would have had more examples, my model would have been able to see a larger part of the distribution and therefor overfit less.

I have to admit I have no knowledge of hardware ANNs and I don’t see how a hardware ANN will adapt to new experiences, other than some training program that tells you how to rearrange your hardware ANN.


   
ReplyQuote
Robo Pi
(@robo-pi)
Robotics Engineer
Joined: 5 years ago
Posts: 1669
Topic starter  
Posted by: @ricknl

It’s not just mathematics that’s required for a ANN to learn (or be trained, depending on where you draw the border). An ANN adapts to new experiences it has, so without those experiences, nothing will happen.

This is a good point you make here, because most ANNs I've seen need to be trained prior to being of any use at all.   And once they have been trained then that's it.  They can only do what they have been trained on.

When you say that an ANN adapts to new experience I would need to question how it does that.  The examples I've seen of ANN  do not contain anything that would allow them to be dynamically retrained.

Of course you did say, (depending on where you draw the border).  That's a very good point.  If you have an ANN that can be dynamically retrained  or updated, then there must be some additional software associated with the ANN that is retraining it.  So that wouldn't be just an ANN alone.   That would be an ANN plus additional software.

Posted by: @ricknl

I have to admit I have no knowledge of hardware ANNs and I don’t see how a hardware ANN will adapt to new experiences, other than some training program that tells you how to rearrange your hardware ANN.

I agree, designing a hardware ANN that can be dynamically reconfigured is quite a bit more complex.  It can be done, but once again, in order to be able to provide the information required to dynamically retrain it, some other type of software algorithm would need to also be included.  There's nothing in an ANN that has the capability of retraining itself.   Unless you are including software other than the ANN itself.

Here's how a static hardware ANN can be used:

Let's say you have designed a software ANN and  you have trained it to recognize a specific individual.   That's all it does.   That's what it has been trained to do, and it cannot doing anything other than this without further  training or updating.

Now you have an ANN that can recognize a specific in individual.   The ANN no longer needs the training software.   In other words, you can upload the trained configuration into the ANN.  And now you have a stand  alone ANN that can  recognize a specific individual.  And that's all it can do.

But now that you have this ANN configured and trained it no longer needs to be retained in software.  In fact, to remain in software requires that a CPU is constantly doing mathematics in order for the ANN to continue to function because  equations need to be performed at each neural node to decide whether or not they need to fire.

If you now take that ANN configuration and move it over to a hardwired op-amp circuit board where the op-amps have been hardwired for the correct  "mathematical relationships", then this hardware  ANN can still recognize that single individual and produce the output that  signals that recognition has been made.

So now  you have an electronic  hardware ANN that can recognize a single specific individual.    No mathematical calculations or CPU power are  required at this point.  It's just a PC board that reports when a  specific individual has been recognized.   This hardware ANN would then  be much faster than its  software origins in terms of  response, as well as no longer requiring  CPU and memory resources to hold it in memory and do the calculations that op-amps would basically do instantly.   Of course op-amps don't actually do mathematical calculations, they simply produce an  output signal that can be mathematically described relative to their input signals.   So at this point it's all analog with no digital mathematics going on at all.

 ~~~~~

Now to the question of being able to dynamically reconfigure or train a hardware ANN.   The answer is yes, this can  indeed be done.   However, in order to do this, just like a software ANN, you would need additional software programs that can  calculate the new configuration, and then upload that new configuration into the hardware ANN.

How could you load a new configuration into a hardware ANN?   FPGAs or CPDLs are the answer to that question.   The FPGA or CPDL would need to be part of the hardware of the ANN.   And they would be what determine the configuration of the op-amps.

In fact, it's my guess that this will eventually be invented at some point.  Instead of having to build  hardware ANNs that are reconfigured with  FPGAs or CPDLs, it will all be done inside of one chip.   And that chip will simply be a programmable hardware ANN.

I've searched everywhere on the Internet looking to buy such a device but apparently they haven't yet been invented.   At least not for mass production.  I have, however,  found articles where other researches are working on very similar technologies.   So I imagine we'll be seeing these in the future.  I'm  certainly not the only one who has recognized the value of  dynamically reconfigurable hardware  ANNs.

I'll be designing my own from scratch, but it's highly unlikely that I'll ever be recognized for this invention because other people who have far more money and research  teams are already working on this idea.   So I'll just be doing it for fun and never get paid for it or even recognized for having done it.  The perils of being a low-budget hobbyist. ?

DroneBot Workshop Robotics Engineer
James


   
ReplyQuote
ricknl
(@ricknl)
Member
Joined: 5 years ago
Posts: 15
 

@robo-pi

That sounds amazing!


   
ReplyQuote
Robo Pi
(@robo-pi)
Robotics Engineer
Joined: 5 years ago
Posts: 1669
Topic starter  

@ricknl

It is an exciting and interesting project and that's what give me incentive to work on it even though there won't be any monetary rewards.  In fact, the whole project is costing me money. ?   I just ordered a bunch of op amps and I'm hoping to create some very simple perceptrons.  No multi-layered neural networks planned for now.

In fact, to be quite open and  honest about it my knowledge of designing software ANNs is quite limited.  I've read a lot about it but have  virtually no experience actually designing them.  This is slowly changing as I continue to study.   But for now, I  feel that I have a good overview of what's going on.

You may be just the person I needed to meet. ? 

I'm currently having some problems with a very simple perceptron program.    Maybe I can share with you what I'm trying to do and you could help me understand what I'm doing wrong, or why I'm not getting the results I'm expecting.

The program is a very short Python program that emulates a simple perceptron.    I  can post the code,  along with a link to the video where I got it.  There doesn't appear to be anything wrong with the actual perceptron.  I get the results from that program that I expect to get and it all makes sense.

However, where I'm running into problems is when I try to graphically draw the decision boundaries.    For some reason I can't seem to get the decision boundaries to draw correctly.   To draw the decision boundaries I'm also using Python along with MatPlotLib to graphically display the decision boundaries.

Are you familiar with Python, Numpy, and MatPlotLib?

If so, you might be able to help me out.  I'll have to dig up my programs and try to figure out where I last left off.  I was becoming frustrated at not being able to draw the decision boundaries based on the data generated by the perceptron program and so I gave up on it for a while to move on to other projects.   But I'd really like to get back to this and figure out how to get these decision boundaries draw correctly from the data provided by the perceptron program.

If you're familiar with Python, Numpy, and MatPlotLib I can dig out my programs and post what I have thus far.  Maybe you can see where I'm going wrong.

DroneBot Workshop Robotics Engineer
James


   
ReplyQuote
ricknl
(@ricknl)
Member
Joined: 5 years ago
Posts: 15
 

@robo-pi

I regularly use numpy and I try to avoid using matplotlib as I know visuals aren’t my strong suit. Nonetheless, post it and I’ll have a look at it.


   
ReplyQuote
Robo Pi
(@robo-pi)
Robotics Engineer
Joined: 5 years ago
Posts: 1669
Topic starter  

@ricknl

Ok, I'll get all the information organized and try to present it in the same way it evolved for me.  I'll probably start a whole new thread dedicated to that conversation.   This will take a while.  I'll need to dig up the old software, re-familiarize myself with it, and then write up posts to explain what I'm doing.    So it's going to be a while.

 

 

DroneBot Workshop Robotics Engineer
James


   
ReplyQuote
robotBuilder
(@robotbuilder)
Member
Joined: 5 years ago
Posts: 2230
 
Posted by: @ricknl
I’ve recently finished a project at work where among other things, insufficient examples (experiences) were available for the Convolutional Neural Net (CNN) to generalize well.

So do you use a commercial CNN product at work?

https://towardsdatascience.com/a-comprehensive-guide-to-convolutional-neural-networks-the-eli5-way-3bd2b1164a53

Take the simple visual problem I gave in another thread.  Find a target in an image and return its x,y coordinates and the dimensions of a box that would enclose the target.  Can you train the CNN to do that?

https://towardsdatascience.com/object-detection-with-neural-networks-a4e2c46b4491

 


   
ReplyQuote
ricknl
(@ricknl)
Member
Joined: 5 years ago
Posts: 15
 

@casey

Initially I tried some small architectures myself, but they were too small to fit even the small amount of data that I had. Later on I tried retraining the Inception-V3 model. Everything I used is open source, with the main packages used being Keras and Tensorflow.

 

As for your target project: my guess is yes, you’ll likely able to build and train a network able to to that. The number of labeled images you need may be large, where a label is for example the height and width of the bounding box and the coordinates of a fixed point on each bounding box, for example the center or upper left corner of the box.


   
ReplyQuote