What is an Artificial Neural Network or ANN?
A Neural Network is man's attempt to try to imitate the biological neurons of a human brain. Please note that no one fully understands how the neural networks within a human brain actually work. And ANNs (Artificial Neural Networks) are known to not work in the same way that a human brain works. These are just man's ultimate beginning toward trying to understand this concept.
A second thing to note is that Neural Networks are made up of many individual Neurons. Artificial Neurons are typically referred to as "Perceptions" to distinguish them from the biological neurons they were designed to try to mimic. Another thing to note here is that within the human brain there are actually several different types of neurons that behave in different ways. It is also true that there are different types of perceptions. This is to say that there are different ways that perceptrons can be designed to behave in different ways as well. It might also be noteworthy to recognize that perceptrons are just man's attempt to model biological neurons, and therefore perceptrons may not achieve this goal as well as we might like.
Having said all of the above the following video addresses a rather simple ANN that actually contains over a thousand perceptrons. To truly understand how ANNs work it would be wise to study and fully understand how individual perceptrons work before attempting to deal with complex ANNs. Unfortunately due to impatience most people jump straight into working with complex ANNs immediately without having ever taken the time to fully understand how individual perceptrons actually work. But that's a topic for another thread.
Here is a video that explains a fairly "simple" ANN consisting of over a thousand perceptions. This first video explains the basic structure and function of the ANN.
Note in the above video at time = 12:20 to 12:40
He explains that "Learning" simply means: "Setting all the weights and biases for a desired output based on known training data input."
It's important to understand that this is what is meant when we say that an ANN is "learning". It's actually being programmed by a computer algorithm. The ANN itself isn't doing any "learning" at all. It could be argued that to say that the ANN is "learning" is basically a misnomer. Just a bad term to use. It is being "trained" however. And it is being trained by another computer program that goes through thousands, if not millions, of mathematical iterations in order to determine what the weights should be.
In the next video he discusses the mathematics used by this external computer program that is not really part of the ANN at all.
Again, he's talking about how machine "learn", but if you pay close attention the ANN isn't "learning", instead it's simply being designed via another program that determines what the weights of the ANN must be to produce the desired outputs. The term should definitely be "training" instead of "learning".
In fact, event the term "training" is a bit misleading because all that's really happening here is that the computer is "programing" the weights of the ANN based on an iterative mathematical algorithm.
In this second video (time = 0.10) he speaks of "Gradient Descent" and says that this is how ANNs learn.
However, in truth, Gradient Descent does not equal how neural networks learn but rather how they are programmed by a separate computer algorithm. There is nothing in the ANN itself that is capable of programming its own weights.
Untrained Neural Networks produce trash. In other words, an untrained ANN is basically useless.
Neural Networks are "trained" or "programmed" using a Cost Function (video time = 3:00 to 5:20)
Note that neither this Cost Function nor the mathematical algorithm that is used to "train" the ANN is part of the ANN itself. The ANN itself has absolutely no way of "learning" anything on its own.
How to learn how Neural Networks are trained (see video Time 16:37 to 17:30)
You can download a free book that explains precisely how this particular ANN is "trained". The real work is done by the computer algorithm that actually calculates what the weights of the ANN should be. And this book explains how that process is accomplished.
Once the ANN has been trained it then becomes useful. But the ANN itself is incapable of learning anything on its own. At best it can only have its weights updated by an external computer program which isn't actually part of the ANN itself.
Keep in mind that the ANN = Artificial Neural Network. This does not include any external computer programs that might be used to calculate what the weights of the ANN should be. The ANN itself has no way of improving or modifying its own performance.
Just thought I'd share this information for whatever it might be worth for anyone who his interested in learning about ANNs.
If you have comments or different views on any of this please do share them.
An "artificial neural network" may refer to an end product like "a full adder network of NOR gates". However during training it is part of a learning system. If you rip out the "artificial neural network" part of that learning system then of course that part cannot learn by itself. The training program is really just wiring up a network to perform some function which if you were smart enough you could just wire up yourself as you might with a a network of NOR gates in a full adder.
An "artificial neural network" may refer to an end product like "a full adder network of NOR gates". However during training it is part of a learning system. If you rip out the "artificial neural network" part of that learning system then of course that part cannot learn by itself.
Exactly my observation as well. This is why I say that an ANN alone cannot learn anything. It requires a very complex system of computer programs and mathematics to "train" the ANN. So it's those computer programs that represent the learning process, not the ANNs.
The training program is really just wiring up a network to perform some function which if you were smart enough you could just wire up yourself as you might with a a network of NOR gates in a full adder.
Again exactly my observation as well. This is why I'm excited about building hardware ANNs. A hardware ANN is more efficient than running a computer simulation of an ANN for two reasons.
First a hardware ANN would produce the answer much faster than any simulation could ever hope to achieve.
Secondly, placing the ANN in hardware frees up the computer memory and resources to move on to bigger and better things (like designing new ANNs to do other things for example). Why bog down the computer with software ANN simulations when it's not required?
I mean, I can see why large companies might not care since they have extremely fast computers and computing space and power isn't a major problem for them. But for a hobbyist converting ANNs to hardware modules would be highly advantageous on many levels.