Evolution of Adaptive Synapses: Robots with Fast Adaptive Behavior in New Environments (2000), Floreano, D., Urzelai, J.

When I first read this paper I was intrigued with their idea of using  genetic evolution to evolve parameters for self organization.  “The method consists of encoding a set of local adaptation rules that synapses follow while the robot freely moves in the environment.”

Since I’m always looking for a new robot controller that I don’t have to hand tune, I thought it might be interesting to try to re-produce their experiment and create a robot controller that can adapt to its environment.

Most of the papers that I’ve read about evolving artificial neural networks (ANN) have been about using genetic algorithms (GA) to find the fittest set of neural connection weights.  But here the weights are randomly selected at the beginning of life for each new generation.  They aren’t pulled from the the parents of the previous generation, the adaptation rules are.

GAs use strings that are encoded to represent some parameter set and in the paper the authors experimented with different types of genetic encodings.  They discovered that encoding neurons with Hebb rules that get applied to each synapse performed better than the typical synapse encoding of synaptic weights.

I decided that I want to try to reproduce their experiment using a physical robot, but not all of the tests they performed just the one that uses the features and properties that were the most performant, namely neuron encoding using Hebb rules.

Encoding the Adaptation Rules

In 1949 a Canadian psychologist, Donald Hebb, came up with the idea that the laws of classical conditioning (remember Pavlov’s dogs?)  reflected the functioning of the nervous system.  The hebb rule states that when two connected neurons are simultaneously active the synaptic weight of their connection is strengthened. 

The experimenters used four adaptation rules.  They all use the idea that there is a direction of signal flow from a pre-synaptic neuron to a post-synaptic neuron.

1. Plain Hebb rule will proportionally strengthen the synapse when the pre-synaptic neuron and the post-synaptic neuron are active.
2. Post-synaptic rule acts like the Hebb rule but it also weakens the synapse when the post-synaptic neuron is active and the pre-synaptic neuron is not.
3. Pre-synaptic rule is the compliment of the post-synaptic rule.
4. Covariance rule will strengthen the synapse when the two neurons have a similar activity level, otherwise it will weaken the synapse.

To keep the synaptic values from growing indefinitely the output is bound to the range with a self-limiting mechanism.  I have no idea what that self limiting mechanism was…normalization? sigmoid()? they don’t say.  

This is an interesting difference from what I’ve done before, the parameters used in the GA are at the neuron level and not the synapse level so all incoming synapses to a neuron share the neuron’s adaptation rule which is used to determine the individual synaptic weights.  Clear as mud?

Tom

@zander 

I just saw an announcement of a RaspberryPi team winning an autonomous car race. That sounded exciting, but then I read that the following steps were required.

1. Blue and Yellow cones are placed around the track.

2. Humans then choose the optimal path

3. Humans created a map.

Car drives 'autonomously'

I don't get it, I am quite certain Tesla cars do not need coloured cones or humans to pre-map their route.

What the heck is different between what Tesla et al do, these contests (I have seen a few recently) and what y'all are doing with this genetics approach?

Is your goal to create an autonomous car/robot? If so, don't you need to find out what Tesla is doing?

Confused am I.

Although the subject matter has been of great interest to me for decades, the reason i have responded, I didn't want to hijack the thread which is about the progress being made in a particular experiment.

With regards your reference I assume you mean this,

https://www.raspberrypi.com/news/raspberry-pi-powers-first-driverless-car-in-formula-sae-brazil-competition

I think the experiment in randell's thread is seeing if he can evolve a set of weights using genetic algorithms for a neural net that will produce a behavior (obstacle avoidance in this case) given a set of inputs (distance measuring sensors) and a set of outputs (two motors).

Self driving cars on the other hand rely on accurate GPS readings and internal road maps just like your GPS navigation system but instead of telling you to turn at 800m the car does the turning itself. An important addition is obstacle avoidance for things not present in this internal map like people and other cars and traffic signs. To detect people and other cars (or any other obstacle) they might use LIDAR, RADAR and visual recognition of traffic lights, road signs as input to the self driving car programs.

The programs use trained (not evolved) neural networks to recognize things like signs or people but the actual programs are not neural networks. They are not evolved solutions either.

Efforts have been made to reverse engineer neural networks solutions.

Biological brain research does the same thing in trying to understand how biological brains process information.

In this example a neural network was trained to do binary addition.

We know how to wire up logic gates to achieve this mapping but what method would a neural network come up with to achieve the same result?

This researcher wanted to know what his trained network was doing internally to generate its solutions.

The researcher figured out the surprising way in which the neural network was accomplishing binary addition.

https://cprimozic.net/blog/reverse-engineering-a-small-neural-network/

@inq wrote:

I'd like you to elaborate on why you believe a wall following bot requires knowing its past time slices.

It might not require its past time slices. However you might note the extract in my previous post. "If some kind of short-term memory mechanism is provided to the neural classifiers, their performances are improved in general."

I just think that without memory there may well be tasks it cannot execute. Let us say it comes to a fork in the road. If the only input is "a fork in the road" it can only have one of two fixed outputs. Turn left or turn right.

If the task was to turn left at the first fork and turn right at the second fork it would need memory of its last action, so with memory the input would become, "a fork in the road, last time you turned left".

I think it was mentioned with the network @THRandell posted the context neurons (memory) enabled it to get out of a trapped situation.

My simulated robot would often get trapped (no memory was implemented) where it would essentially oscillate between two actions. Turn left resulted in an input to turn right which when executed returned an input that said turn left which when executed return an input that said turn right ...

I guess we just have to wait and see if anything comes of it.

One of my roboticist heroes, Dario Floreano, has written about using genetic algorithms to evolve neural networks in his robotics research.  One of his early experiments was in solving a familiar robot task, i.e. getting a robot to move and avoid obstacles. A task that has come up many times in my work with robots.  His idea was to use a “simple” feedforward neural network to map IR proximity sensors to the two motors of a differential drive robot and to use genetic algorithms to derive the fittest individual to perform the task.  Wow!  All new territory for me, but I’m hooked.

First on my list was to understand what a feedforward neural network with recurrent connections is and to get more details on how it was implemented in his experiment.  Lots of jargon to learn.

Automatic Creation of an Autonomous Agent: Genetic Evolution of a Neural-Network Driven Robot (1994), Floreano, D., Mondada, F.

A second thing to note is that Neural Networks are made up of many individual Neurons.    Artificial Neurons are typically referred to as "Perceptions" to distinguish them from the biological neurons they were designed to try to mimic.    Another thing to note here is that within the human brain there are actually several different types of neurons that behave in different ways.   It is also true that there are different types of perceptions.  This is to say that there are different ways that perceptrons can be designed to behave in different ways as well.    It might also be noteworthy to recognize that perceptrons are just man's attempt to model biological neurons, and therefore perceptrons may not achieve this goal as well as we might like.

Having said all of the above the following video addresses a rather simple ANN that actually contains over a thousand perceptrons.   To truly understand how ANNs work it would be wise to study and fully understand how individual perceptrons work before attempting to deal with complex ANNs.   Unfortunately due to impatience most people jump straight into working with complex ANNs immediately without having ever taken the time to fully understand how individual perceptrons actually work.  But that's a topic for another thread. 

Here is a video that explains a fairly "simple"  ANN consisting of over a thousand perceptions.   This first video explains the basic structure and function of the ANN.

https://www.youtube.com/watch?v=aircAruvnKk&t=991s

Note in the above video at time =  12:20 to 12:40

He explains that "Learning" simply means:  "Setting all the weights and biases for a desired output based on known training data input."

It's important to understand that this is what is meant when we say that an ANN is "learning".    It's actually being programmed by a computer algorithm.  The ANN itself isn't doing any "learning" at all.  It could be argued that to say that the ANN is "learning"  is basically a misnomer.   Just a bad term to use.   It is being "trained" however.  And it is being trained by another computer program that goes through thousands, if not millions, of mathematical iterations in order to determine what the weights should be.

In the next video he discusses the mathematics used by this external computer program that is not really part of the ANN at all.

https://www.youtube.com/watch?v=IHZwWFHWa-w

Again, he's talking about how machine "learn", but if you pay close attention the ANN isn't "learning", instead it's simply being designed via another program that determines what the weights of the ANN must  be to produce the desired outputs.  The term should definitely be "training" instead of "learning".

In fact, event the term "training" is a bit misleading because all that's really happening here is that the computer is "programing" the weights of the ANN based on an iterative mathematical algorithm.

In this second video (time = 0.10) he speaks of "Gradient Descent" and says that this is how ANNs learn.

However, in truth, Gradient Descent does not equal how neural networks learn but rather how they are programmed by a separate computer algorithm.   There is nothing in the ANN itself that is capable of programming its own weights.

Untrained Neural Networks produce trash.   In other words, an untrained ANN is basically useless.

Neural Networks are "trained" or "programmed" using a Cost Function (video time = 3:00 to 5:20)

Note that neither this Cost Function nor the mathematical algorithm that is used to "train" the ANN is part of the ANN itself.   The ANN itself has absolutely no way of "learning" anything on its own.

How to learn how Neural Networks are trained (see video Time 16:37 to 17:30)

You can download a free book that explains precisely how this particular ANN is "trained".  The real work is done by the computer algorithm that actually calculates what the weights of the ANN should be.   And this book explains how that process is accomplished.

Once the ANN has been trained it then becomes useful.   But the ANN itself is incapable of learning anything on its own.  At best it can only have its weights updated by an external computer program which isn't actually part of the ANN itself.

Keep in mind that the ANN = Artificial Neural Network.   This does not include any external computer programs that might be used to calculate what the weights of the ANN should be.   The ANN itself has no way of improving or modifying its own performance.

Just thought I'd  share this information for whatever it might be worth for anyone who his interested in learning about ANNs.

If you have comments or different views on any of this please do share them.