reverse engineer a neural network

Neural network solutions are commonly used today but without much understanding of exactly how they wired themselves up  to  map a set of inputs to a desired set of outputs.

Efforts have been made to reverse engineer neural networks solutions.

Biological brain research does the same thing in trying to understand how biological brains process information.

In this example a neural network was trained to do binary addition.

We know how to wire up logic gates to achieve this mapping but what method would a neural network come up with to achieve the same result?

This researcher wanted to know what his trained network was doing internally to generate its solutions.

The researcher figured out the surprising way in which the neural network was accomplishing binary addition.

https://cprimozic.net/blog/reverse-engineering-a-small-neural-network/