Once you complete this (ignoring the first level where you actually build a NAND gate) you essentially get very much what looks like a Neural Network (since it takes 2 neurons to represent a NAND gate), n layers deep, with a lot of zeros in the weight matrices, and some storage.
Here is my question: given the input/output semantics at the assembly level, is it possible to train a blank neural network to look like this? Backprop obviously wouldn't work, but perhaps there is some form of directed search one could use?
I tried this with a friend last year (using 3-majority and a variety of other gates). We couldn't get backprop to work past one layer, probably because there isn't enough bit precision. The "train without backprop" issue is actually pretty difficult to accomplish, so we gave up.
Does anyone have a decent method to train neural networks without backprop? I think the information bottleneck sort-of works, but it's hard to evaluate the mutual information without a neural network.
Playing around with classic three layer feed-forward classifier networks decades ago, it became apparent that successful networks tended to converge on a sort of standard solution: layer 1 would be various linear discriminators - each node would cut the input space into two half spaces, such that any input point was either in the selected half-space or not, essentially reducing the N dimensional real-valued input to some number of bits. Layer 2 nodes tended toward AND functionality, using some subset of the layer 1 outputs to identify convex regions of the input space. Layer 3 nodes tended to OR together outputs of layer 2 to combine convex regions into arbitrary subsets of the input space. I.e. more often than not, the 2nd and 3rd layer were just doing Boolean logic.
Of course, occasionally there would be a system where this didn't fully happen, and you'd get some node/s doing some funky stuff with the linear parts of the activation function, but that doesn't negate the fact that such networks can and often do find the slice/AND/OR solutions.
Once you complete this (ignoring the first level where you actually build a NAND gate) you essentially get very much what looks like a Neural Network (since it takes 2 neurons to represent a NAND gate), n layers deep, with a lot of zeros in the weight matrices, and some storage.
Here is my question: given the input/output semantics at the assembly level, is it possible to train a blank neural network to look like this? Backprop obviously wouldn't work, but perhaps there is some form of directed search one could use?