September 2, 2012 @ 1:05 pm
Nancy Fulda did a great job of explaining local minima last time, didn’t she? I thought so, anyway. But maybe it only made sense to me because I spent two and a half years of a postdoc dealing with similar problems. I was building models of a biological neural network. Not the kind of artificial neural network she worked with, where the individual neurons are simple and identical, and all the complexity comes out of the way they connect to one another during the learning process. I was trying to reproduce the spiking patterns of real neurons by varying the properties of individual neurons connected into a network with a wiring diagram based on experiments, injecting colored dyes into real neurons. My bosses had been fairly successful at tuning very small networks of 2 or 3 neurons by hand this way, but with a network of 20 neurons, each of which had dozens of electrical parameters and multiple connections, that was not going to work. With such a huge parameter space, there was no way to guarantee that any solution I stumbled onto by hand was the best solution, the solution with the smallest error, or whether it was only a local minimum in the error function. We needed to search the parameter space in a more systematic and automatic fashion, and we needed to do it fast.
I am not a good mathematician, and barely a competent computer programmer, so it took me about a year to convince them of this, and another year to adapt a search algorithm into something that kind of worked, slowly. I found an expert who promised his methods would be much faster and more accurate, but I had no idea how they worked. I’ve never studied linear algebra, with the matrices and stuff, beyond reading a lot of image processing papers when I was studying vision, so even his explanations made no sense to me. I could see the results, but as to how they got there, it was magic. He could have drawn a circle on the ground in pig blood and salt, and it would have been just as good, as far as I could tell. I could at least explain to people how my crappy slow solution worked.
That was a humbling experience, but an empowering one, too. I gave up the idea that I knew what I was doing, or that I ever would know, or even that I needed to know exactly what I was doing to make progress. It’s a lesson that has come in extremely handy as a parent, as we’ll hear in today’s episode. As long as I can keep the cost of my mistakes low, and I don’t have to make the same ones over and over, I’m happy to keep making new ones.
Anyway, what did you all think?
Nancy Fulda's machine learning papers
Nancy Fulda's "Nothing this Fun Could be Good for You"
How Huizinga relates to video games
A play/game called Homo Ludens