instrumentalism vs realism: part 3

I wrote two posts in January of this year ("instrumentalism vs realism" and "instrumentalism vs realism part 2") and wrote "to be continued..." at the end of part 2. The main point of both of them was really to voice my thoughts on the Copenhagen Interpretation of quantum mechanics, to see if I could make any sense of it, and to compare it to the Many Worlds Interpretation, which has always seemed easier for me to understand.

Several new things have happened since then. I started thinking along these lines to try to come up with some material for the microtalk I gave at FreezingWoman in March 2015. I ended up deciding to avoid the dicey subject of realism vs instrumentalism, and even for the most part avoided the entire topic of quantum mechanics. Instead focusing on the question of "what is the universe made of?" and keeping to things I feel that I understand well such as relativity and some aspects of quantum field theory. By the time I finished putting it together, I realized that I had a pretty good case that something more like neutral monism is the right way to look at metaphysics rather than materialism. The idea that metaphysics is even meaningful sort of presumes realism over instrumentalism. And yet because I defined "neutral monism" in my talk as "none of the above" (metaphysical theories) I felt like I left it a little bit open that perhaps instrumentalism is true after all and we just need to give up metaphysics entirely.

After returning from Freezing Woman, I spent a month and a half expanding my 5 minute microtalk into a 15-minute video presentation, which I released on Vimeo and linked to on Facebook and Google+. A handful of my friends viewed it and gave me positive feedback, some of them resharing it, but overall it didn't get a lot of attention. Then later, I found out that someone on Youtube had downloaded the Vimeo video and uploaded it to their Youtube channel, where it did get a lot of attention. (13,467 views, 164 upvotes, and only 5 downvotes... with lots of positive comments from people, many asking if there will be a sequel!):

Materialism and Beyond: What is Our Universe Made Of?

This weekend I uploaded it to my own Youtube channel, which I had been meaning to do (apparently, hardly anyone is on Vimeo; I original chose it primarily because I don't like the idea of ads being inserted in the middle of my video). So far not much action there either, but we'll see I guess.

I can't remember when it was, but at some point this year (maybe around May?) I ran across a *really* interesting post that my adviser in graduate school, Tom Banks, made defending the Copenhagen Interpretation of quantum mechanics:

http://www.preposterousuniverse.com/blog/2011/11/16/guest-post-tom-banks-on-probability-and-quantum-mechanics/

(There's another version of it hosted by Discover magazine but the mathematical equations don't show up right there.) I was shocked that this has been online since 2011 and I somehow managed to not find it until 2015. Not only because it is written by someone I knew personally and hold in great regard, but because it basically explains almost everything I've ever wanted to understand about quantum mechanics in one shot. I often wanted to ask him about this subject, but I was always too shy to do it. I guess I felt like to him, it might be considered a waste of time. But if he could have summed it up this well in one sitting, I would have surely asked and gotten a lot of benefit out of it. Sadly, I finally find it now long after I've quit physics.

So, it took me a while to understand everything he says there. He does make a lot of simple mistakes in his explanation, which confuses things. (For example, he uses the term "independent" several times to mean "mutually exclusive", something anyone--including him--who knows anything about probability knows are two very different things.) Nevertheless, there is a core of what he's saying that turns out to be very important. At first when I read it I sensed that, but it hand't fully sunk in. Since then, I have read a lot more things, gotten into some discussions and debates with people coming from different perspectives on this (one being a mailing list I got invited to as a consequence of people liking my video), and mulled it over in my head. And gradually, it sunk in and I feel like I have now absorbed the message. And it's a really important message that I had sort of suspected before but hadn't really understood.

This week I was thinking through this stuff again and went back to read the Koopman-von Neumann (KvN) formulation of classical mechanics Wikipedia page again (for like the third time since reading my advisor's post about KvN, which I had never heard of until then). (And in connection with the mailing list I'm on, just before that reading some more about Quantum Darwinism and Zurek's existential interpretation of QM). And suddenly halfway through the week, I felt like everything clicked. After all of these years, I finally understand Copenhagen. And it's a lot more coherent than I had imagined.

This doesn't mean I have converted now to a Copenhagenist. I'm still not sure whether I prefer Copenhagen, Many Worlds, or something in between. (And almost certainly, the right answer is somewhere in between, at least compared to what Bohr's original ideas were and what Everett's original ideas were.) And while I call my advisor a Copenhagenist, I'm not even sure he uses that term. I think his view is a modern version of Copenhagen, but does include all of the insights that have been gleaned since the time of Heisenberg and Bohr.
(Although I think he denies that those new insights have significantly changed anything about the interpretation.) I've also read a bit more about consistent histories lately and decided that there are slight differences between it and Copenhagen, it's not just a clarification of Copenhagen because in some ways, it does away with the idea of quantum measurement (or makes it less central/important to the theory). I still think QBism is a form of Copenhagen, although some of its advocates seem to think it has features which distinguish it from Copenhagen.

At any rate, using the broad definition of Copenhagen which I have always used (to include modern versions of it rather than a more narrow one focusing strictly on Bohr and Heisenberg's writings), I'd like to try to sum up the new insights I've absorbed. This was my intention in writing this post, but since I've only introduced that intention and not gotten there yet, I'll start my summary in part 4.

To be continued...

instrumentalism vs realism, part 4: KvN

As I mentioned in part 3, I had never heard of the Koopman-von Neumann formulation of classical mechanics until reading Tom Banks' 2011 post about probability in quantum mechanics on cosmic variance.

But finding out about it makes so many things about quantum clear to me that were murky in the past. The main thing that's now crystal clear is this: quantum mechanics is a generalization of statistical mechanics. They aren't really two different theories, rather quantum mechanics is statistical mechanics... it's just that a central assumption of statistical mechanics had to be dropped in light of the evidence.

I had made it most of the way to understanding this when I wrote my series on Wandering Sets in 2013. In some ways, I think it's probably the best thing I've ever written on this blog, even though I think it ended up being too long, meandering, and esoteric for my friends to follow all the way through. I want to write a popular physics book at some point where I explain these ideas more clearly, with pictures and more analogies and examples. What I've learned via KvN solidifies my hunch that QM and SM are really the same theory.

I think one of the first things any student is struck with when they take their first course on quantum mechanics is how different the math is from classical mechanics or stat mech. In classical mechanics, you have lots of differential equations that come from a single important master entity called a Lagrangian, and if you want you can write this in an alternate way as something similar called a Hamiltonian. But all of the variables in the theory just stand for regular real numbers (like 2, pi, 53.8, etc.) that describe the world. In quantum mechanics, you start from the assumption that there is a complex Hilbert space of operators. And you can write down a Hamiltonian, which you're told is an analog of the Hamiltonian used in classical mechanics. The Hamiltonian seemed like a weird way of writing the Lagrangian in classical mechanics, but in quantum mechanics it takes on a more important role. But the "variables" used in the quantum Hamiltonian are not ordinary real numbers, they're operators. These operators correspond to observables (things you can observe about the world), but instead of being a single number they are more like a technique used for making measurements and getting a set of possible results out with associated probabilities. And instead of these operators acting on states in a more familiar space (like the ordinary 3-dimensional space we live in, or the phase space used in statistical mechanics), they act on states in a complex Hilbert space. Complex numbers like 5+i play an important role in this space, and yet as a student there's really no way of understanding why or what the purpose is. You're just asked to accept that if you start with these assumptions, somehow they end up predicting the results of experiments correctly where the corresponding classical predictions fail.

There were many reasons why I ended up leaning towards many worlds rather than other interpretations. I've always preferred representational realism to instrumentalism, so that was one reason. Another was locality (reading David Deutsch's 1999 paper on how quantum mechanics is entirely local as long as you assume that wave functions never collapse was the most influential piece of evidence that convinced me.) But there was a third reason.

The third reason was that whenever I had asked myself "what's the essential difference between classical mechanics and quantum mechanics?" it came down to the idea that instead of regular numbers representing a single outcome, you have operators which represent a set of possible outcomes. In other words, instead of reality being single threaded (one possibility happens at a time), it's multi-threaded. Things operate in parallel instead of in series. This especially resonated with my computing background, and my hope that one day quantum computers would be developed. I knew that it was a little more complicated than just "replace single-threaded process with multi-threaded process", but I thought it was the biggest difference between how the two theories work and what they say.

Learning about the KvN formalism hasn't completely destroyed my preference for Many Worlds, but it has obliterated my view that this is the most important difference between the theories. I now understand that this is just not true.

While I was writing my wandering set series in 2013, I discovered the phase space formalism of quantum mechanics (and discussed it a bit in that series, I believe). This was very interesting to me, and I wondered why it wasn't taught more. It demonstrates that you can write quantum mechanics in a different way, using a phase space like you use in statistical mechanics, instead of using the usual Hilbert space used in quantum mechanics. That was surprising and shocking to me. It hinted that maybe the two theories are more similar than I'd realized. But even more surprising and shocking was my discovery this year of KvN, which shows that you can write statistical mechanics... ordinary classical statistical mechanics... in an alternate formalism using a Hilbert space! What this means is that I was just totally wrong about the number/operator distinction between quantum and classical. This is not a difference in the theories, this is just a difference in how they are written down. Why was I mistaken about this for so long? Because the standard procedure for taking any classical theory and making it a quantum theory is called "canonical quantization", and the procedure says that you just take whatever variables you had in the classical theory and "promote" them to operators. It's true that this is how you can convert one theory to the other, but it's extremely misleading because it obscures the fact that what you're doing is not making it quantum but just rewriting the math in a different way. What makes it quantum is solely the set of commutation relations used!

to be continued in part 5...