Introduction

• Mountcastle proposed that the reason the regions look similar is that they are all doing the same thing. What makes them different is not their intrinsic function but what they are connected to. If you connect a cortical region to eyes, you get vision; if you connect the same cortical region to ears, you get hearing; (Location 446)
• Darwin knew what the algorithm was: evolution is based on random variation and natural selection. However, Darwin didn’t know where the algorithm was in the body. This was not known until the discovery of DNA many years later. Mountcastle, by contrast, didn’t know what the cortical algorithm was; he didn’t know what the principles of intelligence were. But he did know where this algorithm resided in the brain. (Location 458)
• Thoughts and experiences are always the result of a set of neurons that are active at the same time. (Location 637)
• However, over the past few decades, scientists have discovered that in many parts of the brain, including the neocortex, new synapses form and old ones disappear. Every day, many of the synapses on an individual neuron will disappear and new ones will replace them. Thus, much of learning occurs by forming new connections between neurons that were previously not connected. Forgetting happens when old or unused connections are removed entirely. (Location 648)

Neuroscience

Neuron

• This discovery led to an important question. How does the brain make predictions? One potential answer is that the brain has two types of neurons: neurons that fire when the brain is actually seeing something, and neurons that fire when the brain is predicting it will see something. (Location 681)
  • Predicting the next note in a melody, also known as sequence memory, is the simpler of the two problems, so we worked on it first. Sequence memory is used for a lot more than just learning melodies; it is also used in creating behaviors. For example, when I dry (Location 700)
  • myself off with a towel after showering, I typically follow a nearly identical pattern of movements, which is a form of sequence memory. Sequence memory is also used in language. Recognizing a spoken word is like recognizing a short melody. The word is defined by a sequence of phonemes, whereas a melody is defined by a sequence of musical intervals. (Location 701)
  • How?
    • Starting around 1990, this picture changed. Scientists discovered new types of spikes that travel along the dendrites. Before, we knew of only one type of spike: it started at the cell body and traveled along the axon to reach other cells. Now, we’d learned that there were other spikes that traveled along the dendrites. One type of dendrite spike begins when a group of twenty or so synapses situated next to each other on a dendrite branch receive input at the same time. Once a dendrite spike is activated, it travels along the dendrite until it reaches the cell body. When it gets there, it raises the voltage of the cell, but not enough to make the neuron spike. It is like the dendrite spike is teasing the neuron—it is almost strong enough to make the neuron active, but not quite. (Location 737)
    • The big insight I had was that dendrite spikes are predictions. A dendrite spike occurs when a set of synapses close to each other on a distal dendrite get input at the same time, and it means that the neuron has recognized a pattern of activity in some other neurons. When the pattern of activity is detected, it creates a dendrite spike, which raises the voltage at the cell body, putting the cell into what we call a predictive state. The neuron is then primed to spike. It is similar to how a runner who hears “Ready, set…” is primed to start running. If a neuron in a predictive state subsequently gets enough proximal input to create an action potential spike, then the cell spikes a little bit sooner than it would have if the neuron was not in a predictive state. (Location 747)
  • The brain needs to know two things: what object it is touching (in this case the coffee cup) and where my finger will be on the cup after my finger moves. (Location 792)
  • What matters is the location of my finger relative to the cup. (Location 795)
• This observation means there must be neurons in the neocortex that represent the location of my finger in a reference frame that is attached to the cup. The movement-related signal we had been searching for, the signal we needed to predict the next input, was “location on the object.” (Location 796)
• Vision, I realized, is doing the same thing as touch. Patches of retina are analogous to patches of skin. Each patch of your retina sees only a small part of an entire object, in the same way that each patch of your skin touches only a small part of an object. The brain doesn’t process a picture; it starts with a picture on the back of the eye but then breaks it up into hundreds of pieces. It then assigns each piece to a location relative to the object being observed. (Location 811)

Reference frames

• Up to that point, most neuroscientists, including me, thought that the neocortex primarily processed sensory input. What I realized that day is that we need to think of the neocortex as primarily processing reference frames. Most of the circuitry is there to create reference frames and track locations. Sensory input is of course essential. As I will explain in coming chapters, the brain builds models of the world by associating sensory input with locations in reference frames. (Location 818)
• What does the brain gain from having them? First, a reference frame allows the brain to learn the structure of something. A coffee cup is a thing because it is composed of a set of features and surfaces arranged relative to each other in space. Similarly, a face is a nose, eyes, and mouth arranged in relative positions. (Location 821)
• Second, by defining an object using a reference frame, the brain can manipulate the entire object at once. For example, a car has many features arranged relative to each other. Once we learn a car, we can imagine what it looks like from different points of view or if it were stretched in one dimension. To accomplish these feats, the brain only has to rotate or stretch the reference frame and all the features of the car rotate and stretch with it. (Location 824)
• Third, a reference frame is needed to plan and create movements. Say my finger is touching the front of my phone and I want to press the power button at the top. (Location 827)
• Why, then, do we not perceive objects as being in the eye? If the chairs, desks, and courthouse are imaged next to each other on my retina, how is it that I perceive them to be at different distances and different locations? Similarly, if I hear a car approaching, why do I perceive the car as one hundred feet away to my right and not in my ear, where the sound actually is? This simple observation, that we perceive objects as being somewhere—not in our eyes and ears, but at some location out in the world—tells us that the brain must have neurons whose activity represents the location of every object that we perceive. (Location 892)
• They discovered what are now called place cells: neurons that fire every time the rat is in a particular location in a particular environment. A place cell is like a “you are here” marker on a map. As the rat moves, different place cells become active in each new location. If the rat returns to a location where it was before, (Location 923)
• The details of how place cells and grid cells work are complicated and still not completely understood, but you can think of them as creating (Location 930)
• Grid cells and place cells in the old brain mostly track the location of one thing: the body. They know where the body is in its current environment. The neocortex, on the other hand, has about 150,000 copies of this circuit, one per cortical column. Therefore, the neocortex tracks thousands of locations simultaneously. (Location 966)