Home

Response geometry explains adversarial robustness

2021-06-24T00:00:00-07:00

Defending against adversarial attacks is critical for modern AI applications. Here I describe how we can use response geometry to predict a neuron’s susceptibility to attacks.

This writeup builds on my previous post about using the curvature of a neuron’s response surface to understand its behavior. If that idea is unfamiliar to you, then I recommend reading that post before this one.

Adversarial examples

Adversarial examples are a worst-case demonstration of artificial neural networks’ (ANNs) inability to gracefully cope with shifts or distortions of their inputs. To construct one, an adversary must modify, or perturb, the input in a very small but specific way such that the output of the ANN changes significantly. The figure below is adapted from an iconic example provided in one of the first papers demonstrating adversarial attacks on ANNs, authored by Dr. Christian Szegedy and colleagues. The original input images are in the left column and they belong to the categories “mantis” and “dog.” The images in the middle column are adversarial perturbations that can be added to the original inputs to produce adversarial images in the right column. All of the adversarial images are labeled as “ostrich” by the network. The perturbations have to be magnified to allow you to see what they look like because their original scale is at the lower bound of what a typical computer image can convey.^† Importantly, these attacks do not just apply to images. They have been demonstrated to work pretty much anywhere that ANNs are useful, and they almost always seem innocuous or are imperceptible to a person. Successful demonstrations include tricking text analyzers that are used for product review summaries, social media feeds, and search result rankings; tricking smart home devices to execute a command; and tricking a self-driving car to mistake a stop sign for a yield sign.

fig. 1 Natural images (left column) can be modified by small perturbations (middle column, magnified) to produce adversarial images (right column). The adversarial images are confidently misclassified by the neural network.

To be more precise, adversarial perturbations are modifications of the input of a network that are small in magnitude, but cause a big enough shift in its output to change its behavior. While it is not a hard requirement, they are also often semantically unrelated to the new output of the network – for example the “ostrich” perturbation is not recognizable as an ostrich. These two criteria, small size and uninterpretable semantic quality, are what makes them so interesting to researchers. There are adversarial perturbations that could trick a classifier in an uninteresting way, such as adding a horizontal stroke to a “0” to trick a classifier into calling it an “8”, or putting wheels on a canoe to trick the classifier into thinking it is a bicycle (or is it a bicycle disguised as a canoe?). If these obvious attacks were what was demonstrated by Dr. Szegedy and colleagues then few other researchers would have paid attention. Unfortunately, we do not have a good way to measure “semantic quality,” so the research community can only use the average size of the perturbations to quantify adversarial robustness, which indicates how much a network can handle adversarial attacks. However, they do often also show examples of perturbed inputs to allow readers to judge for themselves whether the semantic content is reasonable given the original and new network outputs. Now that we have defined the adversarial attacks and robustness we can better understand how such attacks are created.

Following gradients

There are many methods for constructing adversarial perturbations and they can roughly be categorized by how much access they have to the network they are attacking. All of the methods, however, seek the same goal: to maximally change a function’s output given a minimal change in its input. To illustrate how attackers achieve this, let’s consider an example where the function input is the position on a map and the function output is the elevation at that position. We can summarize such a function with a topographic map. Topographic maps use iso-elevation lines to indicate altitude, which means your elevation is constant if you walk along the line. This is illustrated in the following figure:

fig. 2 Topographic maps indicate altitude with iso-elevation lines.

Following the goal we defined earlier, we want to determine how to minimally change our position and maximally increase our elevation. There are two things to consider to solve this: one is where to start and the second is what direction to walk from there. In the case of adversarial examples the starting position is fixed to be the original input, like the dog or mantis above. In the figure below I indicated a starting position with a smiley face. The optimal direction to climb at any given position is orthogonal to the iso-elevation lines. This orthogonal direction aligns with the maximum gradient of the function, which is a term in math that has an intuitive mapping on to gradients in common language, such as a temperature gradient or the grade of a hill. That is to say that the gradient indicates how much the function’s output changes for a given minimal directional change in the function’s input.

fig. 3 A path that is orthogonal to the iso-elevation lines on a topographic map is the fastest way to change altitude.

All methods for finding adversarial examples are derived from this same guiding principle, which is to use gradients (or approximations thereof) to determine what direction to go. I’ll explain in the wrap-up that this is considered a local perspective on finding an optimal path. In my previous post on iso-response contours, I explained how images can be thought of as arrows reaching to points in a high-dimensional space. Along this line of reasoning, an adversarial perturbation can be drawn as an arrow extending from the original input with some direction and magnitude. This is the same concept that we are using in figure 3: each red arrow starts at some position and ends at some other position. And just like each point on that map has an analogous location in the world (in this case Hawaii), the points in our neuron response figures have analogous images. To determine the input perturbation that maximally changes a function’s output, we choose arrow directions that are orthogonal to the function’s iso-response contours. Usually that function is an entire neural network, but the explanation also applies to single neurons.

Neuron robustness

Scientists use a large variety of different types of models to describe neurons in both neuroscience as well as artificial intelligence research. One way to organize the models is by their level of abstraction, which describes the tradeoff between biological accuracy and computational complexity. Indeed, one major goal of computational neuroscience is to discover the level of abstraction that has minimal complexity while still emulating the behavior of biological organisms. A neuron’s response surface geometry can help us quantify differences in behavior, such as a model’s selectivity, invariance, or robustness. It can also help us understand complexity, in that neurons with straight iso-response contours are minimally complex and so any deviation from straight contours will indicate more complex (or conversely, less linear) processing. The above climbing example maps well to how we will use iso-response contours to understand neuron robustness. Let’s consider two models: one that produces straight iso-response contours and one that produces outward (i.e. away from the origin) bent iso-response contours. The next figure shows the iso-response contours for these models as well as the arrows indicating optimal adversarial perturbation directions for increasing their response with minimal perturbation size.

fig. 4 Two neurons are simulated and their iso-response contours are shown. The numbers above the response indicate the neuron’s activation in arbitrary units. The little gray arrows indicate the maximal gradient direction for increasing the neuron’s activation. The longer colored arrow indicates an optimal path taken from the point indicated by the black dot.

The longer colored arrows in the above figure indicate an optimal adversarial perturbation from a given starting point. The \(\Phi_{k}\) variable indicates the direction of the neuron’s maximally exciting image, or MEI. As the name implies, this image tells us what feature the neuron is most excited about in the world. In biological and artificial neural networks, the MEI features vary in complexity as one ascends a hierarchy of processing stages, from simple edges at the lowest levels to faces and complex objects at the highest levels. Both neuroscientists and artificial intelligence researchers often use a neuron’s MEI to label that neuron, for example as a “vertical edge detector” or a “dog detector.” All of this matters when we think about an adversarial attack on the neuron.

For both neurons, as we take steps in the adversarial direction, the output will look more and more like the neuron’s MEI. However, this will happen faster for the neuron with the curved response contours, to the point where the input eventually just ends up looking exactly like the MEI. This means that the optimal perturbation direction for neurons with curved response contours is more aligned with their MEIs than for neurons with straight contours. Thus, if we assume that neurons are interested in semantically relevant features, then we hypothesize that the resulting adversarial perturbations are going to be more semantically relevant if the neurons have curved response contours. And as I discussed earlier, if we modify a picture of the number “0” to look more like an “8,” then we should not be too surprised if a neuron that has an MEI that looks like an “8” gets more excited.

It turns out that the curved contours can result in larger overall perturbation magnitudes when we have a network of these neurons as well. This is because an ensemble of neurons with bent contours will constrain the adversary, thus limiting its perturbation options. The next figure demonstrates the idea, where we show all possible pixel values that will result in an increased activation for neuron \(k\).

fig. 5 Inputs from the shaded region will have increased activation when compared to inputs not in the shaded region. By restricting the shaded region, bent iso-response contours constraint the space of possible inputs to increase a neuron’s activation.

The dotted box represents the range of allowable pixel values for input images. The yellow shaded area represents all possible inputs that would increase the activation of neuron \(k\) if it had straight iso-response contours, relative to the non-shaded area. Conversely, the green shaded area represents all possible inputs that would increase the activation of neuron \(k\) if it had bent iso-response contours. As you can see when we overlay the two shapes, any amount of bending will reduce the number of possible inputs that have an increased response. This means an adversary will have fewer options to choose from if it wishes to increase the output of neuron \(k\).

Wrap up

At a high level, adversarial examples are one of many pieces of evidence supporting the idea that ANNs don’t perceive the world like we do. Given that there are other mismatches, exclusively solving adversarial susceptibility in ANNs is unlikely to align human and machine perception. However, adversarial examples still serve as a valuable test of ideas in neural computation: if we have an idea to explain perception, then our model of the idea should be robust to adversarial attacks.

In this post I explained how we can use the iso-response surface of a neuron to determine adversarial robustness. This might be intuitive for you if you think about a counter-example: what should we do if we wanted to change an input to minimally change the output? The answer is to follow the iso-response contours. Therefore, moving perpendicular to the iso-response contours will have the opposite effect. Following this intuition, we hypothesized here that the outward bent contours will result in larger adversarial images that look more semantically meaningful. I go into more detail in my paper, where I present supporting experimental evidence using multi-layer neural networks.

One detail that would be remiss of me to ignore is the spacing between the iso-response contours. You might have noticed that the bent and straight contours in figure 4 were (approximately) equally spaced. The optimal adversarial perturbation direction at any given point is indeed orthogonal to these contours. However, if the spacing is not guaranteed to be equal then there may be a suboptimal direction to take at that moment that ultimately achieves the goal with fewer steps. If you wish to climb a mountain in as few steps as possible, for example, you might need to walk in a sub-optimal direction for a little while to get to a cliff face that will let you climb at the fastest rate. This distinction is subtle, but important when we are trying to build defenses against adversarial attacks. The difference I’m describing is between a local and a global optimization process. Importantly, though, a global perspective, such as knowing that there is a cliff on the other side of some valley, is rare when attacking neural networks because of the high dimensionality of the inputs.

Thus, iso-response contour curvature provides part of the story of adversarial susceptibility, and iso-response contour spacing provides another part. No one has provided a complete picture of adversarial vulnerability using iso-response geometry, and so we do not know if there are more aspects to consider. However, there has been some excellent progress recently. Hopefully this post increased your understanding and demonstrated yet another way that the geometry of a neuron’s response surface can teach us about how that neuron processes its inputs.

Footnotes

† Computer images are composed of pixels that each take on one of 256 possible values. This is the same as saying that they are 8-bit images. If one were to rescale the pixel values to be between 0 and 1, then the average pixel perturbation size for the examples from Szegedy et al. is 0.006. This corresponds to changing a single pixel by about 2, or in reality changing many pixels by less than 1. As such, adversarial perturbations are often smaller than the available bit-depth of current displays.

Understanding neurons using response geometry

2021-05-18T00:00:00-07:00

Scientists have long summarized neurons in terms of the relationship between their inputs and outputs. Here I describe a technique that allows us to go beyond previous approaches to succinctly describe important and complex neural behavior.

Neuron selectivity and invariance

An important computational goal of the early visual pathway, from the retina to the primary visual cortex (V1), is to produce a representation of visual features that are useful for downstream tasks, like navigating the world and identifying objects. In order for a V1 neuron’s output to be decoded by downstream brain areas, it must reliably modulate its response with respect to certain variations in the visual world and remain constant for others. For example, let’s suppose we live in a binary edge world, where all scenes are made up of combinations of oriented edges. In such a world, images of vertical edges would occur when a column of 1s immediately precedes a column of 0s, or vice versa. To encode such a world, our brain might want to have a “vertical edge detector” neuron that signals the presence of any vertical edges. This means we want our neuron to be selective for the presence of a vertical edge, as opposed to other orientations, but invariant to the phase of the edge, i.e. whether it is 0s followed by 1s or the other way around. In neuroscience, such a neuron is called a complex cell and is a common basic cell type.

It was precisely the discovery of analogous neuron types in the visual cortex of cats that led scientists to hypothesize that the brain is wired up to perform computations on signals coming from the world. This discovery occured in the late 1950s and created a revolution in neuroscience research. About twenty years later a scientist named Dr. Kunihiko Fukushima proposed a computer model to emulate the discovered behavior. His invention, called the neocognitron, was one of the earliest ancestors of modern day deep neural networks. One of the defining elements of the neocognitron were “C-cells”, which were designed to be selective for features in the world, but invariant to their position. In modern artificial neural networks, we often label neurons by what they are most selective for, such as snout or car detectors, and we explicitly engineer components that create invariance, such as pooling. Today, scientists have a variety of methods to measure and quantify selectivity and invariance in both biological as well as artificial neurons. These measurements are extremely important for developing an understanding of how we process information, and are often used to categorize neurons into groups. This blog post aims to explain one such method, which is primarily used in neuroscience and relies on characterizing the geometry of a neuron’s response surface.

The neural response surface

First, an obvious statement: neurons receive signals as input and produce signals as output. Let’s assume that some target neuron receives input from 100 other neurons, and produces a single output that might be read by any number of downstream neurons. Going forward, we will define the target neuron as the neuron that we are currently interested in studying, and we will indicate it mathematically with the index \(k\). Let’s assume that the target neuron will produce an output for any combination of signals coming from the 100 input neurons. Or, put another way, the target neuron is a function that maps a 100-dimensional input to a scalar (i.e. a single number, which would then be single-dimensional) output. The neural response surface is a description of the neuron’s output for all possible combinations of inputs.

It is impossibly difficult to measure the entire response surface of a neuron, even in the case of modern artificial neural networks. However, if we assume that the response surface is relatively smooth and regular, then we can still learn a lot about the neuron by looking at cross sections, which are much more tractable to estimate. This presents our next challenge of choosing which cross sections to take. Just like cutting an onion in half, one type of cross section will reveal a different structure from another one.

fig. 1 Taking cross sections of complicated surfaces can reveal structure, but the structure looks different depending on the orientation of the cross section.

Instead of input from other neurons, let’s consider a neuron that receives images as input and produces a scalar output. Individual input images can also be thought of as points in a high dimensional space. For example, a 10-by-10 image thumbnail has 10*10=100 pixels. Therefore, we can equivalently think of the image as a point in a 100-dimensional space. Each dimension can be represented as an axis, where the position along the axis is the value of the pixel. If individual pixels are allowed to be numbers between 0 and 1 (like in a grayscale image, where 0 is black, 0.5 is gray, and 1 is white), then all allowable images exist in a 100-dimensional cube. Just to really drive this idea home – every single point in the 100-dimensional cube is also an image. The next figure has an illustration of a 3D image space with such a 2D cross section drawn in light gray. If we draw arrows starting from a single origin such that the tips reach two image points, then we can compute the angle between the arrows to determine how similar those images are. If two image arrows are perpendicular, then they are maximally dissimilar from each other.

fig. 2 Images are high-dimensional points. In this illustration, each yellow axis represents the value of a pixel for a three-pixel image. The gray sheet represents a cross section, or plane. The purple and blue arrows define the cross section. The red arrow points in the remaining direction, which is perpendicular to the cross section.

Here’s a roadmap of what is to come: To explore the response geometry of our target neuron, we will first take a two-dimensional cross section of the high-dimensional input space. Next we will discretize the cross section, that is to say we will sample the cross section with a tiling of points evenly spaced from the minimum allowable pixel value to the maximum. Then we will measure the output of the neuron for each of the images selected from the discretization process. Finally, we will measure the curvature of the response surface to quantify selectivity and invariance of the neuron.

Choosing a cross section

We are interested in the behavior of the target neuron for images that it cares about, or ones it would see in its natural environment. As such, we shouldn’t use just any old cross section to analyze a neuron, but instead cross sections that contain images (i.e. points) that are relevant for that neuron. Cross sections are defined by two vectors, which are mathematical objects that point in a particular direction in space and are drawn as arrows. If we set at least one of the vectors to be pointing in the direction of an image that the neuron cares about, then we know that the cross section at least partially contains interesting inputs for the neuron. The next figure illustrates how each point in the grid can be reshaped to be visualized as an image. A few of the points around the edges are displayed as images. Notice how these images have some structure to them and are systematic in how they look. For any given cross section, all of the images are coplanar, which is to say that they can all be described as a linear combination of two exemplar images. This idea of linearity is also illustrated in the figure below in that all of the corner images can be constructed by adding two of the center images.

fig. 3 We discretize a cross section of the input space into a two-dimensional grid and collect the points to define a dataset of images. The black arrows are perpendicular vectors that define the cross section. The origin for all of the cross sections we consider here will be a zero-contrast image, in other words a uniform gray image.

In addition to constraining our input space to be coplanar, we are also going to constrain our output space to simplify the analysis. Specifically, we are going to look at the curvature of the iso-response contours of the neuron. These contours are connected points that all produce the same (hense “iso”) output from the neuron and they all exist on the cross section. To see the iso-response contours, we will compute the neuron’s activity (i.e. its output) for all of the cross section images and then recolor the points according to the activity.

As I said before, we want to pick a cross section that is relevant for our neuron. To do this we define one of the two cross section axes as the neuron’s maximally exciting image (MEI), which is an image that was optimized to look like the feature in the world that is most interesting to the neuron. There are a lot of ways to find a neuron’s MEI in neuroscience and in artificial neural network research, but for now let’s leave aside how the image was produced and just assume it is representative of what the neuron likes most. Going forward, we will represent the MEI with the symbol \(\Phi_{k}\), where the \(k\) tells us it is the MEI for the \(k\)th neuron in our assembly. For now the other axis can be any random orthogonal (i.e. perpendicular) image, but we will choose more specific orthogonal images later.

fig. 4 The horizontal axis for an image cross section is always the target neuron’s maximally exciting image, or MEI.

Plotting the contours

We can now color the points according to the neuron’s activity, \(a_{k}\). Visualizing the iso-response contours is easy, we just collect the output values into a fixed number of bins and then the bin boundaries will lie along iso-response contours. Here’s what it would look like for a simple linear neuron:

fig. 5 The iso-response contours for a linear neuron are straight and orthogonal to the neuron’s MEI, \(\Phi_{k}\). In this image, \(\Phi_{j}\) indicates the MEI for some other neuron and \(\nu\) indicates the chosen orthogonal axis. For a linear neuron, the iso-response contours will be straight regardless of the chosen direction for \(\nu\). The color indicates the output activity of neuron \(k\), where blue is low and yellow is high.

Notice how the lines are straight and orthogonal to \(\Phi_{k}\). This will be true for any linear system, and it means that the neuron is insensitive to orthogonal perturbations away from it’s preferred input. What happens if we apply a pointwise nonlinearity to our neuron’s output? A pointwise nonlinearity is a function that maps a single scalar to another scalar, and therefore a pointwise nonlinear neuron does not interact directly with its neighbors in a given layer. This is exactly the type of operation that is used in almost all standard deep neural networks: a linear operation followed by a pointwise nonlinearity. For example, in the next figure we include the Rectified Linear Unit (ReLU) nonlinearity, which sets all values below some threshold to zero and then acts as an identity mapping (i.e. output equals input) for all values above that same threshold.

fig. 6 The iso-response contours for a pointwise nonlinear neuron are still straight & orthogonal to the neuron’s MEI. The nonlinearity can only change the spacing of the lines, or set values to zero.

We can see that the iso-response contours are still straight. This is because the nonlinearity is applied after the linear operation, and so the nonlinearity can only change the spacing between the lines. This is true for all types of pointwise nonlinearities, which we demonstrate by also including a sigmoid nonlinearity.

Straight vs. curved contours

Biological neurons tend to have more complicated iso-response contours, as well as some more sophisticated artificial neural network neurons. Specifically, the contours tend to have curvature in the sense that they bend towards or away from the origin. Since linear neurons produce straight contours, it is reasonable to think about curvature as indicating a deviation from linearity – more curvy contours indicate less linear functions. Additionally, let’s differentiate between outward curvature, which means it’s bending away from the origin, and inward curvature, which means it is bending towards the origin. One reason curvature is significant is because it tells us about what variations in the world the neuron is selective for and invariant against. To understand that better, let’s consider a few different types of coplanar perturbations that we can make from an input image. Here is a list of some options for a starting image along the horizontal axis:

Right, toward the neuron’s MEI, then the target neuron’s activity will grow
Left, away from the neuron’s MEI, then the target neuron’s activity will shrink
Up, orthogonal to the neuron’s MEI, then the target neuron’s activity will change differently for each curvature type
Along the iso-response contour, then the target neuron’s activity will not change

fig. 7 Visualizations of four out of many possible perturbation directions. For each direction, assume we start with the image depicted by the square, then perturb to the pentagon, then again to the circle. Straight contours are indicated in shades of purple, outward bending in shades of red, and inward bending in shades of blue. Darker colors indicate higher neural activation.

For linear neuron & pointwise nonlinear neurons, the last two directions are equal. But they are not equal for neurons with bent iso-response contours. What does this tell us? Each of the perturbation options above can be thought of as an experimental condition. We will look at the results from two such experiments to visualize the difference between straight and curved contours.

Experiment 1: Noise perturbations

To start we will look at the “up” perturbation condition for straight and outward bending contours.

fig. 8 The “up” perturbation direction reveals an important difference between neurons with straight contours and neurons with outward curved contours. The orthogonal, \(\nu_{\text{noise}}\) axis is chosen to be a noisy image. The line plots in the middle indicate the response of the neuron (vertical axis) for each image type (horizontal axis). Each perturbation point is displayed on the right.

To start, we have two neurons with identical MEIs, but different curvatures. The vertical axis that helps define our cross section, \(\nu_{\text{noise}}\), is a random noise image. This means that an “up” perturbation will result in the MEI looking more and more noisy. Again let’s consider the pointwise nonlinear neuron. What happens to the output if we take a fixed point on the \(\Phi_{k}\) axis and then perturb orthogonally in the up direction? Nothing; as expected, the response is flat. However, the neuron with outward curvature decreases its output, \(a_{k}\), as we perturb “up” in the \(\nu_{\text{noise}}\) direction. We call this response attenuation. This means the neuron is selective against orthogonal perturbations away from its preferred stimulus. In other words, outward bending contours reveal that the target neuron’s output contains more information about how close the stimulus is to its MEI than straight contours.

This is important if we consider what the “MEI” means. In both studies of artificial neural networks and biology, scientists use MEIs or something similar to label neurons and group them together. For example, if the MEI is a vertical edge, then we would label the neuron a “vertical edge detector”. As we perturb an image away from the MEI axis, it will look less like the MEI and more like the image \(\nu_{\text{noise}}\). For linear and pointwise nonlinear neurons, we can perturb images along any direction that is orthogonal to the neuron’s preferred stimulus as far as we want and the response will not change. However, for neurons with outward facing iso-response contour curvature, orthogonal perturbations will result in response attenuation. Therefore, the neuron outputs have a higher degree of correspondence to their preferred stimulus label, and thus their output carries more relevant information about the input. Another way to say all of this is to say that the outward curvature indicates increased selectivity, where the amount of selectivity is proportional to the amount of curvature.

Experiment 2: Phase perturbations

As another example of the utility of this analysis, let’s look at perturbations that are along the iso-response contour. Again both neurons will have the same MEI, but different curvature. In this case, let’s compare the neuron with flat iso-response contours to the one with inward curved iso-response contours. This time we will define the vertical axis to be a phase shifted version of the neurons’ MEI. There are lines that are defined by the transition between light and dark lobes in the image. If we shift the phase of the edge, then each line will continuously move to the left or right, depending on what direction we shift the phase. This is a continuous and non-linear process.

If we look at the images that lie along evenly spaced intervals of the contours we can see a clear difference between the two neuron types. The perturbations for the top neuron have similar behavior as in the first experiment. With increasing perturbations, the image will look less like \(\Phi_{k}\) and more like \(\nu_{\text{phase}}\). However, this transition is linear, which means you will see both phases superimposed on top of each other. It is not actually shifting the phase of the edge. The perturbations for the bottom neuron, on the other hand, result in a continuous phase shift. By definition, since these images lie along the iso-response contour, the output of the neuron is equal for all of them. This means that the bottom cell has phase invariance, while the top cell does not.

fig. 9 The “contour” perturbation direction reveals invariance for neurons with inward-bending iso-response contours. Perturbing along the top neuron’s iso-response contour will make the image look less like a linear combination of \(\Phi_{k}\) and \(\nu_{\text{phase}}\) instead of a phase shift. However, perturbing along the bottom neuron’s contour results in a phase shift. This is easier to see the farther along the contour we perturb, so we used larger perturbations here than those shown in figure 7.

Wrap up

These two experiments illustrate that iso-response curvature can indicate important information about how a neuron processes its input. The curvature analysis gives us a succinct way to understand the nonlinear function a neuron is performing, and allows us to know what features, or changes in features, that a neuron does and does not care about.

In summary, the big takeaways are:

linear and pointwise nonlinear neurons have flat iso-response contours, regardless of the nonlinearity used
flat iso-response contours indicate linearity and that the neuron’s response will not change if inputs are perturbed orthogonally from their preferred stimulus
outward-curved contours indicate selectivity, and the amount of curvature will tell us how selective the neuron is
inward-curved contours indicate invariance, which is a property of biological complex cells

There is much more that we can learn by looking at the curvature of neuron iso-response surfaces. In my next post, I use them to explain a concept in machine learning called adversarial robustness. If you are interested in digging into more details, check out my paper on the subject, as well as the work of James Golden et al. (PDF).