Psychological Network Analysis is Not a Predictive Method

vision1

The Amazing Capacity of Vision

Most of the time, I talk with other clinical psychologists. However, every once in a while, I get the chance to talk to a vision scientist. If you get the chance to do the same, I highly recommend it — you might be surprised what you hear.

Vision scientists tend not to see vision as something “real”. This can be jarring for the rest of us, but hear me out — I think they make some good points. A vision scientist will explain that the information entering your eyeballs daily is absurdly complex — somewhere around the magnitude of 10^14 photons per second — and what we actually see is surprisingly unrepresentative of the raw information.

Instead, what we see has a lot to do with heuristics that are very important to human survival. We are good at seeing edges and movement. We are good at parsing edges into objects. We are even quite good at gauging the depth of these objects relative to one another. Amazingly, we are pretty good at parsing these objects in terms of their threat, reward, and even their social relationships to one another– all before we’ve even realized that we’ve seen them.

What does this anecdote have to do with anything? Well, vision is an amazing capacity for understanding. The amount of data you have taken in through your eyes today is extraordinary, yet you understood it seamlessly. If you’re even remotely interested in the acquisition or the production of knowledge, you should care very much how you managed to do this.

Vision is not the truth — vision is better than the truth. It is an understandable approximation of the truth, and remarkably unlike the raw truth, it is actually useful to us.

Predictive and Explanatory Statistics

Most of the time, I talk with other clinical psychologists. However, every once in a while, I get the chance to talk to a statistician or data scientist. If you get the chance to do the same, I highly recommend it — you might be surprised what you hear.

Statisticians tend not to see statistics as something “real”. This can be jarring for the rest of us who work in applied settings. Statisticians know — have always known — that statistics are just something humans made up to roughly force complex datasets into our very limited human brains.

According to Yarkoni & Westfall, statistical methods fall roughly into two camps: predictive and explanatory. Predictive methods are good at turning inputs into powerful predictions, but they are often opaque to human understanding. On the other hand, explanatory methods enhance human understanding, but often possess relatively limited predictive power.

Psychological Networks are Explanatory, not Predictive

I have thought long and hard about the issue, and feel quite certain that psychological networks fall firmly in the explanatory camp. Why do I think so? There are a few reasons:

  • When I talk to experts in predictive methods (i.e., machine learning), they generally do not consider network analysis to be a predictive method
  • When compared to other statistical benchmarks in the literature, psychological networks aren’t great predictors
  • Psychological networks are highly interpretable. Most of the parameters estimated are of the same type. This makes them very easy to understand, and also very inflexible. This is a hallmark of a good explanatory model (and a limited predictive model).
  • When networks are predictive in other fields, it’s usually because they have a near 1:1 measurement connection with reality (for example, physical electrical networks). In these cases, networks are both explanatory and predictive, not just predictive.
  • Psychological networks were devised as a way for scientists to understand psychological phenomena.
  • Researchers don’t use psychological network analysis like a predictive method. Allow me to explain what I mean. When a machine learner wants to predict an outcome, they typically engage in a workflow of trying out 5-10 different types of models, cross-validating, and deciding which model has the best out-of-sample prediction. They have not a care in the world for which model is most accurate to the real-world phenomenon, as long as it is predictive. In contrast, researchers who use network analysis do so specifically because they believe that the model is a good way of understanding the phenomenon they study.
  • Researchers who use psychological network analyses care a lot about ‘getting it right’. They worry about assumptions that might cause misinterpretation. This assumes of course that interpretation is one of the primary goals.

Good Explanatory Statistics are Overly Simplistic

Just like good vision, explanatory statistics are essentially heuristics that help translate the complexity of the world into something that humans can understand and care about.

Just like vision, explanatory statistics are not the truth. Just like vision, good statistics are better than the truth — they are an understandable approximation of the truth.

Something that most good statisticians tend to appreciate is that over-simplicity is a really great thing. Statisticians appreciate this because they understand that all statistics are an oversimplification — and this means that if you care about explanation, a fluently understandable model that explains 50% of the variance is much better than a mumbo-jumbo-model that explains 55% (for explanation).

To the Point: Network Models

I think that psychometric networks are a wonderful innovation that will play a huge role in advancing psychology, particularly clinical psychology.

There are lots of others who agree with me about psychological networks — perhaps yourself– but I’m afraid that they agree with me for the wrong reasons. They might think that they are going to use EMA data to make networks that precisely predict outcomes. They might think that as soon as we are able to deal with X assumption or Y assumption, we will have a model that matches reality. Even more problematic, they might think that network models are the best way to make treatment decisions or to predict important clinical outcomes.

I believe that many of these thoughts are based in the assumption that we can have our cake and eat it too: have an explanatory model that also provides the best predictions. I’m doubtful.

Psychological networks are wonderful, but I think they are wonderful mostly for their explanatory power, not their predictive power.

Psychological networks are just not good predictive models. In the vast majority of cases, they are just far, far, too simplistic and rigid to every compete with some of the powerhouses in modern machine learning — random forests, neural nets, bagging, boosting, etc. (with a million more well on the way).

But network models are far superior to all of these predictive methods in one very important way — they can make pretty OK predictions, and we can understand why they are making these pretty OK predictions. This makes them extremely valuable. Because many of us scientists care quite a bit about prediction — but we also care quite a bit about explanation.

Far after the fanciest new predictive methods of today are considered outdated, linear regression will continue to be used by those who care about explanation. I hope that network analysis can be there alongside it.

Using Networks for Explanation

I had a very jarring recent experience when a family member asked me about one of my projects. They asked me what the main takeaways were, and I forgot. After hours of building complex models, computing centrality statistics, and so forth, I literally forgot the results I came up with. And if I’m forgetting, what is the chance that someone else will read my article, remember better than I did, and actually use that knowledge to produce some real-world good? Unfortunately for me, not good. I focused on complexity, and I focused on accuracy, but I forgot to focus on explanation.

Network models are heuristics. Network models are explanatory. And explanatory heuristics are only useful when they are easily understood and applied.

Something science journalists tend to realize but actual scientists tend to forget is that humans just can’t deal well with complexity. We don’t have the time to understand everything. We don’t have the energy to understand everything. In fact, in my experience, when our models become complex enough, even we ourselves don’t take the time to really understand them.

I want to conclude with two suggestions:

  1. If you care mostly about predicting things, consider branching out from network models. Seriously. There are much better predictive models available. Let a computer crunch your data and classify your decision for you. And if you absolutely have to use network models, at least settle on 1 outcome you care about, compare with non-network approaches, and take some super basic steps such as cross-validation.
  2. If you care mostly about explaining things, reevaluate the methods you’re using to explain. How you’re writing. Complexity can be valuable, but humans have limits. If you’re going to explain, it’s worth first making sure that there is someone who cares enough to take the time to understand.

 

 

2 thoughts on “Psychological Network Analysis is Not a Predictive Method”

  1. Interesting thoughts. I partially agree I guess. If you want to predict something, you can now just throw stuff in a neural network and get amazing predictive accuracy without understanding a single thing about the system. Most network models have been developed with prediction in mind though. Recommender systems, for example, may rely on Ising models to predict which movies you like given movies you don’t like, and state-space models (pretty much VAR models) are literally used in rocket science to predict where a rocket is, as well as to predict stock markets etcetera. Network models also work very well in minimizing prediction error, especially when cross-validation is used to train, e.g., the regularization parameter. These are properties I plan to use in my upcoming research on adaptive systems (http://sachaepskamp.com/files/VENIproposal.pdf).

    Optimizing prediction error in cross-validation datasets besides though, there is a bigger argument and discussion here: the camps of data science on the one hand and complexity science on the other hand. These are strongly opposing views, even though they are also intertwined. Data science is all about optimizing prediction error using whatever machine learning algorithms we can think of. It will allow us strong prediction accuracy when everything behaves as we expect it to do. Complexity science, on the other hand, is all about using *theory* to model phenomena. Only when a phenomenon is truly understood can we really aim to predict the unpredictable events. Data science might be right often, but when it is wrong it is gloriously wrong. Take for instance the Trump election, which all polling stations predicted would happen with <10% chance, except for fivethirtyeight, who use complexity models in their algorithms and assigned 30% chance (I highly recommend reading "the signal and the noise" btw). Another example I like is the MRT system in Singapore, which is highly complex. Only by modeling such a system using, e.g., agent-based models, can you predict that a breakdown in a station on one side of the country might lead to long queues on a completely different side of the country.

    I feel this is the main goal of the network perspective, to grasp the complexity of human behavior. If we have good an accurate models for that, we main hope to predict the unpredictable, as well as to predict the consequences of interventions etcetera. We are not there yet of course, and working on such theoretical models is an important next step.

    Like

    1. Thanks for commenting, I really value having multiple perspectives on this!

      I like the distinction you make between machine learning and complexity science. I see this as a very similar idea to the distinction between prediction and explanation, although “complexity science” probably has a much more nicer ring to it than explanation 😛 And it’s true that complexity science has led to innovations in prediction as well as explanation.

      Ultimately I agree that theory can take us a step farther than raw black-box prediction. As machine learners like to say, “your model is only as good as your training set”.

      One takeaway for me is that although networks can sometimes be quite useful for prediction, they are *always* useful for explanation due to being theory-driven and interpretable. I guess only time will tell whether psychology is ultimately simple enough that we can have models that give good explanation AND good prediction simultaneously. I personally have my doubts, but I would be happy to be proven wrong. Although complexity science can be used for prediction in e.g. rocket science, in reality rocket science is probably quite simple compared to the human brain!

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s