Why idiography can’t save us

The Move to Idiography

I have sensed a shift towards criticizing and rejecting cross-sectional analyses in favor of within-person longitudinal analyses. Peer reviewers seem to recoil at group-level analyses and gush at time-series. Even for researchers who are still working on cross-sectional data, my conversations with them reveal that they typically see within-person analysis as “the future”. More and more researchers are running phone-based ecological momentary assessment (EMA) studies, hoping that passive and active monitoring can lead us to important insights that will reveal the dynamical systems that are the basis of mental illnesses. This shift seems especially dramatic in the psychometric network community, where I spend a lot of my time.

Why the shift? Critics of between-person analyses have rightly pointed out some of the important limitations of these approaches. It is true that between-person variance is not transmutable to within-person variance. If a certain network structure or factor structure fits between-persons, this does not mean that we should expect the same structure to fit within people. The two solutions are often nowhere even close. It’s true! I have seen this argument presented many times, and I never object, because mathematically, it’s absolutely correct.

Unfortunately, I think that this critique is almost always misinterpreted, especially by listeners whose formal statistical training is limited (true for most psychologists, myself included). The idea that between-person variance could tell us absolutely nothing about parameters within individuals seems like it should be highly alarming, invalidating years and years of work!

The problem is that it’s really not that alarming at all. If you do a bit of a deeper dive into what these parameters actually mean and how they are generated, there is really no compelling reason why we should ever expect between-person parameters to be equivalent to within-person parameters. And — I can’t emphasize this enough — that’s perfectly OK. The parameters mean very different things, so it would actually be a bit alarming if they were always equivalent. This would imply some very weird assumptions, which we should probably never expect to be true.

The Importance of Causal Theories

I think where people tend to get tripped up is in conflating within-person variance and within-person causal models. It is absolutely critical to understand that these are not the same thing.

Moreover, within-person variance is not any closer to a within-person causal model than between-person variance is! Both within- and between-person variance are ultimately the result of within-person causes. This becomes easier to understand in the context of human development. If differences exist between-persons, that difference is resultant from some within-person causal process at some point in time. Differences between people are caused by differences within those people.

Another way to think of this is to realize that in a random sample, there is no such thing as “between-person causes”— these would only exist if the members of the sample were interacting with one another. This leaves only within-person causes.

Both between- and within-person variance tell you different things and each come with their own limitations. It’s difficult, but possible, to come up with sets of reasonable causal hypotheses from either type of data. But now I’ve written the words “within” and “between” so many times that I may be losing you— so it’s time for a more practical example.

An Example of “Within-” and “Between-” Variance

I’m a big fan of “thought experiments”, so allow me to make a slight detour. Imagine a totally naive scientist who is chronically cold. She wants to figure out how to stay warm. She studies a group of 200 individuals, examining their day-to-day practices that influence their warmth. First, the scientist tries a within-persons approach to study warmth. She studies each of the 200 individuals for 1 month using EMA to examine when they feel relatively warm and when they feel relatively cold. Her analysis reveals apparent variables associated with increasing warmth, such as wearing thick clothing or sitting near fires. There are also patterns that differ across people. Some individuals apparently feel warmer when they retreat indoors, avoid the outdoors, and avoid going to places near large bodies of water or wind. However, for other individuals, she finds that the exact opposite is true— they feel warmer when they venture outdoors, especially near the water.

She is puzzled. What could cause these discrepant patterns? Finally, the scientist turns to a between-persons approach, studying a myriad of variables at one time point and comparing them. As it turns out, half the individuals (who are mostly warm) live near the Florida beach while the other half (who are mostly cold) live on the Alaskan coast.

If the scientist wants to stay warm, what should she do? Well, it certainly wouldn’t hurt to adopt the within-person strategies that seem to work for everyone, such as wearing thicker clothing. She could also adapt some specific strategies depending on where she lives. Or, she could take the retiree’s route and move to Florida.

Now, you might be thinking that I just cherry-picked this example to demonstrate one scenario where between-person factors are generally more influential than within-person factors. And you’d be right— we could probably generate all kinds of thought experiments that show the relative importance of within-person factors.

But the point of this thought experiment isn’t just to point out that between-subjects analyses can sometimes be (more) useful. Rather, the broader point is that within-persons and between-persons analyses give us different information, and have different limitations. The between-persons analysis doesn’t tell us anything about the relative states and temporary strategies of any of the individuals. Meanwhile, the within-persons analysis doesn’t tell us anything about causes that remained stable during our measurement period (i.e., location). In short, between-person variance explains why people differ. Within-person variance explains why a person differs from their own self at different time point. Both result from causes within people. Each has its limitations.

Limitations of Within-Persons

Perhaps what most frustrates me is the insistence that the problems of between-subjects analyses should be fixed by switching to within-subjects analyses.

I frequently hear arguments about how between-subjects tests are correlational and cannot reveal causality. I agree. I hear how between-subjects tests are subject to confounding and are only valid when we have ruled out potential extraneous causes. True. I hear that cross-sectional models cannot be interpreted as within-person causes. All accurate. Then, I hear all of this used to justify a switch to within-persons analyses. This is where I get lost.

None of these concerns are addressed by within-persons analysis. Just because an analysis is “within a person” does not mean that it reveals any causal processes about that person. The vast majority of critical limitations for inferring causality still apply in a within-person’s analysis. And although there are some advantages, the list of things to worry about is long:

  1. Did you get the time scale right? Are your lag-1 and lag-2 and lag-3 parameters all telling you the same thing? Probably not – but why is that? Well, imagine you are studying exercise. At lag-1, you see that exercise is associated with higher blood pressure. At lag-5 through lag-2399, the relationship disappears. At lag 2400 (you had a lag-2400 and tested it, right?), you see that it is associated with very slightly lower blood pressure. So which parameter matters more? Decyphering the differences between different lag points is far from trivial without the help of other types of data.
  2. Speaking of time scale, are you sure that the window between your measurements is small enough for the phenomenon you care about? Any interactions that occur on a time scale faster than your measurement interval will be amalgamated within the contemporaneous associations, which aren’t informative in terms of temporal precedence.
  3. Were there any variables that remained very stable over your course of measurement? Tough luck — a lack of within-person variance means a lack of estimable parameters. So you probably won’t be able to learn much about things like genetics, trauma, prenatal environment, cognitive styles, attentional and interpretation biases, or a host of other important variables.
  4. Is the person actually changing at all over time? Are they in therapy? If so, the vast majority of analyses are invalidated because of stationarity problems. Yes, there are methods out there for controlling for it, but in my experience this has the side-effect of making your parameters incredibly hard to interpret, and often means removing the very information that is most important to your theory.
  5. Oh look, it looks like the variance decreases over time and the associations become more reliable. Interesting. Wait— is this just my participants getting into a pattern of responding now that they’ve filled out the same questionnaire 30 times?
  6. Ah, looks like my subject suffering from bipolar disorder stopped responding for about 7 days right before a hospitalization. No worries, I’m sure that data is unimportant and missing completely at random
  7. This is all still correlational. I hope you didn’t forget about those pesky third variables. Do you happen to have any items that just might overlap conceptually? Remember that all your worries about confounders still follow you here.

In Clinical Psychology

It is my view that the problems of within-persons analysis are particularly devastating in clinical psychology. We are often dealing with individuals who are in a more or less stable “state” (e.g., suffering from PTSD) who want to get into a different state (e.g., not suffering from PTSD). In a relatively short-term within-person analysis (say, 2 weeks of EMA prior to treatment), you can only hope to capture the variance associated with the “sick state”. Unfortunately, this variance tells you absolutely nothing about how a state transition might occur. If you decide to measure a patient with EMA long enough so that they experience state transition, then congratulations — your patient is cured and no longer needs treatment! The “2 weeks prior to treatment” type data can tell you some interesting things, but ultimately it is uninformative in telling you about treatment targets, unless you can also compare it to other people who have already undergone treatment with a between-subjects approach.

Now I don’t mean to imply that within-persons analyses aren’t valuable or important, or that you shouldn’t do them just because they are difficult to get right. I actually think they are pretty cool and I’m glad people do them. They capture some really interesting data that has been ignored for many, many years. What I do mean to imply is that within-persons analyses are not a straightforward path to understanding within-persons causality.

Experiments, Experiments, Experiments

Luckily, we have a do have a pretty straightforward path to determining causality—the experiment. I’m almost certainly preaching to the choir here, but experiments are epistemologically just plain awesome. Randomized experiments are probably the closest humans have ever gotten to performing magic. They do reveal causality, and if done carefully, they leave very little room for alternative explanations and confounders.

Both between- and within- persons non-experimental designs can be great for hypothesis generating. But I suspect that most people, and especially those who care about network theory, ultimately care about causal systems, which must be established through solid theory and hypothesis testing. For those, your best bets are:

  1. Experiments
  2. Experiments
  3. Within-persons experiments
  4. Between-persons experiments (aka RCTs)
  5. Within-between ABAB experiments
  6. Large-scale pre-registered experiments with long-term follow-ups AND
  7. Experiments

To me, it is a painful irony that RCTs seem to be falling out of fashion precisely at the same time that complexity theorists are gaining ground. Experiments provide something that neither between- nor within-persons analysis can address.

I don’t think there is anything wrong with within-persons data. But I also don’t think it makes sense to stomp on between-subjects analyses in a misguided attempt to promote within-persons analyses. I truly think we’ve generated some important ideas using hypothesis-generating methods like between-subjects psychometric networks. I hope we make even more progress with within-person hypothesis generating networks! But if it’s theory and causality that we’re after, I don’t think within-persons analyses can get us there without some serious help.

 

Written by Payton J. Jones. Thanks to Shaan McGhie for help with refining and editing.

 

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s