There’s a saying in art that you have to draw things the way they look, not the way they are.
This reminds me of an important but rarely stated principle in statistical reasoning, the distinction between evidence and truth.
The classic error of novices when drawing is to draw essences—for example, drawing a head as a pair of eyes and a nose and a mouth and a couple of ears and a chin etc. The mistake is to draw linguistically rather than visually. It’s easy to recognize this error but hard to fix. If I, as an unskilled draftsman, try to draw visually, I don’t do a good job. I think the only way I could really do this is by cheating and putting a grid across my field of view and a grid on the paper I’m drawing on, and fill in one little grid square at a time.
That said, there are tricks to teach novices how to draw, and the tricks involve constructing an image from essences, but essences that are geometrical rather than real. So you construct a dog picture, say, from a set of circles and boxes. The idea is that it’s so hard not to draw based on essences, that the best step toward truly visual drawing is to use abstract geometric essences. A related trick is to observe and then draw the negative space, or to draw things upside down; again, these are methods for detaching you from your preconceptions. To put it another way, you can’t draw “linguistically” if you stop yourself from “reading” the face as a mouth plus a nose plus etc.
Now let’s move to statistical reasoning. It’s my impression that applied researchers are working with truth (as they perceive it), not evidence. Or, to put it another way, when they try to “draw the picture” of their evidence, they do it by putting together pieces of truth: This effect is real, that effect is zero, etc.
The question, then, is how to help people. By analogy to drawing, it’s not enough to simply tell people to summarize the data without preconceptions (or, to be more precise from a statistical perspective, to express their preconceptions formally within the statistical model): it’s just too hard for novices to do this from scratch, in the same way that “Just draw what you see” is advice that’s too hard for civilians like me to follow while drawing.
So what we need is a set of tools that will allow people to summarize the data they way they look, without getting tangled in essences. My usual recommendation is to display everything (as in figure 3 of this paper) rather than pulling out statistically significant things to tell a story. For an example of what not to do, see the article discussed in section 2.2 of this paper. It’s hard to learn from data when you’re already telling the story you want to see, in the same way that it’s hard to draw a dog if you see it as a collection of existing parts (head, legs, body, tail).
What you need to draw things the way they look, rather than based on your view of essences, is to develop a sort of contextual dissociation. This is the way that drawing can yield new insights rather than just regurging your preconceptions.
Similarly with statistical analysis: you need to go back and forth between your substantive understanding (including your preconceptions) and a more dissociated, data-first, descriptive presentation.
I think more needs to be said and done in this area.