Which experts should we trust?

In a comment on our post, “Expert writes op-ed in NYT recommending that we trust the experts,” commenter DCE writes:

Perhaps this post can have a follow-up on “How do I choose which experts to believe?” While broadly, Pigliucci’s “Nonsense on Stilts” offers some good discussion, there is the real issue of ulterior motives in crafting messages. . . . How to pick your experts is a thorny meta-research issue.

Relatedly, Paul Alper pointed us to this New York Times article entitled, “Worried About That New Medical Study? Read This First. There’s more than meets the eye — here are some tips to help avoid confusion,” in which a physician and medical journalist, with training in biostatistics and epidemiology, offers the following false statement:

When a study or a journalistic publication says that a study’s finding was “statistically significant,” it means that the results were unlikely to have happened by chance.

And also gives some advice which seems questionable to me:

When it comes to study design, not all are created equal. In medicine, randomized clinical trials and systematic reviews are kings. . . .

and

Not all journals are created equal. . . . A good way to spot a high quality journal is to look for one with a high impact factor . . .

That NYT article also offers some more unambiguously good advice, though, such as, “take each study for what it is: information. Over time, it will become clearer whether one conclusion was important enough to change clinical recommendations. . . . One study isn’t likely to shift an entire course of medical practice.”

The larger question

The larger question is, if we can’t trust the experts, who can we trust? Or, if we can’t trust anyone, what can replace “trust” in our reasoning? No easy answer here. We’ve already discussed problems with putting trust in your friends. I think the only solution is to think of post-publication review as a way of life, including reputational incentives in some way.

Should you trust what I write? I don’t know. I try to make my reasoning clear (the “trail of breadcrumbs” linking data, substantive theories, statistical models, and conclusions) to make it easier for you to judge for yourself.