Category Archives: Psychology

Psychology, Publications

Psychology journal bans P values

Leave a comment

I was rather surprised to see recently (OK, it was a couple of months ago, but I do have a day job to do as well as writing this blog) that the journal Basic and Applied Social Psychology has banned P values.

That’s quite a bold move. There are of course many problems with P values, about which David Colquhoun has written some sensible thoughts. Those problems seem to be particularly acute in the field of psychology, which suffers from something of a problem when it comes to replicating results. It’s undoubtedly true that many published papers with significant P values haven’t really discovered what they claimed to have discovered, but have just made type I errors, or in other words, have obtained significant results just by chance, rather than because what they claim to have discovered is actually true.

It’s worth reminding ourselves what the conventional test of statistical significance actually means. If we say we have a significant result with P < 0.05, then that means that there is a 1 in 20 chance we would have seen that result if in fact we had completely random data. A 1 in 20 chance is not at all rare, particularly when you consider the huge number of papers that are published every day. Many of them are going to have type I errors.

Clearly, something must be done.

However, call me a cynic if you like, but I’m not sure how banning P values (and confidence intervals as well, if you thought just banning P values was radical enough) is going to help. Perhaps if all articles in Basic and Applied Social Psychology in the future have robust Bayesian analyses that would be an improvement. But I hardly think that’s likely to happen. What is more likely is that researchers will claim to have discovered effects even if they are not conventionally statistically significant, which surely is even worse than where we were before.

I suspect one of the problems with psychology research is that much research, particularly negative research, goes unpublished. It’s probably a lot easier to get a paper published showing that you have just demonstrated some fascinating psychological effect than if you have just demonstrated that the effect you had hypothesised doesn’t in fact exist.

This is a problem we know well in my world of clinical trials. There is abundant evidence that positive clinical trials are more likely to be published than negative ones. This is a problem that the clinical research community has become very much aware of, and has been working quite hard to solve. I wouldn’t say it is completely solved yet, but things are a lot better now than they were a decade or two ago.

One relevant factor is the move to prospective trial registration.  It seems that prospectively registering trials is helping to solve the problem of publication bias. While clinical research doesn’t yet have a 100% publication record (though some recent studies do show disclosure rates of > 80%), I suspect clinical research is far ahead of the social sciences.

Perhaps a better solution to the replication crisis in psychology would be a system for prospectively registering all psychology experiments and a commitment by researchers and journals to publish all results, positive or negative. That wouldn’t necessarily mean more results get replicated, of course, but it would mean that we’d be more likely to know about it when results are not replicated.

I’m not pretending this would be easy. Clinical trials are often multi-million dollar affairs, and the extra bureaucracy involved in trial registration is trivial in comparison with the overall effort. Many psychology experiments are done on a much smaller scale, and the extra bureaucracy would probably add proportionately a lot more to the costs. But personally, I think we’d all be better off with fewer experiments done and more of them being published.

I don’t think the move by Basic and Applied Social Psychology is likely to improve the quality of reporting in that journal. But if it gets us all talking about the limitations of P values, then maybe that’s not such a bad thing.

 

Cancer, Dodgy reporting, Politics, Psychology

The Saatchi Bill

3 Comments

I was disappointed to see yesterday that the Saatchi Bill (or Medical Innovations Bill, to give it its official name) passed its third reading in the House of Lords.

The Saatchi Bill, if passed, will be a dreadful piece of legislation. The arguments against it have been well rehearsed elsewhere, so I won’t go into them in detail here. But briefly, the bill sets out to solve a problem that doesn’t exist, and then offers solutions that wouldn’t solve it even if it did exist.

It is based on the premise that the main reason no progress is ever made in medical research (which is nonsense to start with, of course, because progress made all the time) is because doctors are afraid to try innovative treatments in case they get sued. There is, however, absolutely no evidence that that’s true, and in any case, the bill would not help promote real innovation, as it specifically excludes the use of treatments as part of research. Without research, there is no meaningful innovation.

If the bill were simply ineffective, that would be one thing, but it’s also actively harmful. By removing the legal protection that patients  currently enjoy against doctors acting irresponsibly, the bill will be a quack’s charter. It would certainly make it more likely that someone like Stanislaw Burzynski, an out-and-out quack who makes his fortune from fleecing cancer patients by offering them ineffective and dangerous treatments, could operate legally in the UK. That would not be a good thing.

One thing that has struck me about the sorry story of the Saatchi bill is just how dishonest Maurice Saatchi and his team have been. A particularly dishonourable mention goes to the Daily Telegraph, who have been the bill’s “official media partner“. Seriously? Since when did bills going through parliament have an official media partner? Some of the articles they have written have been breathtakingly dishonest. They wrote recently that the bill had “won over its critics“,  which is very far from the truth. Pretty much the entire medical profession is against it: this response from the Academy of Royal Medical Colleges is typical. The same article says that one way the bill had won over its critics was by amending it to require that doctors treating patients under this law must publish their research. There are 2 problems with that: first, the law doesn’t apply to research, and second, it doesn’t say anything about a requirement to publish results.

In an article in the Telegraph today, Saatchi himself continued the dishonesty. As well as continuing to pretend that the bill is now widely supported, he also claimed that more than 18,000 patients responded to the Department of Health’s consultation on the bill. In fact, the total number of responses to the consultation was only 170.

The dishonesty behind the promotion of the Saatchi bill has been well documented by David Hills (aka “the Wandering Teacake”), and I’d encourage you to read his detailed blogpost.

The question that I want to ask about all this is why? Why is Maurice Saatchi doing all this? What does he have to gain from promoting a bill that’s going to be bad for patients but good for unscrupulous quacks?

I cannot know the answers to any of those questions, of course. Only Saatchi himself can know, and even he may not really know: we are not always fully aware of our own motivations. The rest of us can only speculate. But nonetheless, I think it’s interesting to speculate, so I hope you’ll bear with me while I do so.

The original impetus for the Saatchi bill came when Saatchi lost his wife to ovarian cancer. Losing a loved one to cancer is always difficult, and ovarian cancer is a particularly nasty disease. There can be no doubt that Saatchi was genuinely distressed by the experience, and deserves our sympathy.

No doubt it seemed like a good idea to try to do something about this. After all, as a member of the House of Lords, he has the opportunity to propose new legislation. It is completely understandable that if he thought a new law could help people who were dying of cancer, he would be highly motivated to introduce one.

All of that is very plausible and easy to understand. What has happened subsequently, however, is a little harder to understand.

It can’t have been very long after Saatchi proposed the bill that many people who know more about medicine than he does told him why it simply wouldn’t work, and would have harmful consequences. So I think what is harder to understand is why he persisted with the bill after all the problems with it had been explained to him.

It has been suggested that this is about personal financial gain: his advertising company works for various pharmaceutical companies, and pharmaceutical companies will gain from the bill.

However, I don’t believe that that is a plausible explanation for Saatchi’s behaviour.

For a start, I’m pretty sure that the emotional impact of losing a beloved wife is a far stronger motivator than money, particularly for someone who is already extremely rich. It’s not as if Saatchi needs more money. He’s already rich enough to buy the support of a major national newspaper and to get a truly dreadful bill through parliament.

And for another thing, I’m not at all sure that pharmaceutical companies would do particularly well out of the bill anyway. They are mostly interested in getting their drugs licensed so that they can sell them in large quantities. Selling them as a one-off to individual patients is unlikely to be at the top of their list of priorities.

For what it’s worth, my guess is that Saatchi just has difficulty admitting that he was wrong. It’s not a particularly rare personality trait. He originally thought the bill would genuinely help cancer patients, and when told otherwise, he simply ignored that information. You might see this as an example of the Dunning Kruger effect, and it’s certainly consistent with the widely accepted phenomenon of confirmation bias.

Granted, what we’re seeing here is a pretty extreme case of confirmation bias, and has required some spectacular dishonesty on the part of Saatchi to maintain the illusion that he was right all along. But Saatchi is a politician who originally made his money in advertising, and it would be hard to think of 2 more dishonest professions than politics and advertising. It perhaps shouldn’t be too surprising that dishonesty is something that comes naturally to him.

Whatever the reasons for Saatchi’s insistence on promoting the bill in the face of widespread opposition, this whole story has been a rather scary tale of how money and power can buy your way through the legislative process.

The bill still has to pass its third reading in the House of Commons before it becomes law. We can only hope that our elected MPs are smart enough to see what a travesty the bill is. If you want to write to your MP to ask them to vote against the bill, now would be a good time to do it.

Homeopathy, Psychology

Volunteers wanted for research into homeopathy

3 Comments

I am planning a research project to explore the experiences of people who have used homeopathy, and if you have used homeopathy yourself then I would be really grateful if you would consider taking part in my research.

I would like to interview people who have used homeopathy, have been pleased with the results, but have encountered negative reactions to your use of homeopathy from others. Perhaps your GP has advised you not to use homeopathy, perhaps friends or family have told you that you were wasting your time, or perhaps you got into an argument with someone on the internet. It doesn’t matter who reacted negatively to your use of homeopathy: I am interested in learning about how users of homeopathy experience negative reactions in general.

The research will take the form of a short interview of about 30 minutes (which can take place in a location of your choice), in which I will ask you about your experiences and how they seemed to you. I shall be using a phenomenological approach to the research, which means that I am interested in learning about your experiences from your own point of view, rather than trying to fit them into a pre-existing theory.

This research is part of a degree in social sciences that I am doing with the Open University. Specifically, it is part of a module in social psychology.

In the interests of transparency, I should tell you that I am sceptical of the benefits of homeopathy. However, my intention in this research is not to challenge your views about homeopathy, it is to come to a better understanding of them.

If you decide you want to take part but later change your mind, that is fine. In that case, any materials from your interview would be returned to you or destroyed, as you prefer, and your interview would not be used in my research.

Please be assured that your participation in the study would be kept strictly confidential.

If you are interested in taking part in my research or if you would just like to know more about about the project, please feel free to contact me.

Thank you for taking the time to read this.

Adam Jacobs

Update 29 January 2015:

Many thanks to everyone who volunteered for this project. I now have enough data, and so I no longer need any more volunteers.