[Edit Dec 17th 2016] The Economist has now covered our analysis here. The Guardian, Independent and Telegraph have now retracted their articles because of our findings. [End]
In a report published by the Hope Not Hate campaign last Monday, researchers Dr Imran Awan and Dr Irene Zempi describe widespread celebration of the murder of Labour MP Jo Cox on the Twitter website during the months of June and July of 2016.
The findings of the report do not appear to stand up to fact-checking. Unfortunately, national newspapers have widely reported on the story, taking it at face value:
These claims appear false. We could find evidence of less than seventy tweets in support of Jo Cox’s murder. This implies that the claims are inflated by a factor of around one thousand.
This study risks normalizing extremely rare viewpoints and representing them as more common and more acceptable than they truly are. This will have precisely the opposite effect to that intended. In general, 'fake news' in respected media sources acts to decrease public trust and should not go unchallenged.
The Hope Not Hate report, Jo Cox ‘deserved to die’: Cyber Hate Speech Unleashed on Twitter, seems to be a conflation of two analyses, one concerning brexit-related tweets and another concerning tweets about Jo Cox. It is not carefully written and it is often not clear which data set is being referred to. Our analysis suggests that the 50,000 tweet claim belongs to tweets about ‘brexit’ while a vastly smaller dataset is actually about Jo Cox.
Principal claim of the Hope Not Hate report: False
The authors first research question was ‘What was the nature and frequency of hate speech targeting Jo Cox, following her murder by Thomas Mair?’. The main claim concerning this question is as follows:
The top 20 words used to describe Thomas Mair and Jo Cox MP included the terms: Hero, Patriot, White Power, Rapists and Traitor.
The implication is that these words appeared in large numbers of negative tweets about Jo Cox. Our evidence shows that this is not the case.
We took all tweets from June and July 2016 which included the phrase ‘Jo Cox’ or ‘#JoCox’. There were 340,642 unique tweets in this dataset. We counted the number of times each of the five terms ‘Hero’, ‘Patriot’, ‘White Power’, ‘Rapists’ and ‘Traitor’ occurred in these tweets. We then verified the meaning of each tweet, using two independent human curators. This is critical, because it is important to check whether the word ‘hero’ is used to refer to Jo Cox or Thomas Mair, for instance.
The number of tweets found for each keyword, along with the number of unique tweets critical of Jo Cox, are presented in the following table:
* For the ‘Hero’ keyword, a random sample of 100 tweets were analysed and results extrapolated, due to the large volume of tweets (technically, the 95% confidence interval for the full data set is 0 to 52).
This analysis indicates that only approximately 57 tweets in this dataset are critical of Jo Cox. In fact, examples of the tweets containing these five keywords show that the vast majority of them actually support Jo Cox:
It is wrong to suggest these tweets ‘celebrate’ the murder of Jo Cox. In particular, example number 5 suggests that the Hope Not Hate analysis was badly led astray by the many reports that Mair was a subscriber of the ‘South African Patriot’ magazine.
Other claims in the Hope Not Hate report: Also False
There were two further claims relating to tweets about Jo Cox:
A key theme that emerged on Twitter was the depiction of Thomas Mair as a ‘hero’ for murdering Cox. Individuals had tagged pictures in their tweets praising Mair for killing Jo Cox, using the hashtag #HeroMair.
A key theme that emerged on Twitter was the notion that Jo Cox ‘deserved to die’ because ‘she supported rape gangs’, with references to the child sexual exploitation scandal in Rotherham, South Yorkshire.
These claims also appear false. We could find evidence of only one tweet using the hashtag #HeroMair, for example.
It is unclear what the authors mean by a ‘key theme’ but it is certainly the case that sentiments that Jo Cox ‘deserved to die’ are in the tiny minority. When they were expressed they were harshly criticised, often by people on the same side of the political spectrum as the person expressing them.
There were 198 unique tweets containing the word ‘deserved’ together with ‘Jo Cox’ or ‘#JoCox’ (241 in total). Only ten of them expressed the sentiment that Jo Cox ‘deserved to die’, posted from a handful of accounts. Four additional accounts posting negative tweets have been disabled by Twitter. This is a sample of the remaining tweets
Incidentally, it might be thought that there are tweets in the authors’ dataset that have since been removed by Twitter, and therefore unavailable to us. We have found only four instances where twitter accounts have been removed. The tweets from these accounts are still represented in our dataset, since there are many replies (expressing condemnation) to the original tweets.
We are therefore confident we are not missing data relative to the Hope Not Hate report. Furthermore, we were able to find all of the tweets which the authors directly quote in the report. (We make the assumption here that the tweets they quote were an accurate representation of their dataset.)
Summary
We found evidence for less than seventy tweets that could remotely be described as ‘celebrating the death’ of Jo Cox, instead of the fifty thousand reported.
Although it seems that there are severe methodological errors in the Hope Not Hate report, we do not suggest that the authors have been deliberately misleading. We have informed the authors that we cannot reproduce their results and have asked them to share their dataset or clarify their methods. They told us they are currently unable to do so.
There is always the possibility that the authors of the Hope Not Hate report had access to dfferent data or used queries that they have not detailed in the report. This would change our findings and we would welcome the opportunity to repeat our analysis with the aid of the Hope Not Hate investigators.
Our analysis was aided by using specialised tools but our results can be easily checked by searches on the main twitter website. We will also make our dataset available to any academics or journalists who contact us.
Finally, we would like to state that the massively overwhelming message in over 340 thousand tweets about Jo Cox’s murder is represented by tweets such as the following:
‘Heart still breaking for #JoCox’
Dr Martin Goodson and Rafal Kwasny (Evolution AI)
