It stuck me how much analysis is done on the published results , without asking if there is an audit trail back to the original data collection process. Data doesn’t first exist in an Excel spreadsheet, normally there will be some input medium. Perhaps another Open science best practice would be around securing auditable ‘chain of custody’ type mechanisms for data collection.
Yes! I've just been writing some notes for our extra-spooky Halloween episode that covers this exact issue (you'll see the connection, I promise!). Out on the 31st.
Oh so many thoughts. I am a policy analyst working in police oversight so I want to talk about the Eric Stewart stuff. It blows my mind how often people think "no it wasn't fraud, I am just horrible at my job!" is a compelling defense. And frankly they are not wrong! Universities do clearly care more about fraud than they do about the competency of their researchers.
I think the obsession with outright fraud particularly in the disciplinary process is really detrimental to punishing bad researchers. I frankly don't think it much matters if you botched your paper horribly on purpose or because you simply have no idea what you are doing, and in a lot of ways the latter is worse! A fraudster is at least theoretically capable of good work if they can stop lying, an incompetent researcher is mostly hopeless.
Oh, also I don't know if Eric Stewart was ever actually found guilty of fraud. All the university investigations I saw via retraction watch clear him of fraud and instead accuse him of gross negligence. Basically there's no super strong evidence he's a fraud and huge evidence he's wholly incompetent.
You're right about the investigations - I misspoke and we should clarify that! Having said that:
1) Justin Pickett who investigated Stewart's papers and was even a co-author on one of them, thinks it wasn't just a mistake and was deliberate (see e.g. the retraction note on https://onlinelibrary.wiley.com/doi/10.1111/1745-9125.12235).
2) If you read some of the comments in the report they're going 99% of the way to accusing him of fraud IMO:
"The misconduct claims were not rejected, but in all inquiries into the matter, there was found to be insufficient evidence to support a full investigation into research misconduct which is defined as fabrication, falsification, or plagiarism in proposing, performing, or reviewing research, or in reporting research results."
But there was only "insufficient evidence" because he lost the files! The dog ate his homework. This is the Dean of his College:
“A data file that is ‘lost’ while it is under investigation is particularly peculiar. Three outcomes are potentially possible. First, the data could have been lost (least likely explanation). Second, the data may have never existed. Third, the data existed in some format but were destroyed to prevent others from accessing them. This latter possibility is bolstered by the fact that Dr. Stewart asked a colleague about a data-eraser program that deletes temporary data/files/backups from computers that analyze restricted-use samples."
Hmmmmmm! I suppose that's technically still compatible with the data being massively screwed up by accident and his wanting to conceal that fact. But, c'mon.
I am with you that it's probably the most plausible explanation, but these people tend to be a little litigious. Also his termination letter paints the picture of someone who is just kinda lazy/incompetent. Either way I think my broader point is that it doesn't actually matter - gross negligence and fraud both have the same outcome, so his fallback on "I am not a fraud I am just incompetent" should not be the reputation-saving defense he thinks it is. Whether it WILL be is to be seen, but I suspect claiming incompetence means he has a much better chance of breaking back into the field.
Weird! It works on desktop. Here's the link if you can't get to it https://www.worksinprogress.news/s/notes-on-progress and I'll have a look in the back end and see if I can work out what's going on. Thanks, and glad you're enjoying it!
But then why are the later links working? E.g. ‘article’ in ‘Big New Yorker article’ is a hyperlink... . Whatever is wrong is affecting just that one link.
I worked it out. It's because I put the sponsor in as a block quote so it's centred on the Substack page and stands out a bit. Turns out that switches off the links when it passes the text to Apple Podcasts etc. Oops!
It stuck me how much analysis is done on the published results , without asking if there is an audit trail back to the original data collection process. Data doesn’t first exist in an Excel spreadsheet, normally there will be some input medium. Perhaps another Open science best practice would be around securing auditable ‘chain of custody’ type mechanisms for data collection.
Yes! I've just been writing some notes for our extra-spooky Halloween episode that covers this exact issue (you'll see the connection, I promise!). Out on the 31st.
GREAT job! ❤️ Might be nice to have a future episode on what a reformed scientific system could look like, just to talk everyone off the ledge.
Oh so many thoughts. I am a policy analyst working in police oversight so I want to talk about the Eric Stewart stuff. It blows my mind how often people think "no it wasn't fraud, I am just horrible at my job!" is a compelling defense. And frankly they are not wrong! Universities do clearly care more about fraud than they do about the competency of their researchers.
I think the obsession with outright fraud particularly in the disciplinary process is really detrimental to punishing bad researchers. I frankly don't think it much matters if you botched your paper horribly on purpose or because you simply have no idea what you are doing, and in a lot of ways the latter is worse! A fraudster is at least theoretically capable of good work if they can stop lying, an incompetent researcher is mostly hopeless.
Oh, also I don't know if Eric Stewart was ever actually found guilty of fraud. All the university investigations I saw via retraction watch clear him of fraud and instead accuse him of gross negligence. Basically there's no super strong evidence he's a fraud and huge evidence he's wholly incompetent.
You're right about the investigations - I misspoke and we should clarify that! Having said that:
1) Justin Pickett who investigated Stewart's papers and was even a co-author on one of them, thinks it wasn't just a mistake and was deliberate (see e.g. the retraction note on https://onlinelibrary.wiley.com/doi/10.1111/1745-9125.12235).
2) If you read some of the comments in the report they're going 99% of the way to accusing him of fraud IMO:
"The misconduct claims were not rejected, but in all inquiries into the matter, there was found to be insufficient evidence to support a full investigation into research misconduct which is defined as fabrication, falsification, or plagiarism in proposing, performing, or reviewing research, or in reporting research results."
But there was only "insufficient evidence" because he lost the files! The dog ate his homework. This is the Dean of his College:
“A data file that is ‘lost’ while it is under investigation is particularly peculiar. Three outcomes are potentially possible. First, the data could have been lost (least likely explanation). Second, the data may have never existed. Third, the data existed in some format but were destroyed to prevent others from accessing them. This latter possibility is bolstered by the fact that Dr. Stewart asked a colleague about a data-eraser program that deletes temporary data/files/backups from computers that analyze restricted-use samples."
Hmmmmmm! I suppose that's technically still compatible with the data being massively screwed up by accident and his wanting to conceal that fact. But, c'mon.
I am with you that it's probably the most plausible explanation, but these people tend to be a little litigious. Also his termination letter paints the picture of someone who is just kinda lazy/incompetent. Either way I think my broader point is that it doesn't actually matter - gross negligence and fraud both have the same outcome, so his fallback on "I am not a fraud I am just incompetent" should not be the reputation-saving defense he thinks it is. Whether it WILL be is to be seen, but I suspect claiming incompetence means he has a much better chance of breaking back into the field.
LIstening and enjoying right now. On iOS, the show notes link to Notes on Progress is broken...
Weird! It works on desktop. Here's the link if you can't get to it https://www.worksinprogress.news/s/notes-on-progress and I'll have a look in the back end and see if I can work out what's going on. Thanks, and glad you're enjoying it!
Screenshot in case it helps... https://i.imgur.com/qvLYQE8_d.webp?maxwidth=1520&fidelity=grand
Hopefully I’m not having a brain freeze...
Ah! I wonder if it strips out all the hypertext when it ports it over to Apple Podcasts.
Ah, bugger - yes it does! I'll start putting in the actual URLs from now on. Thanks for letting us know!
But then why are the later links working? E.g. ‘article’ in ‘Big New Yorker article’ is a hyperlink... . Whatever is wrong is affecting just that one link.
I worked it out. It's because I put the sponsor in as a block quote so it's centred on the Substack page and stands out a bit. Turns out that switches off the links when it passes the text to Apple Podcasts etc. Oops!