Lightly abridged excerpt from notes in an email to Heidi Williams, July 17 2022, riffing on metascience, a metascience workshop, and my work with Kanjun Qiu. Those notes are, in turn, abridged from my personal notes.
Much of this applies to almost all metascience, though noted in the context of this particular workshop. Much of it also a riff on Kanjun Qiu's and my forthcoming broad essay on metascience (which I hope to send to you soon in draft)
For me, the key question (and implicit promise) remains: can we achieve transformative improvements in the way human beings make scientific discoveries? I'm still unsure. At the level of in-principle process I'm absolutely [ed, should be: /almost/] certain: we can do vastly better. At the level of: can our institutional ecosystem make the necessary changes, I have doubts. I'd very much like to arrive at more conviction, one way or the other. Absent that conviction, I have a lot of uncertainty about how to personally contribute. I also don't quite know how to leave the damn problem alone! It is a civilizational problem. I guess that sounds like hubris, but think it's literally correct.
Most metascience events are fundamentally incremental. "How to fix such-and-such a problem?" There's often lots of operators present, and they sometimes mistake operational details about existing institutions for fundamental facts about science. There's a complementary first-principles imaginative design point of view possible, too. It'd be fun to mash up both. Over the long run, I think metascience is (or ought to aspire to be) fundamentally an imaginative design discipline.
Absent from the workshop was much discussion of the difference between insiders and outsiders. One of the most important innovations of early modern science was to decentralize change in ideas, so if outsiders had improved ideas they could win over incumbents. This works pretty well in science at the level of ideas, but doesn't work at all at the level of social processes. Two grad students in a garage can't start a new university / funder and grow it to replace Harvard / the NIH over 25 years, merely because their approach is better. And even to the extent that new ventures can be started it's because someone in a position of power or wealth likes the idea, not because it's good. Brian Josephson didn't win the Nobel Prize (for an idea he had at age 21) because the elders of physics liked him, he won because nature said yes.
Strongly related: Much of the common framing of metascience places it in an essentially incremental advisory role to existing institutions. That is, it's a fundamentally centralized point of view. But Netflix didn't come about because Blockbuster took advice from J-PAL-for-entertainment, it came about due to a massive crisis which Blockbuster did everything it could to ignore. People will retort "oh, but what about startups [Arc, Altos, etc]?" But those aren't trying to win by being better, either. They're merely relying on a massive infusion of cash which is (to a worrying degree) independent of the quality of the work. There's almost no feedback loop between quality of process and growth.
I find it striking that there is no metascience canon. A rando could show up to an event like this and understand nearly all the conversation. It's a proto-field, not a field; a rando at an organic chemistry (or whatever) conference would be lost after 2 minutes. This isn't intended as a critique; it's an observation (and a huge creative opportunity, as with any proto-field).
Related: there's a sense in which the field has (almost) no results: nothing non-trivial and surprising and with strong enough evidence to drive and even force action. Again: it's a proto-field, not a field. I very much like your point that there are, however, many interesting facts. This is tremendously important, and not something I had previously understood the power of. I've been thinking about it a lot.
The only really strong exception to the last point I know of is the results about (lack of) replication in social psychology. Those results were strong enough to create a crisis, and then an ongoing renaissance. In some sense, that set of results is almost the entire canon for metascience right now. Crisis is essential to change. Also: I'm fascinated by the question: how would one obtain a second "strong result" (in this sense) in metascience? Something strong enough to force a crisis and change…
It was in part an operators' event. I'm reminded of Alan Kay's observation that you could master Linux and learn little about computers: it's a great big ball of mud. Similarly, lots of people mistake operating contingent science funding (a big ball of mud, albeit one with lots of interesting structure) for anything fundamental about science.
Related: I was in general struck by how detached people were from science. Almost nothing about why it matters, what "good" means, or what the goals of a discovery ecosystem ought to be. Mostly just faith that "more high impact science is good". But there's a through line connecting squiggles on paper and devices in a laboratory to the long-run evolution of humanity. Yet that connection was almost entirely absent. I suspect this is easily remedied by putting a certain type of scientist or philosopher of science in the room, and asking them to give a a particular type of grounding talk.
I've made a lot of critical remarks, but of course they all have an optimistic flipside: there's a lot of low-hanging fruit here!