Over the past five years, Distill has supported authors in publishing artifacts that push beyond the traditional expectations of scientific papers. From Gabriel Goh’s interactive exposition of momentum, to an ongoing collaboration exploring self-organizing systems, to a community discussion of a highly debated paper, Distill has been a venue for authors to experiment in scientific communication.
But over this time, the editorial team has become less certain whether it makes sense to run Distill as a journal, rather than encourage authors to self-publish. Running Distill as a journal creates a great deal of structural friction, making it hard for us to focus on the aspects of scientific publishing we’re most excited about. Distill is volunteer run and these frictions have caused our team to struggle with burnout.
Starting today Distill will be taking a one year hiatus, which may be extended indefinitely. Papers actively under review are not affected by this change, published threads can continue to add to their exploration, and we may publish commentary articles in limited cases. Authors can continue to write Distill-style papers using the Distill template, and either self-publish or submit to venues like VISxAI.
The Distill journal was founded as an adapter between traditional and online scientific publishing. We believed that many valuable scientific contributions — such as explanations, interactive articles, and visualizations — were held back by not being seen as “real scientific publications.” Our theory was that if a journal were to publish such artifacts, it would allow authors to benefit from the traditional academic incentive system and enable more of this kind of work.
After four years, we no longer believe this theory of impact. First, we don’t think that publishing in a journal like Distill significantly affects how seriously most institutions take non-traditional publications. Instead, it seems that more liberal institutions will take high-quality articles seriously regardless of their venue and style, while more conservative institutions remain unmoved. Secondly, we don’t believe that having a venue is the primary bottleneck to authors producing more Distill-style articles. Instead, we believe the primary bottleneck is the amount of effort it takes to produce these articles and the unusual combination of scientific and design expertise required.
We’re proud of the authors Distill has been able to support and the articles it has been able to publish. And we do think that Distill has produced a lot of value. But we don’t think this value has been a product of Distill’s status as a journal. Instead, we believe Distill’s impact has been through:
Providing mentorship to authors and potential authors.
Providing the Distill template (which is used by many non-Distill authors)
Individuals involved in Distill producing excellent articles.
Providing encouragement and community to authors.
Our sense is that Distill’s journal structure may limit, rather than support, these benefits. It creates a great deal of overhead, political concerns, and is in direct tension with some of these goals.
Instead, we think the future for most types of articles is probably self-publication, either on one-off websites or on a hypothetical “Distill Arxiv.” There are a few exceptions where we think centralized journal-like entities probably have an important enduring role, but we think the majority of papers are best served by self-publication.
Behind the scenes, the largest function of Distill is providing feedback and mentorship. For some of our early articles, we provided more than 50 hours of help with designing diagrams, improving writing style, and shaping scientific communication. Although we’ve generally dialed this down over time, each article still requires significant work. All of this is done by our editors in a volunteer capacity, on top of their regular work responsibilities.
The first problem with providing mentorship through an editorial role is that it’s not a very good mechanism for distributing mentorship. Ideally, one wants to provide mentorship early on in projects, to mentees with similar interests, and to a number of mentees that one is capable of providing good mentorship to. Providing mentorship to everyone who submits an article to Distill is overwhelming. Another problem is that our advice is often too late because the article’s foundation is already set. Finally, many authors don’t realize the amount of effort it takes to publish a Distill article.
Providing mentorship also creates a challenging dual relationship for an editor. They have both the role of closely supporting and championing the author while also having to accept or reject them in the end. We’ve found this to be difficult for both the mentor and mentee.
Finally, the kind of deeply-engaged editing and mentorship that we sometimes provide can often amount to an authorship level contribution, with authors offering co-authorship to editors. This is especially true when an editor was a mentor from early on. In many ways, co-authorship would create healthy incentives, rewarding the editor for spending tens of hours improving the article. But it creates a conflict of interest if the editor is to be an independent decision maker, as the journal format suggests they should be. And even if another editor takes over, it’s a political risk: Distill is sometimes criticized for publishing too many articles with editors as authors.
Another important impact of Distill has been articles written by the editors themselves. Distill’s editorial team consists of volunteer researchers who are deeply excited about explanations and interactive articles and have a long history of doing so. Since the set of people with these interests is small, a non-trivial fraction of Distill’s publications have come from editors. In other cases, authors of existing Distill articles were later invited to become an editor.
Editor articles are sometimes cited as a sign of a kind of corruption for Distill, that Distill is a vehicle for promoting editors. We can see how it might seem dubious for a journal to publish articles by people running it, even if editorial decisions are made by an editor who is at arms-length. This has led Distill to avoid publishing several editor articles despite believing that they are of value to readers.
We believe that editor articles are actually a good thing about Distill. Each one represents an immense amount of effort in trying new things scientific publishing. Given the large volume of readers and the positive informal comments we receive, we suspect that for every critic there are many silent but happy readers.
When a structure turns a public good into an appearance of corruption, it suggests it might not be such a good structure. As editors, we want to share our work with the world in a way that is not seen as corrupt.
The vast majority of Distill articles are written by multiple authors, often from multiple institutions. As a result, an important function of Distill is providing somewhere to publish that isn’t someone’s home turf. If a Distill article were published on one person or organization’s blog, it could lead to a perception that it is primarily theirs and make other authors feel less comfortable with collaboration. Arxiv normally fills this role, but it only supports PDFs.
But it turns out there’s a simpler solution: self publication on one-off websites. David Ha and his collaborators have done a great job demonstrating this, using the Distill template and GitHub pages to self-publish articles (eg. the world models article). In these cases, the articles are standalone rather than being with a particular author or institution.
In many areas of physics, self publishing on Arxiv has become the dominant mode of publication. A great deal of machine learning research is also published on Arxiv. We think this type of self-publication is likely the future for a large fraction of publication, possibly along with alternative models of review that are separated from a publisher.
Journal-led peer review provides many benefits. It can protect against scientific misinformation and non-reproducible results. It can save the research community time by filtering out papers that aren’t worth engaging with. It can provide feedback to junior researchers who may not have other sources of feedback. It can push research groups studying similar topics across institutions to engage with each other’s criticism. And double-blind review may support equity and fairness.
But is traditional journal-led peer review the most effective way to achieve these benefits? And is it worth the enormous costs it imposes on editors, reviewers, authors, and readers?
For example, avoiding scientific errors, non-reproducible results, and misinformation is certainly important. But for every paper where there’s a compelling public interest in avoiding misinformation (eg. papers about COVID), there are thousands of papers whose audience is a handful of the same researchers we ask to perform review. Additionally, it’s not clear how effective peer review actually is at catching errors. We suspect that a structure which focuses on reviewing controversial and important papers would be more effective at this goal. Our experience from discussion articles is that reviewers are willing to spend orders of magnitude more energy when they feel like reviewing a paper genuinely matters to the community, rather than being pro-forma, and their work will be seen as a scientific contribution.
Similarly, we suspect that journal-led review isn’t a very effective way of providing feedback to junior researchers or of promoting equity. These are all very worthy aims, and we’d like to free energy to pursue them in effective ways.
We also think there’s a lot of upside to self-publication. Self-publication can move very fast. It doesn’t require a paper to fit into the scope of an existing journal. It allows for more innovation in the format of the paper, such as using interactive diagrams as Distill does. And it aligns incentives better.Self-publication may align certain incentives better than traditional publishing. Many papers go through an informal review process before they’re submitted to a journal or self-published, with authors soliciting feedback from colleagues. This informal review process is often smoother, faster, and provides more constructive and more relevant feedback than a traditional review process. Why is that? In a normal review process, the authors have the highest stakes, but little agency in the process. Meanwhile, neither the reviewers nor the editors share the authors’ incentive to move quickly. And the reviewers are often horribly over-subscribed. In contrast, in an informal review process, the authors have a strong incentive to quickly organize the process and reviewers are focused on providing helpful feedback to someone they know, rather than arbitrating a gatekeeping decision.
Distill isn’t living up to our standards of author experience. Originally, we had a vision of a much more engaged, responsive, and rapid review process with editors deeply involved in helping authors improve their article. But the truth is that, with us being quite burnt out, our review process has become much slower and more similar to a typical journal. It’s unclear to us whether the value added by our present review process is worth the time costs we impose on authors.
Distill also occupies institutional space, potentially discouraging others from starting similar projects. It’s possible that there are others who could execute something like Distill better than us, but aren’t starting their project because Distill exists.
On the flip side, Distill often comes up in conversations about the future of publishers and journals in machine learning, as a positive example of the role a journal can play. But if we no longer believe in our model, Distill may be unintentionally supporting something we don’t really stand behind. We may also be setting unrealistic aspirations: if Distill’s level of editorial engagement and editing was unsustainable, even with a deeply passionate set of volunteers and a relatively small number of articles, we should at least be clearly communicating how difficult it is.
We think that Distill is a really beautiful artifact which illustrates a vision of scientific publishing. But it is not sustainable for us to continue running the journal in its current form. We think preserving it in its present state is more valuable than diluting it with lower quality editing. We also think that it’s a lot healthier for us and frees up our energy to do new projects that provide value to the community.
We’ve considered trying to find others to hand Distill off to. But a lot of the value of Distill is illustrating a weird and idiosyncratic vision. We think there’s value in preserving Distill’s original flavor. We are open to changes to better structure Distill, but we feel protective of Distill’s vision and quirkiness.
Although Distill is going on hiatus, the Distill template is open source, and we’d love to see others run with it!
Over the last few years, Distill has experienced a significant amount of volunteer burnout. The fact that multiple volunteers experienced burnout makes us think it’s partly caused by the issues described in previous sections.
One of the biggest risk factors in burnout is having conflicting goals, and as the previous sections describe, we’ve had many conflicting goals. We wanted to mentor people, but we also needed to reject them. We wanted to write beautiful articles ourselves, but we also wanted to be an independent venue.
Another significant risk factor is having unachievable goals. We set extremely high standards for ourselves: with early articles, volunteer editors would often spend 50 or more hours improving articles that were submitted to Distill and bringing them up to the level of quality we aspired to. This invisible effort was comparable to the work of writing a short article of one’s own. It wasn’t sustainable, and this left us with a constant sense that we were falling short. A related issue is that we had trouble setting well-defined boundaries of what we felt we owed to authors who submitted to us.
By discussing these challenges, we hope that future projects like Distill will be able to learn from our experiences and find ways to balance these competing values.