Beyond being one of the founders of Nelio, I’ve been working as a researcher in software engineering (currently leading the AtlanMod research team, though this is going to change soon) for the last 13 years.
As such I’ve submitted over 200 research proposals, or “papers” as we call them, to get my work accepted in the conferences and journals in my area (curious about what I do research on? read about it). Some of them have been accepted, some have not. It’s worth noting that many of those accepted were not accepted in the first attempt but after improving and resubmitting them.
When I started submitting proposals to WordPress-related events I was expecting to go through a similar process to the one I knew from research conferences. Well, that’s not the case. And it disappointed me.
The selection process for speakers in WordCamps is anything but open
The research community is strongly pushing publishers to switch to an open access (unrestricted online access to published works) mode as part of a much bigger push for open science (initiative to make all aspects of scientific research accessible to everybody, not just completed and accepted research results), though unfortunately we are still far from it (you can read some of the reasons why this is happening here and here), specially if we compare it with the degree of openness we usually see around open source software development projects.
In many respects, WordPress is a model project on this. Not only the code is freely available but also all information on its roadmap, bug and feature discussions, lead developers, etc is open to everybody (both to consult and participate) in the WordPress trac. But there is at least one exception to all this openness and transparency around WordPress: the selection of speakers to talk at WordCamp and other WordPress events (that behave as WordCamps even if they miss the official stamp).
Myself and other Nelio members have applied to a number of WordPress events in the last two years (including WordCamp Europe 2014 and 2015). Never we received any feedback on our proposal (neither for rejected nor for accepted ones, that we’ve also had some, don’t think this is an angry post by a loser complaining) or any other input on how the selection process took place. Without feedback there is no way to know what to improve for future editions. For instance, this is basically everything they said in the rejection email for WordCamp Europe 2015 (and similarly for the acceptance one)
Thank you for your application to speak at WordCamp Europe 2015. We had 171 applications this year, most of which were of an extremely high quality. It was difficult to select the final group of speakers, and I’m sorry to have to tell you that your application was not successful.
You may think this is not a big deal but I strongly disagree. Even if WordCamp talks shouldn’t be about promoting your products, in a highly competitive market like the WordPress one, speaking at WordPress events helps building your personal/company brand which is one of the best ways to gain the trust of potential customers and distinguishing yourself from your competitors (obviously, on top of all other benefits you’ll get speaking or just simply attending a WordCamp 🙂 ). And I think this is not only important for the potential speakers, also attendees deserve to know who and how selected the talks they are going to see at the event.
There are many WordCamps so it could be that my experience is not that representative. Still, I couldn’t find any indication on the selection process in the official WordCamp Planning page, so unless I missed something, WordCamps are not forced to be open nor use any particular kind of transparent peer-review process. And by open I mean that the rules to select speakers are public and explicit , even if it’s just to say something like “all speakers will be directly invited by the organizers at their own will”. I’d also like them to be peer reviewed.
The selection process for speakers in WordCamps is not peer reviewed
Peer review is the standard selection process used by all research conferences. From wikipedia
is the evaluation of work by one or more people of similar competence to the producers of the work (peers). It constitutes a form of self-regulation by qualified members of a profession within the relevant field. Peer review methods are employed to maintain standards of quality, improve performance, and provide credibility. In academia peer review is often used to determine an academic paper’s suitability for publication.
A basic peer review schema for a computer science research conference involves first the appointment of a “program committee chair” (renowned member of the community, independent from the organization team, and appointed by the conference steering committee for one specific edition of the conference). The PC chair invites then a list of PC members that will take care of reviewing and evaluating the proposals. Each member is assigned a subset of the proposals and each proposal is evaluated by typically 3 PC members. In a final round, PC members compare and discuss their respective evaluation and try to reach a consensus monitored by the PC chair. Last step is notifying the authors of the decision, including the full text of the reviews. Typically, the names of the specific revieweres for the submission are hidden (which supposedly allows reviewers to be more honest with their evaluation).
If we compare this selection process with what I’ve observed in WordCamps I think you’ll quickly see the difference:
[table]
Comparison, Conferences, WordCamps
Who makes the final decision?, Program Committee Chair, ?
Who reviews the proposals?, Program Committee, ?
Number of reviewers, >3, ?
What is evaluated, full papers (around 5.000 words), short abstract talks
Feedback for authors, text of the reviews, X
Invited talks?, Yes (but no more than one per day), Yes (many?)
[/table]
Yes, peer review is far from perfect but there are many initiatives to improve it (mentoring schemas for newcomers/young students, adding a rebuttal phase giving authors a chance to reply to criticisms from reviewers, adoption of a two-tier structure: program boards and program committee members to better monitor discussion and quality of reviews, …).
And yes, it’s also a huge time investment by a large number of people. To give you an example, the selection committee for the models conference that should get a similar number of submissions to WCEU 2015 involves around 60 people that will read and discuss the proposals during a 3-month period including a two-days face-to-face meeting with some of them to make the final decision.
But so far the community has not been able to come up with a better schema. So I’m sure WordPress events can learn a few things from it.
Nelio Unlocker
Switch to WordPress safely while keeping your design and content
Improve your SEO today and boost your site speed by converting your pages into HTML, CSS, and WordPress standards. With zero technical knowledge required, you’ll only pay for what you need.
My take: selection of speakers must be open and ideally also peer-reviewed
The TL;DR version of this post would be:
- I believe we need to force all WordCamps to make public the rules they use to select the talks to be part of the event. Giving feedback to all submitters would also be appreciated. This information should be part of the approval process to get the official WordCamp stamp
- I’m not asking WordCamps to adhere to a full peer-review policy similar to what research conferences do. I understand this may not be feasible. But for sure there’s some middle point between that and what is happening now. For instance, having an official selection committee and a chair for that committee in charge of making the final accept/reject decisions would already be an important first step
I’d love to know your thoughts on this. Feel free to tell me that I’m barking up the wrong tree!
Featured image by opensourceway
Leave a Reply