People’s Choice Vote – the final score

Yesterday the LinkedUp Project announced the winners of its Vici Competition, the third and final competition in the LinkedUp Challenge. All 3 competitions have involved the judging of software prototypes and apps that build on open education data using an extensively researched and refined evaluation framework. They have also all included an ‘online open vote’.

Winners of the Vici Competition

Winners of the Vici Competition

The process of implementing the People’s Choice Award has been a bit of a roller coaster ride and I’m keen to share our experiences.

I wanted this assessment of our approaches to be both open and honest, and so it feels appropriate to publish it here – on my own personal blog – rather than on the LinkedUp site, or elsewhere.

Some Background…

Before the launch of the first competition I was in discussion with colleagues at Open Knowledge (where I’m based) and we agreed that although the evaluation framework is a fair way to judge an open data competition it is not a very inclusive process. Only an elite group of reviewers get to judge the entries – everyone else’s views are discounted. Community building within LinkedUp is very much about building new audiences so we came up with the idea of the People’s Choice Vote – a way to involve everyone and anyone in the judging of the competition. It was agreed that the People’s Choice Vote would be run in parallel with the evaluation framework assessment giving competition entrants an opportunity to promote their submission, and external people an opportunity to be involved in the voting process. In my role as dissemination and community building co-ordinator it seemed like an interesting approach to take and and the consortium agreed that it was something we could spend time on. Maybe at this stage we were naive in our thinking that it would be easy to run such a vote.

Although I’d had some experience of running online votes before (mainly related to chosing sessions for a conference I’d been involved in) we did recognise that the People’s Vote would be some extent an experiment. We were unfamiliar with the currently available software used and had only a vague idea of the challenges such a vote would pose. However the LinkedUp Project is partially a research project so this level of experimentation seemed reasonable providing we share our experiences; it was agreed that it was a risk worth taking. Hopefully our insights will prove to be a useful resource for others planning to run a similar open vote in the future.

Veni Vote

For the Veni Competition I carried out quite a bit of investigation into who was running online voting and the type of solutions involved. In the end we went with Ideascale. The setup worked well but unfortunately our freemium version meant that while voters could only vote once on each idea they could vote down as well as up using a dislike button. This meant that some competitors and their supporters were coming to the site and voting everything down except for their own entry. Naturally this upset many of the competitors and to ensure positive results we added in extra opportunities to physically vote at the LinkedUp poster session at the OKConference in September 2013, venue for the Veni awards. The competitors who were causing the problems also ended up getting eventually locked out of the system.

Veni Vote in Ideascale

Veni Vote in Ideascale

I documented my thoughts on the Veni vote in the following blog post: Online Voting: The highs and lows.

Vidi Vote

It was clear that the service to be used for the Vidi Competition Vote had to only allow positive voting. We were offered an alternative software solution: GNOSS, a software platform, created by RIAM Intelearning LAB S.L. who specialise in online social networks with dynamic semantic publishing. The GNOSS team were actually shortlisted for the Veni Competition so had some understanding of what was required for the People’s Choice. The system (described in this post) was much more complex than Ideascale and to prevent vote rigging it required that voters register with the GNOSS site or use an already established account. The GNOSS setup is still available online.

Vidi People's choice in GNOSS

Vidi Vote in GNOSS

The system, whilst being more secure, received some criticism for its complexity. Loging in was either via Facebook, Twitter, or a new GNOSS registration. During this process the interface would change language (from English to Spanish at regular intervals) so for example you were required to make selections related to the community you wanted to participate in with little explanation.

The GNOSS solution was definitely better, but still didn’t tick all the boxes for us.

Vici Vote

Despite being out third effort and the feeling that we should almost be there by now, the Vici People’s Choice was by far the most complicated.

In response to the feedback that the GNOSS system had been overly complicated we opted to use a dedicated, yet fairly simplistic online voting app. The app was called Vote Contest was created by Wishpond and for the first time during the LinkedUp Challenge we actually paid for the software. It’s simplicity was it’s appeal and we were able to embed it in our website. You can read more about the Wishpond system in the Vici People’s Choice launch post.

Vici People's Vote in Vote Contest from Wishpond

Vici Vote in Vote Contest from Wishpond

Unfortunately it became clear fairly early on that there were problems with the software. The mobile version of the site had a bug that enabled people to refresh the page and then vote again with the same email address. There was also no email verification and as the Mobile version failed to remember the MAC address this resulted in opportunities to game the system. After a few days and discussions with the Wishpond team (who initially denied the problem and then admitted that they were unable to fix it) we were left with broken software and scores that were effectively null and void. After a consortium meeting an executive decision was made that due to a relatively short-timescale there would not be a chance to investigate and implement alternative online voting solutions. The only option was to cancel the online vote completely. Not wanting to let the Vici entrants down completely it was decided that we would replace the online People’s Choice Vote with an offline People’s Choice Vote. The People’s Choice winner will be identified during a physical People’s Choice Vote that would take place at the poster session at ISWC in Italy – where the Vici awards were to be held.

We were put in a very difficult position and it was clear that whatever decision was made would cause some upset. Many of the competition entrants complained but came to realise that we had been left with no other options.

Tokens for the Vici People's Choice

Tokens for the Vici People’s Choic

Running a physical People’s Vote at ISWC was an arduous task. All ISWC attendees received a token (shown above) in their ISWC delegate bags. This involved the creation of 650 ‘tokens’ (pieces of card with the details of the vote on). The tokens had a space for people to write their name and their entry on – the thinking behind this was that it would involve competitors ‘borrowing’ tokens from unused bags. The Vici Challenge entrants were invited to showcase their entries and their posters during the ISWC minute madness session encouraging delegates to come and vote for them. The posters were then on display in a dedicated Poster and Demonstration session and delegates were able to vote for their favourite submission by placing your token in the relevant poster bag.

At ISWC alongside the LinkedUp Challenge there was also a best poster competition. Over 130 posters were competing in the competition and delegates could vote for their favourite poster using the ISWC website or phone app – this involved using a code number placed in their delegate bags – again more ways to avoid gaming the system. It was clear that with so many choices delegates were at voting saturation level. Although over 100 votes were placed for the LinkedUp Vici Challenge it was a relatively small sample of the 650 attendees.

Conclusions

Over the last year the vote process has given me extensive food for thought around our innate tendency to compete, and at times cheat. Any voting system has to be set up with this in mind.

In hindsight the time and effort in running a successful People’s Choice Vote was underestimated. Software solutions need to be well tested by a committed team looking for loopholes and opportunities to break the system. They also need to be tried out on multiple devices. While all the LinkedUp People’s Choice votes has been marred with attempts to game the system I still glad that we run it and believe that it was established with the right motivation in mind. LinkedUp has attempted to be inclusive and innovative in its approaches to raising the profile of open data in education. In fulfilling this task we have worked hard as a consortium to reach out to new audiences and include the general public when possible.

At this year’s ISWC the complexities of running the People’s Vote seemed overwhelming and at times I felt like we’d let entrants to the competition down. However one shortlisted candidate kindly pointed out that it is impossible to keep everyone happy but if you try your best and your motivation is good then you just have accept that there is little more you can do.

Making things ‘open’ often highlights flaws in the system. These flaws may be with a service or device, or sometimes they are human flaws – the need to win at all costs. It’s unsurprising that online election voting has yet to reach fruition! However I still believe that transparency is a worthy approach and hope that those who competed in the LinkedUp Challenge will recognise that it was in pursuit of the goal of openness that we made our mistakes.

10 point specification for open voting software

Experience has shown us that to fully work an online voting system will need to consider the following points:

  1. Email validation – those with a creative mind can make up emails, there needs to be some way of proving that a voter is an individual.
  2. One person = one vote – basically the ability to block voters after one vote.
  3. Only allow positive voting – having negative votes is depressing for all.
  4. Random listings of entries – people will often vote for what is at the top of the, random listing of entries goes to some lengths to ensure more even voting.
  5. Urls links for entries – it’s not totally necessary but it can be useful to have one url per entry – this allows easier promotion entries.
  6. Easy to use – If it isn’t easy users will give up before voting.
  7. The ability to add tags – not necessary but allows you to do interesting things with the content.
  8. The ability to add images and links in the description – some software will only let you add plain text.
  9. Embedding into your website, Facebook etc. – versatile services will let you embed the app and not require users to go to a separate website.
  10. Test your software – test it, and then test it, and test it again on lots of different devices.

It is also worth thinking about openness – how open do you want your system to be? Could you run it on Facebook or as a smart phone app or do you feel that is excluding people?

Some current open voting systems you could try

  • Google docs – you could create a poll
  • GNOSS – see Veni!
  • Hackdash.org – they have some great voting and collaboration software going on
  • Facebook Like count linked to descriptions of items. Each user is required to have a facebook account to vote and a “like” would be consider a vote, and by design, each user can only like something once.
  • Polldaddy plugin for wordpress
  • Opinionstage plugin
  • Jisc Elevator used a drupal module – not possible for us
  • Ideascale – see Veni vote!
    Simply voting – costs, looks complex
  • Estonia released its national e-voting system on github – it’s probably overkill for a competition though, fairly complex
  • SnapSurvey – costs, very corporate
  • Election buddy – costs, maybe overkill
  • Eballot – costs

Related blog posts