Shareware Beach

Monday, 4 April 2005

Online Polls Are Unreliable

Filed under: Conferences — Jan @ 20:52

ESWC 2005 voteOnline polls are quite popular. A webmaster puts up a question with multiple-choice answers. Visitors click on their answer, and a script on the web site tallies them, usually showing the results in real time. While such polls may be fun, they’re not useful for collecting information to base decisions on.

The European Shareware Conference will be held for the 5th time this year. It was held in Cologne the first three years, and in Strasbourg last year. This year, the organizers [I'm not one of them] have made a short-list of five cities: Salzburg (Austria), Brussels (Belgium), Galway (Ireland), Budapest (Hungary) and Utrecht (The Netherlands). Quite a diverse list. During the past two weeks, people could vote for their preferred city in a simple, unrestricted online poll.

The past four years, attendance was around 100 people. When I took the screen shot of the poll results so far (one more day to go), 1671 votes had been cast. That tells me the results are pretty meaningless. Obviously, some people voted multiple times trying to skew the poll results, and other people voted without intending to attend the conference. The poll was announced in several newsgroups and blogs frequented by shareware authors.

The first step to take to get meaningful results is obviously to limit the vote to those people whose meaning you’re interested in. In this case: people who are likely to attend. Not just possible attendees, but likely attendees. That could be people who signed up for the early bird registration, and past attendees. To cast a wider net, one could include people who signed up to receive news by email about the conference, and did so before the vote was announced. The ESWC currently doesn’t have an opt-in announcement mailing list, but I think it would be a good idea, not just for polling purposes.

The second step is to make the poll itself more meaningful. Having people indicate their favorite city is not enough. Suppose half vote for Budapest, and the other half are divided between the other cities. At first blush, the conference should be held in Budapest. But what if the people who didn’t vote for Budapest really don’t want to go there at all? And what if those who voted for Budapest wouldn’t mind to go to any of the other cities? In that situation, Budapest would be the worst choice, since it would reduce attendance in half. While the other cities might not really excite most attendees, at least people would be there. In the end, people attend such conferences for the people and not for the city.

Of course, I just made this up as an example. But these considerations do come into play. As an actual example, I like to combine attending the conference with visiting family back in Belgium. Going to Brussels or Utrecht is a no-brainer. Budapest wouldn’t be too bad, since SN Brussels flies direct. But there’s no direct flight to Salzburg or Galway. Some people won’t care and attend regardless, but others, certainly those with more time and/or money constraints, might decide it’s not worth the trouble or expense.

Attending the European shareware conference definitely is worth the trouble and expense if you’re starting out in the shareware business. It’s a common mistake to think that one’s fledgling shareware business isn’t bringing in enough money yet to afford attending a conference. People just starting out stand to benefit most.

So, instead of simply polling likely attendees, make it a short survey. Instead of picking one city, let people pick a preferred city, and also indicate for the other cities if they would attend in that city or not.

With detailed information obtained from worthwhile sources, an organization can make informed decisions. It’ll be interesting to see what the ESWC board does next.

2 Comments

  1. The “board” must feel that something is wrong it looks like the extended the deadline for the vote(s). Suddently still 11 days to go.
    Also from day 1 Brussels was top and then out of nowere Budapest is climbing and gets the number one spot.
    Also if you look at the difference between April 4th (I suppose you’re number are of that day?) and April 5th (that’s the today) Brussels gets about 250 extra votes….strange…although I would prefer Brussels.

    Comment by Dirk — Tuesday, 5 April 2005 @ 13:30

  2. Yes, I took the screen shot a few hours before I wrote the blog entry. Another oddity is that Utrecht jumped from about 12 votes before the weekend to over 400 votes after the weekend. My guess is there’s a few people driving up the votes for their favorite cities.

    In my opinion, the ESWC board should simply ignore the poll results. They’re clearly meaningless. I’m not part of the organisation, so I don’t know what they’ll do. But if they really want to leave it to a vote, they should require voters to register with name and verified email address.

    Comment by Jan Goyvaerts — Wednesday, 6 April 2005 @ 7:18

Sorry, the comment form is closed at this time.