Failure of the Survey: iPhone 3GS and Pre return/exchange rates compared


Surveys can be conducted with a variety of outcomes in mind. One might be to learn about a product or service; it could also be used to find out a personal information. A popular reason to conduct surveys, however, is to prove pre-conceived notions; unfortunately, this sort of survey is the tool of marketers and turn-coat surveyors – integrity isn’t a strong suit of many surveys. For instance, if TMA conducted a survey of how satisfied readers were with our articles and, at the same time, offered a free headphone, to a ‘randomly selected’ individual, the survey would be flawed.

A recent one approached 3GS and Pre users in order to ascertain which one suffered higher return/exchange rates. I have issues with it on many levels and I hope that you will read along.

Why this article is important to point out is threefold. Firstly, the 3GS is a 3rd generation product. Apple’s phone isn’t being compared in its maiden form to the fresh-to-market Pre. No, rather the Pre, a brand new device, is tested against a product which has gone through 2 years of hardware and software tweaks. Hardware generations can make a huge difference on how well a product performs. Go to Apple Discussions to see the durability of first-generation hardware (more on the flaw of relying on forums later).

Secondly, the survey which was referenced by an ‘analyst’ was conducted from an impromptu poll at pre central, a Palm Pre dedicated forum. While that can have many implications, one of the most important is that a hardware-dedicated forum is often a haven for people who have troubles. Discussion forums like Apple’s are proof of this. So, it isn’t newsworthy that at a Pre centred forum, the Palm device suffered high return rates.

Finally, there is the issue of surveyed population. The final survey proved a much smaller return rate for the Pre. I judge that though the smaller rates are probably closer to reality, they miss the mark. Firstly, a survey conducted in response to a previous survey likely engineered for different results, to prove a hunch or prejudice . If the first number is embarrassing, a second survey, which will retouch the result, can eschewed the same result. Pre central took a different survey, this time with the first and the analyst’s results in mind. Members could plug in their personal results based on the following format:

Yes 17%
Yes, more than once 9%
Nope 59%
I don’t have a pre 15%

The methodology and results of the second survey are above, and the earlier survey, taken at the same forum is here. The second survey is taken from the same forum, but conducted from the homepage of Pre Central rather than buried in a specific forum. Tto my estimates, it reveals about 44% return rate. The final number, 15% (users who don’t have a Pre), shouldn’t be counted as part of the result. But still, I am not satisfied with the number which is embedded in a Pre forum and returns a higher value.

However, a final ‘survey’ (PDF) (if I may call it that), revealed the lowest yet return results: 11% for the Pre and 7% for the iPhone 3GS. Survey participants, however, are too few to prove anything: 192 3GS users and 38 Pre users. Such a small number of participants raises these questions: how lazy was ChangeWave Research; and did they conduct the survey among real, living, breathing humans, or at their ‘survey branch’, in the next building?

One this is sure: the Pre and the 3GS are destined by their marketing teams to be at each other’s throats the entire time. Apple’s hegemony will keep it safe from the Pre – I cannot forsee the Pre toppling any significant population of Apple customers – but Palm’s tactic may need to change. A market with fewer players is bad, and though I never intend on owning a Pre, losing it would be very bad for this incredibly tired smartphone race.

The article which prompted this rant? [via Brainstorm Tech]

Next ArticleChillingo's Inkvaders is OUT! Time to load up on paper bullets