ITpreneurs have recently published the results of a survey that they performed on the benefits of ITIL qualifications (The Benefits of Investing in ITIL Advanced Certification, ITpreneurs 2010).
It paints a glowing picture of the benefits that accrue to the individual from taking advanced ITIL qualifications. This isn’t a surprise, since that’s what it stated it was going to do when it was launched – a fact I pointed out to the authors at the time.
To me, this is one of the most dangerous types of research, for the following reasons:
- Vested interest. The survey was sponsored by and effectively driven by, an entity with a strong vested interest in the outcome. Being branded as part of the “ITIL Expert Program” makes it hard not to see it as perhaps biased – and therefore more marketing hype than serious research. Surveys always carry more weight when they are performed by a truly independent body with no such baggage.
- Respondents. The respondents were self-selecting and given the way in which the initial premise was couched, they were far more likely to be positive. Given that the number of respondents in this case was only 160, it is dubious that they are a completely representative sample of the large numbers of service management personnel. Given the number of people who have attained an ITIL qualification, it is a very small sample.
- Imprecision. The collective group is referred to as “ITIL Experts”, though apparently some are V2 Manager qualified and some are V3 Experts. The scope of the V2 and V3 qualifications is radically different, and it is questionable whether one can treat them as a common pool. Equally, there is no indication as to which route any V3 Experts followed in achieving that status, nor of the length of time that has elapsed between them achieving a qualification and realizing any of the claimed benefits. Experience is acknowledged as playing an important part in some of the respondent’s quotes, but there is nothing to identify just how that plays into the picture. Clearly an individual with significant practical experience in a variety of roles and market sectors acquired prior to taking the training and passing the exam, where the qualification is almost “back-filling”, will take quite different things from it than a relative novice where the training is part of their learning curve.
- Subjective information. There is little or no hard data to support many of the claims about increased productivity, etc. It is the respondents’ personal opinion about the impact that the qualification has had on them and their organizations. It would be more meaningful if the manager of an individual had been asked about whether their performance had improved and how that could be attributed to the training received. Similarly, whether peers see them in a different light would be better asked of the peers rather than the subject.
I didn’t participate in the survey, so I don’t know about the exact questions that were asked or how they were couched. It is a well-known fact that one can demonstrate anything that you desire by carefully choosing the questions that you ask and by phrasing them in a particular manner. I experience frustration with almost every survey or questionnaire that I am asked to complete, since they rarely seem to ask the questions that I believe matter and often offer a range of answers that don’t allow me to respond as I feel I want to.
One of my greatest fears about such “research” is how it can become twisted and misused. While with itSMF I produced a slide containing quotes from a number of pieces of research as part of a presentation. I used this in a particular way, not least stressing that the headline statements were questionable since there was no way of assessing the real validity of the claims due to many of the same factors as quoted above. To my horror, that same slide is now being used in presentations by a large number of people, where the information is simply presented as a set of “facts” without any of the caveats that I wrapped around it.
Some folks will say I’m being negative and cynical. I don’t intend to be. What I want is for people to be more and better informed.
With regards to ITIL, I believe it is a valuable piece of material as part of the complex world of service management. Perfect – no! Useful – yes! Similarly, with ITIL qualifications I have concerns about the current structure – a view that others share. Considering that Continual Improvement plays such a prominent role in the latest ITIL iteration, it is strange that debate and the search for improvement is so often seen as negative!
Scepticism isn’t necessarily bad.
++++++++++++++++
Statistics
In ancient times they had no statistics so they had to fall back on lies.
Consumers are statistics. Customers are people.
Definition of Statistics: The science of producing unreliable facts from reliable figures.
Statistics: the only science that enables different experts using the same figures to draw the different conclusions.
Any feedback and comments are always welcome!!
--------------------------------------------------------------------------------
2nd July 2010
Hi Aidan,
Loved your blog on statistics – couldn’t agree more. Thought I would pass along the following song lyrics from Todd Snider and it’s called the Statistician Blues;
They say 3 percent of the people use 5 to 6 percent of their brain
97 percent use 3 percent and the rest goes down the drain
I'll never know which one I am but I'll bet you my last dime
99 percent think we're 3 percent 100 percent of the time
64 percent of all the world's statistics are made up right there on the spot
82.4 percent of people believe 'em whether they're accurate statistics or not
I don't know what you believe but I do know there's no doubt
I need another double shot of something 90 proof
I got too much to think about
They say 92 percent of everything you learned in school was just bullshit you'll never need 84 percent of everything you got you bought to satisfy your greed
Because 90 percent of the world's population links possessions to success
Even though 80 percent of the wealthiest 1 percent of the population
Drinks to an alarming excess
More money, more stress
84 percent of all statisticians truly hate their jobs
They say the average bank robber lives within say about 20 miles of the bank that he robs
There's this little bank not far from here I've been watching now for a while
Lately all I can think about's how bad I wanna go out in style
John Clark
---------------------------------------------------------------------------------------------
.gif)
