January/February
2005

Printer
Friendly
Email
this story

Columns
A
New Metrics System for IT

How
Cisco is approaching the Holy Grail of truly measuring IT
value

By
Ted Smalley Bowen
CIO
U.S.

THE OLD EFFORT to figure out
IT's impact on organisational productivity has new life. Leaving macroeconomic
theories to others, some CIOs are applying IT productivity measures to
operations at their own companies. At Cisco Systems, for instance, a
cross-departmental council has been working to hash out metrics for improving
business process operations, according to CIO Brad Boston.

In 2003,
the council developed metrics to evaluate the efficiency of its online order
processing. Less than 30 percent of orders were getting automatically routed to
manufacturing, since high error rates necessitated manual input, according to
Boston. "We mapped the process of orders coming in and developed metrics. We
optimised the process, put new tools in and doubled the percentage of orders
that went directly to manufacturing within six to nine months," he
says.

Cisco also
relies heavily on IT productivity metrics for budgeting, particularly in
negotiating contract labour. "We use [productivity measures] for budgeting for
development projects to decide how much to do with internal staff and how much
with contract labour," he says. "And we do follow-up evaluations of the actual
delivery at total cost."

At a time
when CIOs are under greater pressure than ever to prove the value of IT,
productivity metrics that measure the "true" value of IT are the Holy Grail.
When it comes to IT, productivity is everywhere and nowhere. The axiom still
holds that the effects of IT show up everywhere except in productivity
statistics, to paraphrase economist Robert Solow.

CIOs are
inching towards better high-level measures that relate to overall business
productivity, but they are finding it easier to collect empirical data about
throughput, uptime, service levels and the like. A new twist in productivity
metrics is the brave new world of business process design and analysis. CIOs are
using (and in some cases creating) performance metrics to help their
organisations get a better handle on partnerships and service agreements, and to
claim (or keep) a place at the strategic planning table.

CIOs say
that these emerging "value" metrics—so termed because they gauge the value that
IT contributes to the enterprise—are helping them to better manage their
operations, and to communicate and coordinate with line-of-business and senior
management.

Value
metrics occupy the intersection of hard and soft performance measures, combining
observation and intuition. "Most clients I work with are shifting their focus
away from internal [IT productivity] measures and more onto value measures,"
says Craig Symons, a principal analyst with Forrester Research. "They're asking,
What value are we contributing to revenue growth, to being able to roll out new
products and services, to cut the time to do those things, add to customer
satisfaction and retention and cut costs?"

Productivity's Common Currency
While
acknowledging the limitations of productivity metrics, CIOs use them to
strategise, budget, weigh outsourcing decisions, run (and terminate) projects
and communicate with other senior executives.

Unsurprisingly, technology-sector CIOs tend to be more
advanced in their thinking about productivity metrics than their peers in other
industries.

"Directly
or indirectly, probably 10 to 15 percent of my time is tied up in this," says
Intel Vice President and CIO Doug Busch. "A huge fraction of what we do in IT is
aimed at improving productivity, either task productivity or knowledge worker
productivity. [But] I think as an industry we're kind of thrashing about on this
problem."

For Cisco's
Boston, the figure is closer to 20 percent. "Some of that is focused on the cost
of contract labour, but it's mostly making business more productive," he
says.

CIOs are
awash in performance data: internal IT measures of hardware, software,
network—not to mention personnel, departmental and organisational ratings of IT
services, vendor and service provider claims (and actual
results).

Traditional
high-level measures of productivity include IT budget as a percent of total
revenue, typically compared with peer organisations in the same industry and the
ratio of IT staff to total employees. Other staffing ratios count servers or PCs
per support person.

How Many Programmers Can Dance on the Head of a
Pin?

It's always
been easy to lose the forest for the trees when considering productivity. But
especially now that IT statistics are so easy to come by, CIOs have to think
carefully about what they want to get out of the metrics.

"Personally, I couldn't care less how many lines of code, how
many tasks it runs," says Sinclair Stockman, CIO of British Telecom. "If the
system is coming out earlier than they told me it would, and I'm able to deliver
more benefit than I thought I could, and it's going to cost me less, I'm very
happy."

According
to Stockman, getting the most out of IT systems depends on "the efficiency and
effectiveness of the process change, the training process, and the ruthlessness
with which you actually go after the business benefits that were delivered as a
result of the systems being deployed."

This casts
productivity in terms of business change. The trick is relating individual
project-level performance to bottom-line results.

"Between
small teams and the very top level where you're looking at the business benefits
of massive programs, there's this massive chasm," says
Stockman.

The
one-to-one communication that's required at the top level of an organisation
often injects less quantifiable elements into the productivity mix, according to
Andy Rowsell-Jones, a Gartner vice president and research
director.

IT
productivity depends on the CIO's personal standing and political skills, he
says. "You've got to have that relationship piece and credibility piece. Do you
get asked about non-IT things, invited to product team meetings as a possible
source of ideas? These are soft, squishy metrics. But at that level, the
personal relationship is politically important."

If a CIO's
executive relationships are weak, "there's a greater chance that IT gets reduced
to an equation of cost of service," he says. "Then you've got to look out
because, for accountants, service is never good enough and cost is always too
high."

Pitfalls of Productivity
Ironically, the
preoccupation with productivity can be a drain on productivity. "We don't have a
good way to [calculate IT productivity], either to forecast or to measure,"
Intel's Busch says. "As a result, we end up justifying and rejustifying, and
analysing and reanalysing stuff, and coming at it from different angles and
trying to persuade people with different measures."

When it
comes to rating knowledge workers, simple formulae tend to miss the
point.

"I don't
think personnel appraisals are as dependent on productivity metrics," Busch
says. "We measure people on results rather than on unit output per period of
time. At one level you can say, Yes, productivity has a huge impact on that. But
it's kind of an indirect impact.

"Making a
smart decision that produces the best result in the shortest period of time is
really what we're looking for rather than how fast you turn a crank," he
says.

One of the
problems in quantifying the role of IT in business is the one-off nature of so
many projects and processes.

"It may
sound a little philosophical, but I'm not sure whether you can talk about
productivity in a world where you're not actually talking about repeatable
tasks," BT's Stockman says.

While it
may be relatively straightforward to gauge the productivity of offshore software
houses that bang out low-level code, it's tougher to quantify the business
benefit delivered by an in-house IT team.

If metrics
are unable to capture that value, it can prejudice managers toward
outsourcing.

Measuring
productivity of in-house IT boils down to tying it to business change. "You need
to ensure that there are programmes of activity in place which actually effect
business benefit improvements for the company, and there are some measures you
can apply to that," Stockman says. "How much business benefit have you
delivered? How many people did it take you to do that? What was the level of
investment that you put in? What caused it to be good? What caused it to be
bad?"

Nonetheless, the more closely the IT organisation participates
in business process design, the harder it becomes to distinguish its
contributions.

"One of the
most fundamental complications in this whole productivity question is: What's
the impact of business process and what is the impact of the tools that enable
the business process?" Busch says. "It's very difficult to disentangle the two,
and I frankly have come to the conclusion that you shouldn't even try to. You
ought to look at them as a whole."

Looking
ahead, CIOs say they anticipate—sometime in the next five years or so—to see
some standardised performance metrics that more closely tie IT to business
value. Productivity metrics are morphing to fit the environment. The key is
finding a way to establish a productivity baseline, according to
Busch.

"I don't
think we're very close to it," he says. "It's hard to tie [economic measurement
of companies' results] back to individual project activity in a lot of cases. We
need something that's much more trackable."

Ted Smalley Bowen is a Boston-based freelancer covering technology.

|  Contact
Us  |  Privacy
Policy  |  Conditions
of Use  |  ^TOP

Website hosted by Pacific Internet