Conquering the hurdles of managing IT production

October 3, 2009 Digest

I was recently speaking with Andy Howell about his training plans for the next few years. Andy (see breakout box) has a long-term dream to reach the 2012 Olympics. Andy’s photo is a powerful marketing image. But as I thought about what Andy was trying to achieve, I was aware of deeper similarities between his world, and the world of IT production management.

Andy’s goal is initially to reduce his personal best (PB) time in the 400 meters hurdles, in order to qualify for the GB Olympic team. So keeping track of timings is essential. But it goes beyond that. How much time he puts into different training tasks in his regime, how many calories he eats, how much sleep he gets – these are all factors in achieving his goal.

In managing IT, particularly the area of IT production (the day-to-day, business as usual, or “BAU” tasks), we sometimes forget to have meaningful management metrics.

It is a basic truism that you cannot manage what you cannot measure. As with all simplifications, this is not entirely correct, of course; the key soft skills that distinguish a good manager from a mediocre team leader are difficult to quantify. Nevertheless, it is correct to say that a significant proportion of our work as IT managers needs to be clearly quantified.

Metrics, in the sense of management information, can be used for two purposes: to run the department, and to broadcast the achievements. As far as running the department is concerned, it is essential that IT production managers have metrics in place to capture what infrastructure the department is responsible for, what they are achieving and what resources are being consumed in order to reach their goals.

Basic raw metrics, such as the numbers of servers in the data centre, their applications, uses and roles, are essential in order to understand where resources are being used. This information, properly presented, is also very valuable in demonstrating the ‘value add’ of IT to the business and sponsors. This can be extremely useful during the annual budget negotiations!

Equally valuable for an IT Production department is a basic timesheet system. This helps managers identify ‘problem applications’ which are consuming excessive resources. Such knowledge helps facilitate discussions with development teams or business managers to identify efficiencies or alternative solutions.

Just as Andy and his coach pay minute detail to every aspect of his hurdling (stride patterns etc), so as IT production managers we need to have information at our fingertips to help improve our departmental performance.

No athlete these days would consider taking part without the necessary equipment (running shoes, clothing etc), and the understanding of how to use them. Similarly, the necessary software tools are essential to managing IT production, and training and experienced knowledge are essential in gaining the potential benefit.

Unfortunately, software can be relegated to the role of ‘shelf ware’ if its use within the organisation is not applied appropriately. Just as an athlete has to carefully watch his progress and be able to anticipate situations that may lead to injury, so as IT managers we need to have monitoring in place for systems to identify potential issues (shortage of disk space etc) ahead of time.

Although the use of specialised running shoes and starting blocks was once considered a luxury, no athlete today would run without them. In the IT world, we are unable to run Production efficiently without our bread-and-butter operational software tools (backup management, patch release management, monitoring and alerting, remote console management, performance monitoring etc). And yet, we frequently deploy these tools in a relatively unsophisticated, uncontrolled way, without considering an overall operational tools strategy. As a result, we sometimes have tools that are using inconsistent naming conventions, reporting on common systems in different ways.

How many times have we struggled with understanding what is the ‘correct’ answer to the questions about what servers are used for what application? We need to have a clear ‘referential’ or ‘master’ source of data for each relevant piece of information, be it definitive name for every server or application, to naming conventions for SAN storage.

Use of the right equipment, in a structured, strategic way, is important in trying to reach our goals, be they 400 metres of hurdles, or 400 Linux servers.

Athletes train hard and work hard. And we all aspire to a working life that is ‘smarter, not harder’. This concept is often misunderstood, however. Working smarter means eliminating ‘re-do’. Good athletes develop a rhythm, where every movement contributes in a smooth way to the overall result. There is no wasted effort, no need to continually correct or change course.

In the IT world, we also need to deliver our services more efficiently, in a predictable, repeatable way. Hence, IT production managers rightly stress the importance of relevant and appropriate processes and procedures. But, like our athlete, we need to keep our end-goal in mind. Beautifully crafted processes are only valuable if they enable the organisation to deliver the correct service in the most appropriate way.

Other people have struggled with similar challenges to us. Process improvement methodologies from industry generally (such as six-sigma) can aid us to achieve the speed and efficiency that we all want to deliver.

Although I work with consultants who are frequently qualified in the ITIL approach to processes and procedures, we always stress with clients and colleagues that such processes must be implemented in a pragmatic way. Properly implemented, good practices, policies, processes and procedures are means to help us achieve our goals, rather than barriers to prevent us doing so.

Although Andy is already a very fast sprinter, he has chosen to focus his attention on hurdling at just one distance – 400 meters. By concentrating on his ‘core skills’ he is not distracted by other challenges.

In the world of IT production, however, we are continually faced with alternative challenges to our ‘core skills’. Most data centres these days are full of servers of all sorts of make, manufacture, operating system and configuration. A typical organisation today may have systems from IBM, Sun, HP / Compaq, Dell, not to mention the multitude of other manufacturers. Not only at the hardware level, but also in the choice of operating system, database, middleware and messaging system, the technology mix proliferates. All of this diverse heterogeneous technology has to be supported and managed. And the more diverse the mixture, the greater the support costs.

Unfortunately, the choice of new technologies can often be made without taking into account the ability of the IT production team to support the solution, once it is deployed. The solution is to encourage IT production teams to focus on ‘production supportable’ technologies. What technologies are they able to manage and support with their existing skill sets?

All of this needs to be part of an overall IT production strategy or roadmap, identifying the technologies which are supportable in production and those which are not (or where support would become unduly expensive to the business). As with athletics, we can be more successful managing IT production systems if we focus on our core skills, instead of losing our competitive edge by spreading our commitments too widely.

Andy will be the first person to admit that his success is not solely due to his own efforts. The achievements he has already had in the AAA championships, running for Harrow, Middlesex and England, are also down to his coach and the support teams who work with him.

Coaches can see the mistakes that are being made, often when you are not aware of them yourself, and help you improve your performance. In IT production management as well, an independent consultant can help clients identify the shortcomings in metrics, in operational tools, in processes and in technology standards, and advise them where their own performance can be improved.

Maybe that’s where I came in.