Quality Approach to Measurements and Metrics: Doing it Right the First Time

Tim W. Steiner

Wilmington University

 

 

 

 

 

 

 

 

 

Metrics are defined as “A quantitative measure of the degree to which a system, component, or process possesses a given attribute. A calculated or composite indicator based upon two or more measures. A quantified measure of the degree to which a system, component, or process possesses a given attribute.” (Ragland, 1995). Metrics are different from measures in that they provide more meaningful information of a measurement in a specified context. For example if a patient has a temperature of 99.1 degrees Fahrenheit that is a measurement. This information alone tells very little about the patient. If say the patient’s temperature is regularly checked and there is a trend in the measurements showing that every hour the patient’s temperature rises this would be a metric. The measurements of the increasing temperature over time show that the patient’s condition is getting worse. This metric is meaningful information that can be used to diagnose the patient and hopefully improve the patient’s condition (Ragland, 1995).

Knowing what and how to measure are key to obtaining effective measurements. It is more than just taking a large quantity of complex measurements, organizations must identify the right measures to use. The main objective of a measurement is to communicate the progress or lack of progress in accomplishing the firm’s goals. Finding the right measurements at the right times can paint a clear picture of the current state of an organization and where that organization needs to go to achieve its corporate goals. Taking timely measurements can produce the best results but deciding when to measure is not always an easy question to answer. Some measurements must be taken continually while others may be dependant on other events. For example, some measurements are dependant on system and network availability, bandwidth and hardware capacity, project state, personnel availability, and man-hour requirements. Measurements need to done on a consistent and repeated basis. This enables effective reporting at the end of a specified period. Just as report cards are distributed at the end of a schools reporting period; reports should be consistently produced at the end of a businesses reporting period (Luftman, 2004).

There are four reasons for measuring processes, products, and resources. First to characterize, this is used to gain understanding of the processes, products, resources, and environments. Characterizing allows establishment of a baseline for comparison and future assessment. Next evaluation is to determine status with respect to plans. Evaluation is also used to assess achievement of quality goals and to observe the impact of technology and process improvements on products and processes. Prediction is used to plan for the future. Projections and estimates based on historical data also help to analyze risks. Improving is the final reason. This is the constant pursuit to make processes and products better. Taking measurements such as quantitative information to help identify roadblocks, root causes, and inefficiencies assist in the improvement process (Luftman, 2004).

Performance is directly related to the quality and effectiveness of measurements. Quality measurements help an organization understand if a process is effective and efficient. Processes can be measured and compared through benchmarking, trend analysis, and satisfaction surveys. Project managers rely on measurement tools to continuously monitor project schedules so that they can be alerted to project difficulties early enough to repair the problems and prevent delays (Luftman, 2004).

In the IT world CIOs often use models such as the 10 step approach to developing measurements, or some other form of benchmarking. The CIO is responsible for taking these measurements and engineering them back into the IT units improving weaknesses and utilizing strengths. The steps in the 10 step approach to developing measurements is designed specifically for IT but the concepts apply many departments. The first step is to establish teams from both IT and business to review and analyze the underlying IT processes. Next the goals of each IT process are defined, these goals should align IT strategy with business strategy. There are critical success factors that are identified, these determine the performance of the IT process in reaching its goals. Guidelines are established to measure if and to what degree the critical success factors have been met. Once these measurements are in place and have been tested, a complete pilot should be done with all team members participating in the pilot. After the pilot is completed successfully and the results are deemed acceptable the recommendation to apply these measurements is forwarded to senior management. Upon managements approval the implementation of the new measures begins. The last step in the process is to analyze the results and this step is an ongoing process of improvement (Luftman, 2004).

In order for any measurements or metrics to be effective they must be of high quality and efficacy. A measurement can be skewed and manipulated to reflect the desired results reflecting an inaccurate situation. There can be manipulation of ROI and other financial measures that can point to inaccurate results. Measurements that forecast trends are the best indicators because these measurements are difficult to manipulate and will paint a clear picture of the actual situation. These measurements are often described as metrics. Peter B. B. Turney states “How wrong we were. It’s poor quality that costs money (and loses customers). Poor quality is doing a job more than once. It’s wasting materials. It’s paying salaries for hordes of inspectors. It’s incurring the cost of warranties and customer returns. And it’s suffering the anger of disgruntled customers… Improving quality is a sure way to reduce cost. Do it right, the first time.” (p.361). It used to be some organization’s view that quality was costly. This perspective was taken by some who look at part of the picture and see that improving quality requires more inspectors and costly warranties. Quality is a process of give and take. Yes, there are substantial costs involved but in the long run quality saves money by doing the job right the first time. Quality measurements show how improvements are performing and justify incremental costs (Luftman, 2004).

Quality measurements and metrics are collected and developed so that indicators can be obtained. Indicators provide insight that enables managers to adjust the process to make things better. Indicators are usually metrics that are compared to a baseline. Back to the example where the trend of the patient’s increasing temperature is a metric showing that the patient’s condition is getting worse. If there was a baseline such as the normal body temperature of 98.6 degrees Fahrenheit to compare the patient’s increasing temperature to, this indicates that not only is the patient’s condition getting worse, but there is a quantifiable difference between the current temperature and the desired temperature. Indicators help us to see where we are, how far this is from the goal, and enable managers to take the required steps in order to reach the goal (Luftman, 2004).

In terms of indicators there are many types but the two types of our focus are process indicators and project indicators. Process indicators enable an organization to gain insight into the efficacy of an existing process. They enable managers to assess what works and what does not. The intent is to provide process indicators that lead to long term software process improvement. Project indicators enable a project manager to assess the status of an ongoing project, track risks, uncover potential problem areas, adjust workflow, and evaluate the project teams’ abilities. Having clear indicators help in the decision making process and enable managers to maintain control. Tom DeMarco, a metrics expert states that “if you don’t measure, then you’re left with only one reason to believe you are still in control: hysterical optimism (“The Foundation of Information Security”, 2009).

Many programs fail due to poorly defined objectives and having the wrong approach to measurement. There are a number of approaches for defining measurable goals. The most well known approaches are Goal-Question-Metrics (GQM), Quality Function Deployment Approch (QFD), and Software Quality Metrics (SQM). The fundamental idea of GQM is to set goals specific to the needs in terms of purpose, perspective, and environment. Refine the goals into quantifiable and traceable questions. Deduce metrics and data to be collected this is essentially answering the questions. The QFD approach is a technique that evolved from Total Quality Management principles and aims at deriving indicators from a user’s point of view. This method uses simple matrices with values weighted according to the judgments of the customer. The SQM approach was developed to allow customers to assess the product being developed. A set of quality factors are defined for the final product, the factors are refined into a criteria, and this criteria is further refined into a set of metrics (“Measurement Frameworks and Standards”, 1999).

Measurements, metrics, and indicators when used effectively in a goal-oriented approach can help to better define the processes and ultimately determine the organizations maturity level. The CMM capability maturity model (Whitten, Bentley, & Dittman, 2004) is “A standardized framework for assessing the maturity level of an organization’s information systems development and management processes and products. It consists of five levels of maturity” (p. 84). In the CMM each level builds and expands on the level preceding it. Most organizations start at Level 1. At this level projects don’t follow a consistent process. They may be unpredictable and tumultuous. Each team follows its own methods without regard for other teams. Projects typically encounter crises, are over budget, and behind schedule. Documentation is inconsistent which creates problems for anyone trying to manage the system (Whitten, Bentley, & Dittman, 2004).

In Level 2 the focus is on project management. Project management procedures are established to track project functionality, schedules, and cost. The system development process may vary from project to project but a process is always followed. Level 2 develops the foundation for the standardization that occurs in Level 3 (Whitten, Bentley, & Dittman, 2004). 

Level 3 develops the system development process into a standard. All projects use the standard process to develop and maintain information systems and software. Consistently using standards will result in high-quality documentation and deliverables. The process is repeatable, stable, and predictable (Whitten, Bentley, & Dittman, 2004).

Measurable goals are set for quality and productivity in Level 4. The measures are collected routinely and stored in a database. This data is used to improve project management. Whenever a project encounters a problem, the process can be adjusted based on predictable and measurable impacts (Whitten, Bentley, & Dittman, 2004).

Level 5 consists of continually monitoring and improving based on the measures and data analysis that was developed in Level 4. Technology is changed as needed and the process itself can be adjusted to meet requirements. Lessons learned are shared across the organization with emphasis on eliminating inefficiencies while sustaining quality (Whitten, Bentley, & Dittman, 2004).

            The CMM shows that organizational maturity is essential for development of processes and products. It follows that organizational maturity would be essential to producing quality solutions. Organizational maturity stimulates growth, increases operational efficiency, allows effective management, and ensures quality.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

References

(1999). Measurement Frameworks and Standards. Retrieved Jun 22, 2009, from http://www.dcs.qmul.ac.uk/~norman/papers/qa_metrics_article/section_7_standards.html

(2008). Systems development life cycle (SDLC) resources from MKS. Retrieved June 22, 2009, from http://www.mks.com/sdlc

(2009). The Foundation of Information Security. Retrieved May 25, 2009, from http://www.safenet-inc.com/

Luftman, J. (2004). New Upper Saddle River, NJ: Person Education Inc.

Ragland, B. (1995) Measure, Metric, or Indicator. Retrieved June 22, 2009, from http://www.stsc.hill.af.mil/crosstalk/1995/03/Measure.asp

Whitten, J. L., Bentley, L. D., & Dittman, K. C. (2004). New York, NY: McGraw-Hill Companies, Inc.