The listed performance measurement systems identified are as diverse as health care organizations themselves and were developed for a variety of purposes by for-profit and not-for-profit groups from every segment of the health care industry. These systems do not encompass all performance measurement systems operating today. The listed systems are among those that requested review by the Advisory Council on Performance Measurement and were found to meet the Joint Commission's initial screening criteria. Additional systems will likely be added to this listing over the next several years. Systems that do not continue to meet Joint Commission criteria will be deleted from the list. The increasing number and diversity of available systems can make the selection process difficult and time consuming. The following pages describe an approach for making the process as efficient and easy as possible. Intended to guide you through the selection process, they describe several steps you might take in order to select a system that meets your organization's measurement goals.
To ensure that all systems of interest to you are thoroughly and consistently evaluated, establish a structured system-selection procedure, such as a point-based scoring procedure. Such a procedure will help you distinguish among the evaluations you make of each measurement system. Your first step toward selecting a performance measurement system and performance measures within a system is to identify your organization's measurement goals.
Identify Your Organization's Measurement Goals
Knowing your organization's measurement goals will help you not only select a performance measurement system, but also measures within the system. Start identifying your organization's measurement goals by considering your mission and strategic plan for improving performance. Only you can determine what is important to measure about the patients/residents for whom your organization provides care. First, describe your patient/resident population and the services you provide them, then identify which services are high volume, high risk, high cost, known to be problematic, or perhaps are of significant interest to the community you serve. For example, you may care primarily for obstetrical patients (high volume services), suggesting that newly delivered mothers and newborns should be a focus of your strategic measurement and improvement plan. Maybe the incidence of decubitus ulcers among your nursing home's subacute population is unknown, but is believed to be problematic. Perhaps you suspect that your processes for providing infusion therapy in the home can be improved. In this case, you may want to use measures focused on outcomes of patients receiving such therapy.
After you target specific patient/resident populations and services and identify your organization's performance improvement priorities, consider any other information needs, such as the information other organizations are requesting from you. Also, are business coalitions and managed care plans (for example, integrated delivery systems, health plans, or preferred provider organizations) asking for particular information? Beginning in 1997, managed care plans accredited by the Joint Commission will also be expected to meet specific performance measurement requirements, some of which could affect provider organizations. You may want to check the plan(s) with whom you contract to determine how your organization will be affected by these requirements.
Once identified, factor any requests for information you receive into the design of your organization's strategic plan to measure and improve care. Without a deliberate assessment of measurement priorities, your organization may find itself in the position of either not measuring what is necessary or over-measuring; that is, expending effort in data collection without a clear purpose. Measuring a few key processes and outcomes well and using the collected data to improve care are far more worthwhile and effective than measuring many things and leaving few resources for analyzing and interpreting the data.
After you determine what to measure, you can turn your attention to finding the measurement system(s) and specific indicators to help you measure these care processes and outcomes efficiently and effectively.
Establish a Clear and Objective Evaluation Process
To select the best performance measurement system(s) for your organization, establish a clear and objective process to evaluate the systems profiled. It can be helpful to establish timelines and milestones to guide and track the progress of your selection process. Spell out what evaluation criteria will be considered, if and how they will be weighted, and who will participate in the process. Ultimately, this process should result in a recommendation of one or more systems that you can present to your organization's management. Include staff in the process who will be closely involved in using or maintaining the system selected. For example, this might include members of the medical staff, nursing staff, and staff from administration, performance improvement, health information management, and information technology. Having such representation not only provides the required expertise, but also increases the likelihood of each area's cooperation when implementing a new system.
Evaluate Systems Using Selection Criteria
Your selection criteria should include both must-have and desirable criteria. Must-have criteria should correspond to your critical needs of a measurement system that, if not met, should immediately remove a system from further consideration. Desirable criteria are those that are not as critical as your must-have criteria, but correspond to your preferences about a system. For example, perhaps your organization prefers to install a system with 24-hour online support, but would consider a system without it if other criteria are met.
When establishing your must-have criteria, include your organization's requirements for a system's measures and capabilities. Also, if your organization is part of a network (for example, integrated delivery system, health plan), find out what the network's requirements are for a system and factor them in with your own. Other must-have criteria should be derived from your strategic capabilities. For example, your organization may have already decided that the selected measurement system must accommodate electronic submission of data.
In addition, factor in criteria from the Joint Commission's "Framework for the Selection of Performance Measurement Systems." Although inclusion on the Joint Commission's list tells you that a system meets the initial screening criteria, it does not tell you how well it meets the criteria. Part of your review of systems should include an evaluation of how well a system meets these criteria. With the exception of the criteria listed under Attribute 6 (Relevance to Accreditation), each criterion could be appropriately included among your own must-have or desirable criteria. The Framework criteria were developed by national experts whose ideas about what is important in a performance measurement system represent some of the best advice you can get today. Moreover, you want the system you select to meet your accreditation and internal performance improvement needs, not to mention any reporting requests you receive from external groups.
The section entitled "Using the Joint Commission's Framework Criteria" provides details about the Framework's six attributes and corresponding criteria, including suggestions for specific approaches to reviewing a system's status on each criterion and recommended questions you might ask a system under review. Review these pages in detail before finalizing your selection criteria.
Other must-have or desirable criteria might be developed related to the following questions:
- Will your data be sold to others?
- Will you be mandated to upgrade your system or hardware?
- What is the cost of ongoing service contracts?
- Are price increases expected?
Use a selection grid or checklist to facilitate comparison of the different systems
Using a selection grid or checklist can help you make "at-a-glance" comparisons of the systems under consideration. A selection grid could include your must-have and desirable criteria, the weight assigned to each criterion, and blank rows or columns in which you can enter for each system the total number of points for a particular criterion and the system's overall score. An example of such a tool is shown in Figure 1. The tool in this figure includes a weighting system. A simpler checklist approach is shown in Figure 2. As illustrated in this figure, a checklist should also include your must-have and desirable criteria.
Review the list of acceptable systems and develop a "short list"
To quickly create a short list of systems on which to focus a more intensive review, apply the must-have criteria to each listed system you are considering. Then, check the measures of each system that meets your must-have criteria. Look for a written description of each measure's purpose or intent. Does the system include measures that match your needs? For example, if you have determined that you want to measure and improve the care provided to obstetrical patients, you will want to participate in a system that provides measures for this population.
Additional criteria relating to the system's measures should include the following:
- The system includes performance measures that are relevant to your organization's measurement goals.
- The system includes performance measures that can identify opportunities for improvement in the services you provide and the quality of the health care results you achieve.
- The system includes performance measures that monitor processes and outcomes your organization can affect.
- The system includes performance measures that your organization can reasonably implement.
- The performance measures and their data elements are precisely defined and specified to ensure uniform application.
- The performance measures can accurately identify the events they were designed to identify.
- The performance measures produce data that can be interpreted and transformed into meaningful and useful information.
- Performance measure results can be communicated to interested parties.
Also, does, or will, the system include a variety of measures that will likely meet your future measurement needs? For example, over time, it is probable that you will change your measurement emphases and need measures for other patient/resident populations. In addition, over time, the Joint Commission will expect you to select and send data for clinical measures that address up to 80% of your patient/resident population, as well as functional status and patient/resident satisfaction. Therefore, you want to be fairly certain that the system(s) you select will help you meet these future needs and requirements.
You may also want to identify systems whose data may be used to meet the requirements of other users of performance data (whether or not you've been asked for such data already). If you plan to use comparative information from your selected system to respond to such requests, be sure the system will allow you to do so. Some systems, for example, may have prohibitions against public disclosure of comparative information.
Also, consider whether other data users have suggested that your organization use a particular system. Perhaps a business coalition, insurer, government agency, or managed care plan has indicated a preference for a particular measurement system or for particular measures. Including such systems on your short list will ensure that you review them and will suggest that you talk with the requesting organization about the system it recommends, especially if you are leaning toward an alternative.
And certainly, you will want to assess resource investments necessary to purchase and maintain the system. Consider what the initial implementation fees are and what annual operating expenses include. Will you need to purchase software to support participation in the system? How many full-time staff will need to be devoted to this effort? Will staff training or retraining be necessary? The extent of your investments will depend on the number and type of measures you decide to use (and, thus, how much data collection is necessary), the extent of automation in your organization, and the system's participation or licensing fees. Depending on your allotted budget, you may eliminate some systems because of cost. Costs you are likely to incur when selecting, installing, and using a system include
- the staff resources invested to carefully select a system;
- the initial purchase of the performance measurement system (that is, the fees to the system);
- new hardware if indicated, which may include computers, printers, modems, and software for collecting and submitting data to the system;
- staff resources associated with installation, including possible customized programming needs;
- annual participation and/or licensing fees; and
- staff resources associated with collection, submission, and use of the data.
Also, if you are already participating in a performance measurement system included on the Joint Commission's list, add it to your short list. You will want to review it with the others to determine if it will continue to meet your measurement needs into the future. And finally, keep in mind that some measurement systems are "niche" players, that is they are specifically designed to measure processes or outcomes in specific areas (for example, cardiovascular care).
This initial effort will limit the number of in-depth reviews you will need to conduct.
Narrow your short list
Your short list should now be limited to those systems that appear to have measures that meet your current and future needs, have been suggested by other users of data, appear to be within your budgetary means, and meet your organization's other must-have criteria. At this time, using a point-based scoring system can be very effective for narrowing your short list. The following approach is offered as an example of the kind of scoring system you could use.
Begin by weighting your desirable criteria according to their importance. For example, you could use a three-point weighting system in which 1 is "Somewhat Important," and 2 is "Important," and 3 is "Very Important." If you prefer a measurement system with 24-hour support, you would assign a weight of 3 to a system's ability to provide this. Then, develop a scoring system. You might, for example, use a six-point system in which 5 is "Fully Meets," 4 is "Substantially Meets," 3 is "Partially Meets," 2 is "Minimally Meets," 1 is "Will Meet," and 0 is "Does Not Meet." The total number of points for each criterion is then calculated by multiplying the criterion's weight by the score you assign the system for that criterion.
For example, perhaps your organization prefers to install a system that uses the same type of information technology you use. If Measurement System X runs successfully on a system identical to yours, you would assign a score of 5 ("Fully Meets") to the applicable criterion. You would then multiply the score of 5 by 3 (your assigned weight for "Very Important"), giving Measurement System X a total of 15 points for that criterion.
After calculating the points for each criterion, total the points for all the criteria. Then, rank each system by its total score and narrow the list of possible systems to a final list for review against the Joint Commission's Framework criteria. The selection grid in Figure 2 illustrates this type of point-based scoring system. As a final check, compare those systems of greatest interest to you against the Joint Commission's Framework criteria. Although you may have included many or all of these criteria in your must-have and desirable criteria, making one, final check will ensure that the system you select meets your current and future accreditation needs.
Check customer references for each system on your final list
Ask other organizations that use the system(s) on your final list for a first-hand account of how the system works, the quality of customer support, their experience installing the system, and the system's value in meeting their measurement goals. If possible, view a demonstration of the system at an organization where it has already been implemented or obtain a demonstration copy of the software.
Select the system or systems that seem most appropriate for your organization
At this point, you will have completed detailed reviews of the few systems of greatest interest to your organization. It is unlikely that any one system today will fully meet all your desirable criteria or the Joint Commission's future criteria. Your review process should help you identify the system(s) that best meets your current needs and has the promise of continuing to meet your future needs.