Reviewing systems against the Joint Commission's Framework criteria will give you a sense of each system's status against criteria derived by experts in outcomes measurement whose ideas about what is important in a performance measurement system represent some of the best advice available today. You also want the system you select to meet your accreditation and internal performance improvement needs, not to mention any reporting requests you receive from external groups.
The fact that a system is on the Joint Commission's list tells you it meets the initial screening criteria; it does not provide quantitative information on how well it meets the criteria. For example, some systems do a much better job of ensuring the accuracy of performance measurement data. If a system does not meet a future criterion today, ask the system how it intends to meet it in the future. The following pages explain each criterion, provide suggestions for specific approaches to reviewing a system's status on each criterion, and recommend questions you might ask a system under review.
Attribute 1: Performance Measures. Criterion A focuses on the types of measures included in the performance measurement system and their relevance to Joint Commission accreditation programs. Clinical measures are required for selection in 1997, data collection in 1998, and data submission in 1999. Although in later years, other types of measures, such as patient/resident satisfaction and health status, will also be required by the Joint Commission, you may have other reasons for using these types of measures today. Consider asking the following criteria-based questions:
- Does the system include clinical measures? (See the glossary for a definition of "clinical measures.")
- Does the system include satisfaction, health status, and administrative/financial measures?
- Does the system identify the health care delivery setting to which each measure relates?
Knowing the applicability of the system's measures to Joint Commission accreditation programs will assist you with the Joint Commission's requirement to select measures relevant to your patient/resident population.
Criterion B discusses the extent of documentation for and definition of each measure in the system and the relationship of measures to the Joint Commission's accreditation programs. The documentation of definitions increases the likelihood that participants are in fact measuring the same thing. Without that assurance, comparative information could be suspect. The information derived from your participation will be used by the Joint Commission to evaluate your performance, by your organization to identify improvement opportunities, and perhaps by others to purchase or select care. In other words, important decisions about your organization will be made based on the exact data you collect. This criterion also stipulates that the specific directions (algorithms) for determining the occurrence of the event described by the measure are documented.
Consider asking the following criteria-based questions:
- Does the system provide definitions of the measures, including the populations of interest (denominator) and the events being measured (typically referred to as the numerator or measurement)? Ask to see examples.
- Is the intent or rationale for each measure described?
Which measures has the system identified as available to meet Joint Commission requirements?
- Do you have the option to use only the measures of interest to your organization, or must you use the entire set provided by the system?
- Are the data elements that comprise the measures listed and defined, including allowable values for each?
What are the sources of the required data? Systems that use only claims data often do not capture important clinical data for calculating or risk-adjusting measures of clinical care.
- Are directions for sampling provided when applicable?
You may also want to ask if the performance measures have been field tested to ensure that data can be captured completely, accurately, and consistently by provider organizations.
Criterion B increases the likelihood of providing valid, comparative information to participants because the system has better data-auditing and risk-adjustment capabilities (patient-/resident-level data), and the review of data for each patient/resident record is accomplished through the use of a defined, standardized algorithm.
Criterion C, a future criterion, expects that the system has a defined process for keeping the measure set current.
Attribute 2: Database (Measurement System Technical Capabilities). Criterion D is a current requirement and lists several points related to the database itself. Each listed system has indicated that it has an operational, automated, centralized database. Because there are different types of databases, you may still want to ask whether it is centralized, stores your data separately from those of other participants, and supports intraorganizational and interorganizational comparisons. In addition, ask the system whether it is capable and willing to send your data to the Joint Commission on your behalf. Other questions you may want to consider asking include the following:
- How is the integrity of the database maintained?
- How are the confidentiality of, and access to, the database protected?
- Can you have access to the database? If yes, how? If no, why not?
- How are data submitted and stored in the database (paper, diskette, modem)?
- If your organization must collect and submit data manually, will the system accommodate data submitted on paper?
- Are there certain timelines, restrictions, and rules associated with data submission?
- What is the technical structure and capacity of the database?
- What are the qualifications of the technical staff?
- Is there a support structure (for example, help desk) for your use?
- Can the system send monthly data points electronically to the Joint Commission at least every quarter?
These details can provide insight about the longevity of the database and flexibility to support your continued data submission and participation. The number of enrollees will also affect the database. Therefore, it is important to determine if the system has adequate volume to provide meaningful comparative data. Additional questions you may want to consider asking include the following:
- How many organizations currently participate in the system?
- How large is the database now? How many patient/resident cases are actually in the database?
- What expectations does the system have for new enrollment? On what is this based?
- What has the growth pattern of the database been?
- How will the database continue to meet participants' needs if there is significant new enrollment?
Attribute 3: Data Quality (Performance Measure Accuracy). The two criteria under this attribute address the accuracy and completeness of performance measurement data. Criterion E addresses the system's auditing practices and support provided to participants to ensure data accuracy and the measures' validity. Currently, a system should have an audit process in place, the results of which are shared with users. In the future, the system will also need to demonstrate that
- the results of data checks are used to improve the quality of the data;
- calculations of measures are accurate; and
- measures are identifying all events that should be captured by the measure.
Consider asking these questions:
- Does the system regularly check the quality of submitted data?
- Ask the system representative to describe the process to you.
- Does the participating organization receive feedback about its own particular data quality problems?
- Are general data quality problems shared with all participants?
- Are the results of the data checks shared with participants?
- Can you correct and resubmit your data?
- Does the system use audit results to make changes to the measures and data elements?
- Are there clear directions (algorithms) for calculating the measures?
- Were the measures field tested for the above characteristics?
- Does the system provide a mechanism for assuring that a sufficient number of participants have selected a measure to generate statistically meaningful reports?
- Are current participants actually identifying improvement opportunities?
- To what organizational or clinical functions do the measures relate?
Criterion F focuses on the system's provision of education and ongoing support as related to data quality issues. Systems should be currently providing this type of education and support. In the future, systems will be expected to monitor the quality of data at the individual participant level. The following questions might be asked of system representatives:
- Is education about data quality issues provided to participants? Ask for a description of this service.
- Are participants expected to submit to the system the results of their own internal data audits? If not now, how will the system meet this future criterion?
- Are users' groups held?
- Is support documentation provided to users?
- Can you get assistance from the system to improve the quality of your data?
Attribute 4: Risk Adjustment/Stratification. It cannot be stressed strongly enough that today's health care environment is characterized by the movement toward accountability and measurement. Your organization will likely be asked, if it has not been already, to provide data for use by various publics to make judgements and decisions about your organization. Even your own use of comparative data will drive resource investments into particular care processes of concern.
Therefore, it is important to know that your performance is compared fairly to other participants who are using the same measure from the same system. The comparisons must be fair, believable, and actionable when necessary. Fair comparisons are facilitated by risk adjusting the measure to consider the severity and complexity of the patient's/resident's condition, or, when appropriate, by stratifying the rates according to particular patient/resident characteristics, such as age or gender, that are known to be related to the outcome being measured.
Criterion G, an initial screening criterion, expects systems to explain the process for risk adjusting or stratifying measures, as well as provide a rationale for those measures that are not risk adjusted or stratified.
To assess this criterion, consider asking these questions:
- Does the system include any measures describing events that may be confounded by patient/resident factors, such as demographics, comorbidities, severity of illness, or functional status?
- If yes, ask how the patient/resident factors are identified to "adjust" participants' rates.
- What statistical techniques are used to adjust data?
- If the system does not risk adjust or stratify rates, ask for an explanation as to why it does not and what its plans are for future use of such techniques.
Attribute 5: Performance Measure-Related Feedback. Access to individualized, timely, comparative reports is the primary reason you are participating in a performance measurement system. You can collect meaningful data of high quality, but it must be turned into comparative information and provided to you in a timely manner to be useful. Use of this additional information will help you achieve the ultimate goal of performance measurement--performance improvement. This attribute describes expectations related to the feedback provided by the system to participants.
Criterion I, a current criterion, expects that participants who are actively submitting data are receiving comparative feedback. A system must be able to provide valid comparisons across its participants. Ask if the system can currently meet this criterion.
Criterion J, also a current criterion, states that this feedback should be updated at least annually (current criterion) and, in the future, updated according to the Joint Commission's transmission schedule (quarterly). Frequent provision of reports will help you find connections between data and current care processes.
For example, we would all agree that a useful outcome measure for oncology treatment is long term survival. Today's five-year survival rate, however, is in part the result of the processes of care provided five years ago--a situation you cannot change today unless the identical processes are still in place. Similarly, feedback provided to you today that is more than one year old may not be meaningful in identifying processes needing to be improved.
To assess Criterion J, consider asking the following questions:
- How often are reports provided?
- How timely are they?
- How are risk-adjusted rates displayed?
- What is the report turnaround time from data submission to report receipt?
- How are reports provided to you--hard copy through the mail or electronically?
- What type of support does the system provide to help you interpret the reports?
- Does the system publicly disclose your data?
- Does the system allow you to share the data (reports) with other users if you so choose?
- Will the system assist you with disclosing data to others?
- Are ad hoc reports available? If so, at what cost?
- Will the system be able to send quarterly data with monthly data points to the Joint Commission every quarter beginning in January 1999 and no later than March 31, 1999? (Trial submissions will be accepted by July 1998.)
- Does the system and database support benchmarking?
- Do high-level performers disclose their rates and participate in a true benchmarking experience with other participants?
Criterion K, also a current expectation, addresses the system's capability to provide education and interpretation support to participants. Ask the system representative how this support is provided.
Attribute 6: Relevance to Accreditation. Finally, for the system to be integrated into the Joint Commission's accreditation process, it must meet two additional criteria. Both are marked for future compliance.
Criterion L addresses the system's ability to send system-level data that allows the Joint Commission to view an individual health care organization's rates against other health care organizations using the same measure. The Joint Commission has provided measurement systems with specific information about required system-level data elements. This is a future criterion since data submission is not required until 1999. You want to select a system, however, that assures you it will be able to meet this expectation.
Criterion M stipulates that the system work with the Joint Commission to design and conduct evaluative studies. These studies will be an ongoing part of the evolution of the new accreditation process. This is also a good place to ask about the system's current agreement with the Joint Commission. Each listed system has signed a contract with the Joint Commission indicating its commitment to remain in compliance with current criteria and its intent to meet future criteria. You are strongly encouraged to discuss these issues with the system representative as well.