The use of ORYX data and the ORYX Pre-Survey Report are incorporated into several components of the survey process, most notably in the interviews pertaining to performance measurement and improvement, leadership, strategic planning and resource allocation, and information management. A sample of topics surveyors may cover during these interviews include the following:
- How your selected ORYX measures fit into your organization's strategic approach to performance improvement;
- How the needs and concerns of your patient population(s) were considered in selecting your ORYX measures, as well as who participated in the selection process;
- How your leaders communicated your ORYX measure selections to all staff;
- How you have integrated ORYX data into your organizationwide performance improvement activities;
- What kind of comparative feedback reports you receive from your selected measurement system and whether the reports match those sent to the Joint Commission;
- The roles of clinical and support staff, leaders, and the governing board (if applicable) in monitoring ORYX measurement results and in taking action when results are unsatisfactory; and
- What methods you used to educate and train staff involved in generating, collecting, validating, and interpreting data for ORYX measures.
Surveyors will not score the data you have collected. Rather, they will evaluate how your organization has used the data in the context of your overall performance improvement program.
-Top-
What is the ORYX Pre-Survey Report?
This report
- lists the performance measures your organization has selected;
- summarizes the performance measure data your organization has submitted to meet Joint Commission ORYX performance measurement requirements;
- presents the data (for each selected measure) submitted by your performance measurement system to the Joint Commission in the form of comparison and control charts; and
- identifies any months for which your organization had missing data and/or no cases to report (for each measure selected), performance outliers, and unusual data patterns.
The comparison and control charts in the report are created using the data your organization submitted to your performance measurement system, which in turn transmitted aggregate monthly data to the Joint Commission. You may notice differences between what's shown in your ORYX Pre-Survey Report's comparison and control charts and what appears in the feedback reports from your performance measurement system. Discrepancies may occur when the feedback reports reflect data that have been updated since your performance measurement system transmitted its quarterly data to the Joint Commission, or when your measurement system uses a different method of analysis.
When will you receive your ORYX Pre-Survey Report?
If your organization is required to participate in a listed performance measurement system and submit data to the Joint Commission, the ORYX Pre-Survey Report is mailed to you approximately 21 days before your regularly scheduled survey. If your organization is accredited under the laboratory or network accreditation programs, you are not currently required to submit performance measurement data and will not receive an ORYX Pre-Survey Report. ORYX performance measurement requirements have not yet been established for other accreditation programs.
What's in your ORYX Pre-Survey Report?
The Summary of Measures lists the performance measures your organization has selected to meet the ORYX requirements. The measures are identified by
- an accreditation program code (e.g., HAP = hospital, LTC = long term care)
- a measure identification number,
- an abbreviated measure name,
- the month/year your organization began collecting data for each measure,
- the last month/year your organization submitted data for each measure you no longer use, and
- the number of months each measure has been used.
The summary also indicates whether there were any
- out of statistical control data points (shown graphically in the control chart),
- outlier data points (shown graphically in the comparison chart),
missing data points, or
- potential data quality concerns (aberrant data) for each of the performance measures.
The Joint Commission analyzes each of your selected performance measures in monthly increments. Control charts are provided for each measure (after it has been used for at least 12 months); comparison charts are provided regardless of how long the measure has been used. Both control and comparison charts note the name of the measure, provide a brief description of its focus, and show the "direction" in which data for the measure would demonstrate improvement. Each chart also has an overview of what the data points show:
Comparison chart interpretation: Lists months with missing data and the corresponding reasons, desirable and undesirable outlier points and when they occurred.
Control chart interpretation: Identifies data points that are out of statistical control (i.e., special cause variation) based on the three tests the Joint Commission uses to identify variation in a process.
-Top-
The Joint Commission uses a combination of control charts and comparison charts to evaluate ORYX data. Control chart analysis is based on your organization's own historic data and is used to assess internal process stability. Comparison analysis is based on multiple organizations' performance data and is used to evaluate your organization's relative performance level. The use of comparison analysis in addition to control chart (intraorganizational) analysis is a key feature of the Joint Commission's analytic methods in the ORYX initiative. The two types of analyses evaluate organization performance from two distinct perspectives and thus can provide a more comprehensive framework to assess your overall performance level.
Because of their different focuses, the control and comparison analyses may portray different interpretations of performance. For example, a control chart may show a desirable pattern (e.g., one that is in control), but the comparison chart may illustrate undesirable outliers (e.g., a high rate of infections). Perhaps the organization's performance has been consistently poorer than that of other organizations using the same measure. In this case, the organization needs to think about changing its process for the measure concerned in order to improve its performance. On the other hand, an organization without outliers in the comparison analysis may have a special cause variation (out of control pattern) detected in the control chart. In this case, the organization needs to investigate the special cause variation in its process before making any conclusions about performance level. In general, control chart analysis is done before comparison analysis to ensure a given process is stable before trying to evaluate relative performance level.
-Top-
aberrant data
Gross inconsistencies in the number of cases submitted from quarter to quarter.
attributes data
Also referred to as "count data." The number of individuals who experienced the attribute of interest are counted from all those who had the potential to experience the event of interest. Because the results have limited possibilities or categories, the range is limited to the number or percentage in each category.
common cause variation
Variation in a process that is due to the process itself and is produced by interactions of variables in that process. Common cause variation is inherent in all processes; it can be removed only by making fundamental changes to a process.
in statistical control
A stable process in which variation exists due to common causes.
measurement systems
Are required to provide feedback reports that include both control charts and comparison analysis to their client organizations.
out of statistical control
An unstable process in which variation exists due to a special cause.
outlier
A data point that is statistically significantly different from others within a given data set.
performance measurement system
A vendor that provides an automated database to facilitate performance improvement in health care organizations by collecting and disseminating data pertaining to process/outcome measures of performance. Beginning with first-quarter 2001 data, measurement systems are required to provide feedback reports that include both control charts and comparison analysis to their client organizations.
special cause variation
The variation in performance and data that results from variables that are not a part of the original process or system. Special cause variation is intermittent, unpredictable, and unstable.
standard deviation
A measure of variability that indicates the spread of a set of observations around the mean (average).
trend
Six consecutive data points that show a steady increase or decrease. Some analysts may prematurely see a trend with fewer than six points, which often results in erroneously identifying common cause patterns as special causes.
-Top-
ORYX and the Survey Process
More detailed facts about survey activities and what surveyors will look for is available in Guides to the Survey Process for specific types of health care organizations (hospitals, long term care, home care, ambulatory care, and behavioral health care). To obtain information on these guides, please call the Joint Commission's Customer Service department at 630-792-5800.
Data Analysis
Joint Commission on Accreditation of Healthcare Organizations. Managing Performance Measurement Data in Health Care. Oakbrook Terrace, IL: Joint Commission, 2000.
Statistical Process Control
- Balestracci D Jr, Barlow, JL: Quality Improvement: Practical Applications for Medical Group Practice, 2nd ed. Englewood, CO: Center for Research in Ambulatory Health Care Administration, 1996.
- Boggs PB, Hayati F, Washburne WF, Wheeler DA: Using statistical process control charts for the continual improvement of asthma care. The Joint Commission Journal on Quality Improvement 1999;25(4):163-181.
- Carey RG, Lloyd RC: Measuring Quality Improvement in Healthcare: A Guide to Statistical Process Control Applications. New York: Quality Resources, 1995.
- Carey RG, Teeters JL: CQI case study: Reducing medication errors. The Joint Commission Journal on Quality Improvement. 1995;21(5):232-237.
- Carlin E, Carlson R, Nordin J: Using continuous quality improvement tools to improve pediatric immunization rates. The Joint Commission Journal on Quality Improvement. 1996;22(4):277-288.
- Finison LJ, Finison KS, Bliersbach CM: The use of control charts to improve healthcare quality. Journal for Healthcare Quality 1993;15(1):9-23.
- Gitlow H, Gitlow S, Oppenheim A, Oppenheim R: Tools and Methods for the Improvement of Quality, Homewood, IL: Irwin, 1989.
- Grant E, Leavenworth R: Statistical Quality Control, 7th ed. New York: John Wiley & Sons, 1996.
- Kirk R: Managing Outcomes, Process, and Cost in a Managed Care Environment. Gaithersburg, MD: Aspen Publishers, Inc., 1997.
- Levett J, Carey RG: Measuring for improvement: From Toyota to thoracic surgery. Annals of Thoracic Surgery 1999;68:353-358.
- Mitchell L, Fife S, Chothia AA, et al.: Three teams improving thrombolytic therapy. The Joint Commission Journal on Quality Improvement 1996;22(6):379-390.
- Montgomery D: Introduction to Statistical Quality Control, 3rd ed. New York: John Wiley & Sons, 1996.
- Nugent W, et al.: Designing an instrument panel to monitor and improve coronary artery bypass grafting. Journal of Clinical Outcomes Management 1994;1(2):57-64.
- Ornstein SM, Jenkins RG, Wickham Lee F, et al.: The computer?based patient record as a CQI tool in a family medicine center. The Joint Commission Journal on Quality Improvement 1997;23(7):347-361.
- Page US, Washburn T: Using tracking data to find complications that physicians miss: The case of renal failure in cardiac surgery. The Joint Commission Journal on Quality Improvement 1997;23(10):511-520.
- Pitt H: SPC For the Rest of Us: A Personal Path to Statistical Process Control. Reading, MA: Addison-Wesley, 1994.
- Pyzdek T: Pyzdek's Guide to SPC, Volume 1: Fundamentals. Tucson, AZ: Quality Publishing, LLC, 1998.
- Pyzdek T: Preventing hospital falls. Quality Digest 1999;19(5):26-27.
- Pyzdek T: Variation and your health. Quality Digest 1998;18(8):22.
- Shahian DM, Williamson WA, Svensson L, et al.: Applications of statistical quality control to cardiac surgery. Annals of Thoracic Surgery 1996;62(5):1353-1359.
- Solberg LI, Mosser G, McDonald S: The three faces of performance measurement: Improvement, accountability, and research. The Joint Commission Journal on Quality Improvement 1997;23(3):135-147.
- Wheeler DJ, Chambers DS: Understanding Statistical Process Control. Knoxville, TN: SPC Press, Inc., 1992.
Use of Comparison Charts
Agresti A, Coull BA: Approximate is better than "exact" for interval estimation of binomial proportions. The American Statistician 1998;52(2):119-126.
Holubkov R, Holt VL, Connell FA, LoGerfo JP: Analysis, assessment, and presentation of risk-adjusted statewide obstetrical care data: The StORQS II study in Washington state. Part I. Health Services Research 1998;33(3):531-548.
Shwartz M, Ash AS, Iezzoni LI: Comparing outcomes across providers. Risk Adjustment for Measuring Outcomes. Chicago: Health Administration Press, 1997; pp. 472-516.
Wassertheil-Smoller S: Biostatistics and Epidemiology: A Primer for Health Professionals. New York: Springer-Verlag, 1995.
-Top-