Real World Testing Plan 2024

General Information

• Plan Report ID Number: (To be provided by the ACB)
• Developer Name: Roji Health Intelligence LLC
• Product Name(s): Roji Clinical Data Registry
• Version Number(s): 2016
• Certified Health IT Product List (CHPL) ID(s): 15.04.04.2951.Roji.16.00.0.170202
• Developer Real World Testing Page URL: https://rojihealthintel.com/real-world-testing/

Justification for Real World Testing Approach

The Roji Clinical Data Registry supports the import of patient information through various means depending on the output capabilities of the client organization (a hospital, practice group, etc.). A client organization needs to report on applicable clinical quality measures (CQMs). In cases with lower interoperability conformance, a data spec is developed collaboratively between Roji and the client organization so as to acquire the necessary data for the patients, providers, time frame, and CQMs involved. In cases with higher interoperability conformance, a standardized QRDA I document for patient-level data can be imported by Roji. This testing plan will focus on the importing of QRDA I documents for the § 170.315(c)(2)(i) import certification criteria.

The Roji Clinical Data Registry reports on clinical quality measures (CQMs) through various means depending on the input capabilities of the CQM consumer. For example, Roji reports on CQMs to CMS through the CMS Quality Payment Program (QPP) API. In other cases, a client organization wants to input a standardized QRDA III document for summarized CQM data. This testing plan will focus on the exporting of QRDA III documents for the § 170.315(c)(2)(ii) calculate AND Testing § 170.315(c)(3) report certification criteria.

Standards Updates

A survey of the following standards for revisions since August 31, 2021, that were applicable to the Certification Criteria applicable to this Real World Testing Plan.

• Standards Version Advancement Process (SVAP)
  • None. Roji will NOT participate in the voluntary SVAP.
    • https://ecqi.healthit.gov/sites/default/files/QRDA-HQR-2021-CMS-IG-508.pdf 
    • https://ecqi.healthit.gov/sites/default/files/2021-CMS-QRDA-III-Eligible-Clinicians-and-EP-IG-v1.3.pdf 
• United States Core Data for Interoperability (USCDI)
  • None
• C-CDA Companion Guide Updates
  • None
• Electronic prescribing
  • None
• Security tags
  • None
• ASTM updates
  • None
• Clinical Quality Measures – Report
  • ONC removed the Health Level 7 (HL7R) Quality Reporting Document Architecture (QRDA) standard requirements in the 2015 Edition “Clinical Quality Measures—report” criterion in § 170.315(c)(3) and, in their place, requires Health IT Modules to support the CMS QRDA Implementation Guide(IGs). Any Health IT Developer with Health IT Modules certified to §170.315(c)(3) must update their certified health IT to be compliant with the revised versions of this criterion and provide its customers of the previously certified health IT with the updated certified health IT by December 31, 2022.
  • For more details, see below.

Update 1

• Standard (and version): CMS Implementation Guide for Quality Reporting Document Architecture: Category I; Hospital Quality Reporting; Implementation Guide for 2020 (https://ecqi.healthit.gov/sites/default/files/QRDA-HQR-2020-CMS-IG-v1.1-508.pdf) 
• Updated certification criteria and associated product:
  • Certification Criteria: 45 CFR § 170.315(c)(2) Clinical quality measures (CQMs) – import 
  • Product: Roji Clinical Data Registry
• Health IT Module CHPL ID: 15.04.04.2951.Roji.16.00.0.170202
• Method used for standard update: Update code in Roji QRDA Parser and CQM Engine.
• Date of ONC ACB notification:  (To be notified when implemented in 2022)
• Conformance measure: RWT Measure 2

Update 2

• Standard (and version): CMS Implementation Guide for Quality Reporting Document Architecture: Category III; Eligible Clinicians and Eligible Professionals Programs; Implementation Guide for 2020 (https://ecqi.healthit.gov/sites/default/files/2020-CMS-QRDA-III-Eligible-Clinicians-and-EP-IG-v1.2.1-508.pdf) 
• Updated certification criteria and associated product:
  • Certification Criteria: 45 CFR § 170.315(c)(3) Clinical quality measures (CQMs) – report
  • Product: Roji Clinical Data Registry
• Health IT Module CHPL ID: 15.04.04.2951.Roji.16.00.0.170202
• Method used for standard update: Update code in Roji QRDA Generator and CQM Engine.
• Date of ONC ACB notification: (To be notified when implemented in 2022)
• Conformance measure: RWT Measure 2

Measures used in overall approach

RWT Measure 1

• Description of the measurement/metric: Test a random subset of the patients per client every month. 
  • The numerator will be the number of distinct patients that valid QRDA I documents successfully imported.
  • The denominator will be the number of distinct patients this was attempted for.
• Associated certification criteria: 
  • 45 CFR § 170.315(c)(2) Clinical quality measures (CQMs) – import and calculate
• Justification for selected measurement/metric: Some clients can give us patient-level QRDA I documents that we normalize into our database. Other clients give us raw data that we normalize into our database. A successful import will allow us to perform calculations.
• Care setting(s) that is addressed: A random subset should cover a variety of clinical variables.
• Expected outcomes: We expect to be able to generate a QRDA I document for 100% of the cases.

RWT Measure 2

• Description of the measurement/metric: Test a subset of the practice & provider combinations per client every month. 
  • The numerator will be the number of distinct client-practice-provider combos that a valid QRDA III document was generated.
  • The denominator will be the number of distinct client-practice-provider combos this was attempted for.
  • Report the particular CQMs applicable to the subset.
• Associated certification criteria: 
  • 45 CFR § 170.315(c)(2) Clinical quality measures (CQMs) – import and calculate
  • 45 CFR § 170.315(c)(3) Clinical quality measures (CQMs) – report
• Justification for selected measurement/metric: We should be able to generate valid QRDA III documents regardless of the CQMs.
• Care setting(s) that is addressed: Clients, practices, and providers only report on applicable CQMs.
• Expected outcomes: We expect to be able to generate a QRDA III document for 100% of the cases.

Schedule of Key Milestones

• Started collecting data for Roji RWT Measures. Timeframe: 2021-02 estimated.
• Verified all CQMs that we report on. Timeframe: 2021-05 estimated.

Attestation

This Real World Testing plan is complete with all required elements, including measures that address all certification criteria and care settings. All information in this plan is up to date and fully addresses the health IT developer’s Real World Testing requirements. 

• Authorized Representative Name: Don Newman
  
• Authorized Representative Email: [email protected]
• Authorized Representative Phone: (312) 258-8004 x712
• Authorized Representative Signature:
• Date: October 26, 2023