Roji Health Intelligence

Navigate The Path To Better Health Care.

Real World Testing 2025

Summary

This is the Real World Testing Plan by Roji Health Intelligence LLC that will be submitted in 2024 and applied in 2025. The plan is designed to meet certification requirements for Clinical Quality Measures (CQMs) and Electronic Health Information (EHI) export in compliance with 21st Century Cures Act: Interoperability, Information Blocking, and the ONC Health IT Certification Program final rule, 85 FR 25642 (https://www.govinfo.gov/content/pkg/FR-2020-05-01/pdf/2020-07419.pdf) (May 1, 2020) (ONC Cures Act Final Rule), Section VII.B.5 “Real World Testing”.

This Real World Testing Plan is for the following Certification Criteria 

  • 45 CFR § 170.315(c)(2) Clinical quality measures (CQMs) – import and calculate
  • 45 CFR § 170.315(c)(3) Clinical quality measures (CQMs) – report
  • 45 CFR § 170.315(b)(10) Electronic Health Information (EHI) Export

Audience

  • Drummond Group LLC as an ONC Authorized Certification Body (ACB) and ONC Authorized Testing Lab (ACL).
  • The Office of the National Coordinator for Health Information Technology (ONC) and its ONC Health IT Certification Program for the purposes of Real World Testing and posting publicly on the ONC Certified Health IT Product List (CHPL). 

General Information

  • Plan Report ID Number: (To be provided by the ACB)
  • Developer Name: Roji Health Intelligence LLC
  • Product Name(s): Roji Clinical Data Registry
  • Version Number(s): 2024
  • Certified Health IT Product List (CHPL) ID(s): 15.04.04.2951.Roji.16.00.0.170202
  • Developer Real World Testing Page URL: https://rojihealthintel.com/real-world-testing-2025/ 

Justification for Real World Testing Approach

The Roji Clinical Data Registry supports the import of patient information through various means depending on the output capabilities of the client organization (a hospital, practice group, etc.). A client organization needs to report on applicable clinical quality measures (CQMs). In cases with lower interoperability conformance, a data spec is developed collaboratively between Roji and the client organization so as to acquire the necessary data for the patients, providers, time frame, and CQMs involved. In cases with higher interoperability conformance, a standardized QRDA I document for patient-level data can be imported by Roji. This testing plan will focus on the importing of QRDA I documents for the § 170.315(c)(2)(i) import certification criteria.

The Roji Clinical Data Registry reports on clinical quality measures (CQMs) through various means depending on the input capabilities of the CQM consumer. For example, Roji reports on CQMs to CMS through the CMS Quality Payment Program (QPP) API. In other cases, a client organization wants to input a standardized QRDA III document for summarized CQM data. This testing plan will focus on the exporting of QRDA III documents for the § 170.315(c)(2)(ii) calculate AND Testing § 170.315(c)(3) report certification criteria.

Standards Updates

A survey of the following standards for revisions since August 31, 2024, that were applicable to the Certification Criteria applicable to this Real World Testing Plan.

  • Standards Version Advancement Process (SVAP)
    • None. Roji will NOT participate in the voluntary SVAP.
  • United States Core Data for Interoperability (USCDI)
    • None
  • C-CDA Companion Guide Updates
    • None
  • Electronic prescribing
    • None
  • Security tags
    • None
  • ASTM updates
    • None
  • Clinical Quality Measures – Report
    • ONC removed the Health Level 7 (HL7R) Quality Reporting Document Architecture (QRDA) standard requirements in the 2015 Edition “Clinical Quality Measures—report” criterion in § 170.315(c)(3) and, in their place, requires Health IT Modules to support the CMS QRDA Implementation Guide(IGs). Any Health IT Developer with Health IT Modules certified to §170.315(c)(3) must update their certified health IT to be compliant with the revised versions of this criterion and provide its customers of the previously certified health IT with the updated certified health IT by December 31, 2022.
    • For more details, see below.

Update 1

  • Updated certification criteria and associated product:
    • Certification Criteria: 45 CFR § 170.315(c)(2) Clinical quality measures (CQMs) – import 
    • Product: Roji Clinical Data Registry
  • Health IT Module CHPL ID: 15.04.04.2951.Roji.16.00.0.170202
  • Method used for standard update: Update code in Roji QRDA Parser and CQM Engine.
  • Date of ONC ACB notification:  (To be notified when implemented in 2022)
  • Conformance measure: RWT Measure 2

Update 2

  • Updated certification criteria and associated product:
    • Certification Criteria: 45 CFR § 170.315(c)(3) Clinical quality measures (CQMs) – report
    • Product: Roji Clinical Data Registry
  • Health IT Module CHPL ID: 15.04.04.2951.Roji.16.00.0.170202
  • Method used for standard update: Update code in Roji QRDA Generator and CQM Engine.
  • Date of ONC ACB notification: (To be notified when implemented in 2022)
  • Conformance measure: RWT Measure 2

Update 3

  • Standard (and version): All standards versions are those specified in USCDI v1
  • Updated certification criteria and associated product:
    • Certification Criteria: 45 CFR §170.315(b)(10) Electronic Health Information export
    • Product: Roji Clinical Data Registry
  • Health IT Module CHPL ID: 15.04.04.2951.Roji.16.00.0.170202
  • Method used for standard update: Update code in Roji Patient Export Generator
  • Date of ONC ACB notification:  April 2022
  • USCDI-updated criteria: The plan documents the support of all USCDI v1 data elements
  • Conformance measure: RWT Measure 3

Measures used in overall approach

RWT Measure 1

  • Description of the measurement/metric: Test a random subset of the patients per client that submits QRDA I documents.
    • The numerator will be the number of distinct patients that valid QRDA I documents successfully imported.
    • The denominator will be the number of distinct patients this was attempted for.
  • Associated certification criteria:
    • 45 CFR § 170.315(c)(2) Clinical quality measures (CQMs) – import and calculate
  • Justification for selected measurement/metric: Some clients can give us patient-level QRDA I documents that we normalize into our database. Other clients give us raw data that we normalize into our database. A successful import will allow us to perform calculations.
  • Care setting(s) that is addressed: A random subset should cover a variety of clinical variables.
  • Expected outcomes: We expect to be able to generate a QRDA I document for 100% of the cases.

RWT Measure 2

  • Description of the measurement/metric: Test a subset of the practice & provider combinations per client that submits eCQMs.
    • The numerator will be the number of distinct client-practice-provider combos that a valid QRDA III document was generated.
    • The denominator will be the number of distinct client-practice-provider combos this was attempted for.
    • Report the particular CQMs applicable to the subset.
  • Associated certification criteria:
    • 45 CFR § 170.315(c)(2) Clinical quality measures (CQMs) – import and calculate
    • 45 CFR § 170.315(c)(3) Clinical quality measures (CQMs) – report
  • Justification for selected measurement/metric: We should be able to generate valid QRDA III documents regardless of the CQMs.
  • Care setting(s) that is addressed: Clients, practices, and providers only report on applicable CQMs.
  • Expected outcomes: We expect to be able to generate a QRDA III document for 100% of the cases.

RWT Measure 3

  • Description of measurement/metric: Test the measure that will assess functionality used to export EHI for a single patient and patient population.
    • Create an export file: Single patient data is exported to a JSON file. Patient population data is exported to a CSV, Excel, or table file. 
    • Execute at any time: Feature is available at all times
    • Limit ability of users who can create export: Only authorized users are allowed to export patient data
    • Electronic and computable format: Export file is in JSON format. The format for data can be viewed at: https://client.rojihealthintel.com/helpfiles/ehi-export/PatientExport.yml
  • Associated certification criteria:
    • § 170.315(b)(10) EHI export – Single patient EHI export
      • Justification: Exporting EHI associated with patients is a way to share information with external organizations. Export is typically used when users require a full patient record. This metric provides information on the type of data exported and the frequency of exports.
      • Test Methodology: System logs will be reviewed to determine the frequency of use. Log files are de-identified and used to determine proper functionality. This method is used to ensure compliance with this criteria
      • Expected Outcome: It is expected that authorized users will be able to share patient EHI using the export function. Successful and failed exports will be logged and tracked over time.
    • § 170.315(b)(10) EHI export – Patient population EHI export
      • Justification: Exporting health information associated with patient populations is another way to share health info to an external organization. Export of patient population is restricted to administrative and credential users. It is assumed that the data is used for research or quality purposes. This will provide a metric on the use of the export of EHI for patient population associated with Roji
      • Test Methodology: System logs will be reviewed to determine the frequency of use to ensure the export function is operating properly. Log files obtained are de-identified and used for analysis to validate proper operation. This method is used to ensure compliance with this criteria
      • Expected outcome: It is expected that authorized users will be able to share patient population data using the export function. Successful and failed exports will be logged and tracked over time.

Schedule of Key Milestones

  • Start collecting data for Roji RWT Measures. Timeframe: 2025-02 estimated.
  • Verify all CQMs that Roji reports on. Timeframe: 2025-05 estimated.

Attestation

This Real World Testing plan is complete with all required elements, including measures that address all certification criteria and care settings. All information in this plan is up to date and fully addresses the health IT developer’s Real World Testing requirements. 

Date: October 17, 2024

Authorized Representative Name: Don Newman

Authorized Representative Email: [email protected]

Authorized Representative Phone: (312) 258-8004 x712

Authorized Representative Signature: