Real World Testing Results 2023

Covered by this topic

General Information

Plan Report ID Number
Developer Name	Medical Informatics Engineering
Product Name(s)	WebChart EHR
Version Number(s)	8.4
Certified Health IT Product List ID(s)	15.04.04.1932.WebC.84.01.0.221117
Developer Real World Testing Page URL	https://docs.webchartnow.com/resources/system-specifications/ehr-certification/real-world-testing/
Results Submission Date	01/29/2024, resubmitted 02/29/2024

Changes to the Original Plan

Summary of Change	Reason	Impact
QRDA SVAP was not completed during 2023	Business needs prevented development from finalizing code to meet the new standards.	There was no impact on RWT activities as testing measures did not need to be updated to the newer standard.
Certification of additional CQMs was not completed during 2023	Business needs prioritized development and certification of b.10 and f.5	There was no impact on RWT activities.

Withdrawn Products

Product Names(s)	WebChart EHR
Version Number(s)	7.4
CHPL Product Number(s)	15.04.04.1932.WebC.74.00.0.181219
Date(s) Withdrawn	12/31/2022
Inclusion of Data in Results Report	No data was collected from WebChart EHR v7.4 during the 2023 RWT period.

Certification Criteria to be Tested

Care Coordination
- § 170.315(b)(1) Transitions of care (Cures Update)
- § 170.315(b)(2) Clinical information reconciliation and incorporation (Cures Update)
- § 170.315(b)(3) Electronic prescribing (Cures Update)
- § 170.315(b)(6) Data export
- § 170.315(b)(7) Security tags - summary of care - send (Cures Update)
- § 170.315(b)(8) Security tags - summary of care - receive (Cures Update)
- § 170.315(b)(9) Care plan (Cures Update)
Clinical Quality Measures
- § 170.315(c)(1)—record and export
- § 170.315(c)(2)—import and calculate
- § 170.315(c)(3)—report (Cures Update)
Patient Engagement
- § 170.315(e)(1) View, download, and transmit to 3rd party (Cures Update)
Public Health
- § 170.315(f)(1) Transmission to immunization registries
Application Programming Interfaces
- § 170.315(g)(7) Application access— patient selection
- § 170.315(g)(8) Application access— data category request
- § 170.315(g)(9) Application access— all data request (Cures Update)
- § 170.315(g)(10) Standardized API for patient and population services
Electronic Exchange
- § 170.315(h)(1) Direct Project

Criteria-Measure Matrix

Criteria	Requirement	Measure
§170.315(b)(1): Transitions of Care	(b)(1)(i)(A)(Alternative) - Send Using Edge Protocol for SMTP/IXE XDR	17
(b)(1)(i)(B)(Alternative) - Receive Using Edge Protocol for SMTP/IXE XDR	17
(b)(1)(i)(C)(Conditional) - XDM Processing	17
(b)(1)(ii)(A) - Receive, Parse, and Process	7, 19
(b)(1)(ii)(B) - View	7
(b)(1)(ii)(C) - Section Display	7
(b)(1)(iii) - Create	7
(b)(1)(iii)(A) - Assessment, Plan, Goals, Health Concerns	7
(b)(1)(iii)(B) - Diagnoses	7
(b)(1)(iii)(C) - Cognitive Status	7
(b)(1)(iii)(D) - Functional Status	7
(b)(1)(iii)(E) - Ambulatory Referral Summary	7
(b)(1)(iii)(F) - Inpatient Discharge Instructions	7
(b)(1)(iii)(G) - Patient Matching	7
§170.315(b)(2): Clinical information reconciliation and incorporation	(b)(2)(ii) - Correct Patient	7
(b)(2)(iii)(A) - Simultaneous Display	9
(b)(2)(iii)(B) - Reconciled List	9
(b)(2)(iii)(C) - User Review	9
(b)(2)(iii)(D) - List Acceptance	9
(b)(2)(iv) - CCD Creation	9
§170.315(b)(3): Electronic prescribing	(b)(3)(ii)(A)(1) - NewRx	3
(b)(3)(ii)(A)(2) - RxChangeRequest, RxChangeResponse	3
(b)(3)(ii)(A)(3) - CancelRx, CancelRxResponse	3
(b)(3)(ii)(A)(4) - RxRenewalRequest, RxRenewalResponse	3
(b)(3)(ii)(A)(5) - RxFill	3
(b)(3)(ii)(A)(6) - RxHistoryRequest, RxHistoryResponse	3
(b)(3)(ii)(A)(7) - Status	3
(b)(3)(ii)(A)(8) - Error	3
(b)(3)(ii)(A)(9) - Verify	3
(b)(3)(ii)(C)(1) - Primary/Secondary Diagnosis	4
(b)(3)(ii)(E) - Metric Units	5
(b)(3)(ii)(F) - Decimal Format	6
§170.315(b)(6): Data Export	(b)(6)(i) - Configure and export	18
(b)(6)(ii) - Set Export	18, 19
(b)(6)(ii)(A) - CCDS	18, 19
(b)(6)(ii)(B) - Diagnoses	18, 19
(b)(6)(ii)(C) - Cognitive Status	18, 19
(b)(6)(ii)(D) - Functional Status	18, 19
(b)(6)(ii)(E) - Ambulatory Reason for Referral	18, 19
(b)(6)(ii)(F) - Inpatient Discharge Instructions	18, 19
(b)(6)(iii)(A) - Timeframe configuration	18
(b)(6)(iii)(B) - Export summary	18
(b)(6)(iv) - Save location	18
§170.315(b)(7): Security tags - summary of care - send	(b)(7) - CDA Generated with Privacy & Security Markings	27
§170.315(b)(8): Security tags - summary of care - receive	(b)(8)(i) - Security Tags Document	28
(b)(8)(ii) - Preserve Privacy Markings	28
§170.315(b)(9): Care plan	(b)(9) - Record	24
(b)(9) - Change and Access	24
(b)(9) - Create	25
(b)(9) - Receive	26
§170.315(c)(1): CQMs – record and export	(c )(1)(i) - Report	1
(c )(1)(ii) - Export	1
§170.315(c)(2): CQMs – import and calculate	(c )(2)(i) - Import	2
(c )(2)(ii) - Calculate	1, 2
§170.315(c)(3): CQMs – report	(c )(3)(i) - Report	1, 2
§170.315(e)(1): View, download, and transmit to 3rd party	(e)(1)(i) - Web Content Accessibility	21
(e)(1)(i)(A) - View	14
(e)(1)(i)(A)(1) - USCDI	23
(e)(1)(i)(A)(3)(i) - Assessment and Plan of Treatment	23
(e)(1)(i)(A)(3)(ii) - Goals	23
(e)(1)(i)(A)(3)(iii) - Health Concerns	23
(e)(1)(i)(A)(4) - Provider Data	23
(e)(1)(i)(A)(6) - Laboratory Test Report	23
(e)(1)(i)(A)(7) - Diagnostic Imaging Report	23
(e)(1)(i)(B)(1)(i) - Download Human Readable	15
(e)(1)(i)(B)(1)(ii) - Download CCD	15
(e)(1)(i)(B)(2) - CCD Human Readable	15
(e)(1)(i)(C)(1)(i) - Email	16
(e)(1)(i)(C)(1)(ii) - Encrypted Transmission	16
(e)(1)(i)(D)(1) - Specific Date	14, 15, 16
(e)(1)(i)(D)(2) - Date Range	14, 15, 16
(e)(1)(ii)(A) - Activity Log	14, 15, 16
§170.315(f)(1): Transmission to immunization registries	(f)(1)(i) - Create Content	10
(f)(1)(ii) - Query Records	11
§170.315(g)(7): Application access – patient selection	(g)(7)(i) - Query processing and response	20
(g)(7)(ii)(A)(1) - Functional Documentation	8
(g)(7)(ii)(A)(2) - Implementation Requirements	8
(g)(7)(ii)(A)(3) - Terms of Use	8
(g)(7)(ii)(B) - Public Link	8
§170.315(g)(8): Application access – data category request	(g)(8)(i)(A) - Return CCDS data	20
(g)(8)(i)(B) - Request response	20
(g)(8)(ii)(A)(1) - Documentation	8
(g)(8)(ii)(A)(2) - Implementation Requirements	8
(g)(8)(ii)(A)(3) - Terms of Use	8
(g)(8)(ii)(B) - Public URL	8
§170.315(g)(9): Application access—all data request	(g)(9)(i)(A)(1) - Demonstrate API	20
(g)(9)(i)(A)(3) - Data Classes	20
(g)(9)(i)(B) - Data Return	20
(g)(9)(ii)(A)(1) - Documentation	8
(g)(9)(ii)(A)(2) - Implementation Requirements	8
(g)(9)(ii)(B) - Public URL	8
§170.315(g)(10): Standardized API for patient and population services	(g)(10)(i) - Data response: USCDI v1 + US Core STU v3.1.1	29, 30, 31
(g)(10)(ii) - Supported search operations	29, 30, 31
(g)(10)(iii) - Application registration	29, 30, 31
(g)(10)(iv) - Secure connection	29, 30, 31
(g)(10)(v)(A) - Authentication and authorization for patient and user scopes: SMART 1.0.0	29, 30
(g)(10)(v)(B) - Authentication and authorization for system scopes	29, 31
(g)(10)(vi) - Patient authorization revocation	29, 30
(g)(10)(vii) - Token introspection	30, 31
(g)(10)(vii) - Documentation	22
§170.315(h)(1): Direct Project	(h)(1)(i) - Send	12
(h)(1)(i) - Receive	13
(h)(1)(ii) - Message Disposition Notification: Processed	12
(h)(1)(ii) - Message Disposition Notification: Failed	12

Care Setting(s)

WebChart EHR is a scalable, web-based system designed for ambulatory practices and clinics. The same product is distributed to all care settings with many configuration options. Each practice can use the available configuration to tailor the product to fit their workflows and use requirements. Unless noted, testing for each measure was conducted for all care settings.

Care Setting	Justification
Primary Care	The WebChart EHR clients are divided primarily between primary care and specialty practices. Testing in a primary care setting will cover a large and important portion of our business.
Specialty Practice	The WebChart EHR clients are divided primarily between primary care and specialty practices. Configuration selections are all that differentiate WebChart EHR implementations; however, we will test with several specialty practices to ensure configuration does not impact the functionality of certified capabilities.
Pediatrics	Pediatric clinics are typically configured differently than adult primary care clinics. We will test in a pediatric setting in addition to primary care to again ensure that configuration does not impact the functionality of certified capabilities.
Small/Rural/Underserved Practice	The size and location of a practice can impact their interoperability options. We will test with both small/rural and large/urban practices to ensure all practices have full interoperability functionality.
Large Multi-practice Clinic	The size and location of a practice can impact their interoperability options. We will test with both small/rural and large/urban practices to ensure all practices have full interoperability functionality.

Summary of Testing Methods and Key Findings

WebChart EHR is a cloud-based, fully-inclusive EHR solution. All certified functionality is delivered in all instances of the product regardless of the care setting, size of practice, or required use cases for a given practice. Each production client is maintained in a separate database; however, the implementation of the environment is identical with the exception of optional increased security protocols that a client may choose to add for enhanced data protection. Additionally, the only differences between the client-facing portion of each system are a result of configuration settings that can be selected at go-live or updated at any time during a client’s contract. Due to this philosophy of product delivery, all certified capabilities may not be actively used in all marketed care settings or may not be actively used in any current client production system. To address the Real World Testing requirements, MIE will be using a hybrid approach. Testing will primarily be conducted using de-identified real patient data from production systems as recorded in database tables and log files. For those criteria for which this live production recording is not available or minimal due to lack of client usage, client reported issues will be tracked and reported in addition to enacting automated tests of the certified functionality in a test system in a production environment. The automated tests will be run daily or weekly as appropriate in a system that is identical in substance and delivery to a client production system with the only exception being live real patient data. This blended approach will allow MIE to prove ongoing maintenance of WebChart EHR’s certified technology regardless of the level of implementation by current clients.

Throughout 2023, we generally found a reduction of errors and misuse of modules when compared to 2022. The testing of e-Prescribing functionality, CCDA creation, data export, and documentation availability were particularly successful in production environments with real patient data. Conversely, many of WebChart EHR’s certified capabilities are still heavily underutilized by clients, especially use of the FHIR API, Direct Messaging, and special use of CDAs (care plan, security tags). These APIs present challenges for real world testing since test patients and environments, despite mirroring production systems, do not truly represent the end-user interoperability experience. Due to the development priority of supporting (b)(10), improving the robustness of the testing infrastructure did not receive the development attention as planned; however, MIE remains dedicated to both improving the testing infrastructure as well as continuing to educate clients regarding these features that are available and valuable to their practice.

Measures Used in Overall Approach

The following measures outline and justify how each requirement of all criteria to which WebChart EHR is certified will be tested during the 2023 Real World Testing year. Please review the Criteria-Measure Matrix above to review which measure(s) will cover a specific requirement.

Measure 1: Clinical Quality Measures Outgoing

Description

This measure will review WebChart EHR’s ability to measure clinical quality and export the required information. Compliance will be tested both manually by developers and clients as well as automatically by reporting bodies and the Cypress CUV+ test system.

Associated Certification Criteria

Certification Criteria	Requirement(s)
§170.315(c)(1): CQMs – record and export	(c )(1)(i) - Record

(c )(1)(ii) - Export
§170.315(c)(2): CQMs – import and calculate	(c )(2)(ii) - Calculate
§170.315(c)(3): CQMs – report	(c )(3)(i) - Report

Justification

WebChart EHR should accommodate the full range of §170.315(c)(1), §170.315(c)(2), and §170.315(c)(3) to support providers participating in MIPS and other quality measures. Most data supporting these measures for existing clients will come from data generated internally by their standard clinical workflows of seeing patients or incorporating the CCDA of transitioning patients. Numerical compliance calculations and reporting will be monitored by MIE and the practices selected for testing. The export and report QRDA formats will be validated by reporting partners and Cypress CUV+ to ensure data collected and calculated in WebChart EHR remains interoperable.

Test Methodology

First, MIE will install an instance of Cypress 7+ on our production servers following all of our protocols for maintaining the security of PHI. Cypress CUV+ supports the validation of QRDA reports containing PHI and will be used monthly to validate a random selection of QRDAs from the care settings identified. Any errors identified by Cypress CUV+ will be tracked, reported, and addressed, then followed with testing of a larger sample of files.

Additionally, WebChart EHR has two customers that participate in quarterly attestations using both QRDA I and QRDA III reports. These customers regularly inspect their CQM compliance numbers and will alert MIE to any perceived errors. MIE will then collect and track the attestation results from the reporting bodies including any errors so as to report a success/failure rate.

Results

	Calculations	QRDA I	QRDA III
Client Reported Issues	0	0	0
Submitted Files	N/A	4836	5
Submission Errors	N/A	0	0
Tested Files	N/A	4368	156
Testing Errors	N/A	0	0

Discussion

As expected, no errors were found in formatting or coding of the certified measures.

Measure 2: Clinical Quality Measures Incoming

Description

Associated Certification Criteria

Certification Criteria	Requirement(s)
§170.315(c)(2): CQMs – import and calculate	(c )(2)(i) - Import
(c )(2)(ii) - Calculate
§170.315(c)(3): CQMs – report	(c )(3)(i) - Report

Justification

WebChart EHR should accommodate the full range of §170.315(c)(1), §170.315(c)(2), and §170.315(c)(3) to support providers participating in MIPS and other quality measures. It is rare that an active production client will import a QRDA I file for use in their CQM calculations. To maintain that WebChart EHR is capable of importing and calculating when this does occur, QRDA I files from Cypress will be imported into a test system in a production environment, CQMs will be automatically calculated, and QRDA files will be exported back to Cypress for content and calculation validation.

Test Methodology

MIE will install an instance of Cypress 7+ on our production servers following all of our protocols for maintaining the security of PHI. Automated testing will download QRDA I files from Cypress for each certified CQM, import the files to WebChart EHR, calculate the CQMs, and export the QRDA files for Cypress validation of both the content and calculations to verify that the import was successful. Any errors identified by Cypress will be tracked, reported, and addressed.

Results

	QRDA I	QRDA III
Tested Files	4368	156
Testing Errors	0	0

Discussion

As expected, no errors were found in formatting or coding of the certified measures.

Measure 3: E-Prescribing Messages Sent and Received

Description

This measure will verify that all supported e-prescribing message types are in use in WebChart EHR, including inbound and outbound message types.

Associated Certification Criteria

Certification Criteria	Requirement(s)
§170.315(b)(3): Electronic prescribing	(b)(3)(ii)(A)(1) - NewRx
(b)(3)(ii)(A)(2) - RxChangeRequest, RxChangeResponse
(b)(3)(ii)(A)(3) - CancelRx, CancelRxResponse
(b)(3)(ii)(A)(4) - RxRenewalRequest, RxRenewalResponse
(b)(3)(ii)(A)(5) - RxFill
(b)(3)(ii)(A)(6) - RxHistoryRequest, RxHistoryResponse
(b)(3)(ii)(A)(7) - Status
(b)(3)(ii)(A)(8) - Error
(b)(3)(ii)(A)(9) - Verify

Justification

WebChart EHR should support all of the required e-prescribing messaging types outlined in §170.315(b)(3). Messages are stored locally in each client system in addition to being transmitted to/from pharmacies via the Surescripts network.

Test Methodology

MIE will report a count of messages for each supported message type:

* NewRx
* RxChangeRequest
* RxChangeResponse
* CancelRx
* CancelRxResponse
* RxRenewalRequest
* RxRenewalResponse
* RxFill
* RxHistoryRequest
* RxHistoryResponse
* Status
* Error
* Verify

The report will also include a count of outbound messages unable to be transmitted due to connectivity issues or other errors, for each message type. This report will be based on the contents of each client’s local database table of stored messages. MIE will run the report for each client under consideration and aggregate the results.

Results

Message Type	Client Message Counts	Total Count
NewRx	42246	128192	220909	9648	32279	433274
RxChangeRequest	4	2957	650	0	858	4469
RxChangeResponse	0	2471	346	0	869	3686
CancelRx	58	26	591	0	2	677
CancelRxReponse	58	23	574	0	1	656
RxRenewalRequest	12034	53759	13958	0	7267	87018
RxRenewalResponse	12368	52186	13843	0	7244	85641
RxFill	0	0	0	0	1782	1782
RxHistoryRequest	9	10	0	1	3	23
RxHistoryResponse	5	6	0	1	2	14
Status	110083	389337	430122	17831	82420	1029793
Error	550	2677	1060	83	292	4662
Verify	43477	150276	179860	8261	32206	414080

Total error rate: 4662 / 2065775 * 100 = 0.22%

Discussion

As expected, all supported message types have a greater than zero total message count, and the total number of messages far exceeds the number of errored messages with a total error rate of 0.22%. This is an overall reduction in error rate from 0.24% in 2022 to 0.22% in 2023. Additionally, the number of NewRx messages is significantly greater than the number of RxChangeResponse, CancelRx, and RxRenewalResponse messages.

Measure 4: E-Prescribing Diagnosis Codes

Description

This measure will verify that all diagnosis elements are present in some e-prescribing messages as required by §170.315(b)(3), including inbound and outbound message types.

Associated Certification Criteria

Certification Criteria	Requirement(s)
§170.315(b)(3): Electronic prescribing	(b)(3)(ii)(C)(1) - Primary/Secondary Diagnosis

Justification

WebChart EHR must be able to send Diagnosis codes in outbound e-prescribing messages, and receive inbound messages that include them.

Test Methodology

MIE will report the contents of each stored message in a client’s local database table of stored messages, and counts the inbound and outbound messages that include Diagnosis elements. MIE will run the report for each client under consideration and aggregate the results.

Results

Total NewRx Messages	NewRx Messages with Diagnosis Included
433274	141266

Total rate of diagnosis use: 141266 / 433274 * 100 = 32.6%

Discussion

Since the Diagnosis elements are not a required component of a NewRx message, as anticipated, only a subset (32.6%) of the NewRx messages included a diagnosis. The rate of NewRx messages with a diagnosis included increased from 28.99% in 2022 to 32.6% in 2023 showing an increasing rate of using the diagnosis element. Most clients appear to heavily use the diagnosis element, with only one tested client rarely including a diagnosis and therefore reducing the overall usage rate. Additional client education will be provided in 2024 to further increase usage.

Measure 5: E-Prescribing Oral Liquid Units

Description

This measure will verify that prescriptions for medications with an oral liquid form will have a quantity unit of measurement of mL, not cc or English units as outlined in §170.315(b)(3).

Associated Certification Criteria

Certification Criteria	Requirement(s)
§170.315(b)(3): Electronic prescribing	(b)(3)(ii)(E) - Metric Units

Justification

WebChart EHR should prevent prescriptions of oral liquid medications from being sent electronically if they have an inappropriate quantity unit of measurement.

Test Methodology

MIE will create a system report that examines the contents of each stored NewRx message in a client’s local database table of stored messages, limiting to oral liquid medications, and provides a count of each distinct quantity unit of measure used. MIE will run the report for each client under consideration and aggregate the results. Units of Presentation

Results

Unit Code	Unit Description	Quantity
C28254	Milliliter (ml)	14460
C48155	Gram (g)	1
C48477	Bottle	0
C48480	Capsule	1
C48521	Packet	4
C48542	Tablet	7
C64933	Each	15
C48504	Kit	2

Total rate of incorrect units: 30 / 14490 * 100 = 0.21%

Discussion

As expected, C28254 (milliliters) is the most commonly sent unit of measure for oral liquid medications. Non-C28254 units were only sent in 0.21% of oral liquid medication messages in 2023 which was a decrease from 0.24% sent in 2022.

Measure 6: E-Prescribing Decimal Format

Description

This measure will verify that numeric amounts in prescriptions include leading zeros before decimal points and do not allow trailing zeros after a decimal point.

Associated Certification Criteria

Certification Criteria	Requirement(s)
§170.315(b)(3): Electronic prescribing	(b)(3)(ii)(F) - Decimal Format

Justification

WebChart EHR should prevent prescriptions from being sent electronically if they have directions or total quantity that are missing leading zeros or include trailing zeros. This is essential for preventing misunderstanding by pharmacists regarding the amount to dispense and patients regarding the amount of medication to take.

Test Methodology

MIE will create a system report that examines the contents of each stored NewRx message in a client’s local database table of stored messages, and provides a count of prescription messages that include inappropriate trailing zeros, and a count of those missing leading zeros. MIE will run the report for each client under consideration and aggregate the results.

Results

Total NewRx Messages	NewRx Messages with Improper Decimal Format
437559	2545

Discussion

As expected, the number of NewRx messages sent with inappropriate trailing zeros, or missing leading zeros, occurs rarely in only 0.58% of messages; however, this is again an increase from the 0.43% seen across 2022 and 0.55% seen in Q1 of 2023. Both technical and training options will be investigated to reduce the incidence of improper decimal formats in 2024.

Measure 7: CDA Download

Description

This measure will verify that the system can accept a CDA document uploaded into the system, assign it to the appropriate chart in the system as appropriate, and display the document with a standard stylesheet with all sections being accepted and visible.

Associated Certification Criteria

Certification Criteria	Requirement(s)
§170.315(b)(2): Clinical information reconciliation and incorporation	(b)(2)(ii) - Correct patient.
§170.315(b)(1): Transitions of Care	(b)(1)(ii) - All paragraphs
(b)(1)(iii) - All paragraphs

Justification

Webchart EHR should be able to accept a CDA document and place it into the correct chart based on information within the document. It should also be able to display the CDA documents with an appropriate stylesheet.

Test Methodology

MIE will report on the number of CDA formatted documents uploaded into tracked WebChart EHR systems and the number of upload attempts that failed as stored in client databases and error log files.

MIE will report on the number of requests to view a CDA document within the system, and the number of times it displayed correctly, and when there were errors in display.
Any errors reported by customers or the recipients of their quarterly attestations will be tracked and reported as a baseline. These test assumptions for customer reporting align with the “visual inspection” aspects of the test lab tests.

Results

Documents	Views
93198	28543

Discussion

Significantly more documents were uploaded into WebChart EHR than were viewed; however, all 28543 document views that did occur were successful. No errors were reported by clients.

Measure 8: Application Access Documentation

Description

This measure will verify that WebChart EHR’s API documentation is publicly and perpetually available. Compliance will be recorded by an external uptime monitor and reported quarterly. Upon request, or in the event of downtime, data can additionally be reported in daily, weekly, or monthly increments.

Associated Certification Criteria

Certification Criteria	Requirement(s)
§170.315(g)(7): Application access – patient selection	(g)(7)(ii)(A)(1) - Functional Documentation
(g)(7)(ii)(A)(2) - Implementation Requirements
(g)(7)(ii)(A)(3) - Terms of Use
(g)(7)(ii)(B) - Public Link
§170.315(g)(8): Application access – data category request	(g)(8)(ii)(A)(1) - Documentation
(g)(8)(ii)(A)(2) - Implementation Requirements
(g)(8)(ii)(A)(3) - Terms of Use
(g)(8)(ii)(B) - Public URL
§170.315(g)(9): Application access—all data request	(g)(9)(ii)(A)(i) - Documentation
(g)(9)(ii)(A)(ii) - Implementation Requirements
(g)(9)(ii)(B) - Public URL

Justification

WebChart EHR should provide public access to all API documentation, implementation requirements, and terms of use as outlined in 170.315(g)(7), 170.315(g)(8), and 170.315(g)(9). This documentation should be available at all times throughout the year.

Test Methodology

An external uptime monitor will check the availability of all documentation available at https://docs.webchartnow.com/resources/system-specifications/application-programming-interface-api.html . Both up- and downtime will be logged to be reported quarterly. The cause of any downtime and the duration will also be logged In the event of any downtime, the amount of downtime can be reported at daily, weekly, or monthly intervals in addition to the quarterly reports, and the cause of each downtime occurrence will be reported.

Results

The MIE API documentation was available 99.975% of the time in 2023 representing an overall increase from an availability of 99.967% in 2022.

The MIE API documentation was available 99.991% of Q1. There was a 6 minute down period on March 8, 2023 and a 1 minute down period on March 9, 2023 due to a connection timeout.

The MIE API documentation was available 99.914% of Q2. There was a 1 hour 51 minute down period on June 13, 2023 due to a connection timeout.

The MIE API documentation was available 100% of Q3.

The MIE API documentation was available 99.995% of Q4. There was a 1 minute down period on November 30, 2023 and a 5 minute down period on December 15, 2023 due to a connection timeout.

Discussion

As expected, the documentation maintained an uptime of greater than 99.9% at 99.975% for the year. During any down time the API documentation would not have been available, but, no effect to end user requests for the information was reported. Data regarding the usage of the API can be viewed in Measure 20.

Measure 9: Clinical Information Reconciliation and Incorporation

Description

This measure will verify that the system can take a CCDA transition of care/referral summary formatted according to the standards adopted §170.205(a)(3) and §170.205(a)(4) and read the data for medications, allergies, and conditions from the document, reconcile those into the chart, and that the data is fully incorporated into the chart.

Associated Certification Criteria

Certification Criteria	Requirement(s)
§170.315(b)(2): Clinical information reconciliation and incorporation	(b)(2)(iii)(A), (B), (C), (D)
(b)(2)(iv) - System Verification

Justification

WebChart EHR should be able to reconcile CCDA data for medications, allergies, and conditions into a patient’s chart as outlined in § 170.315(b)(2).

Test Methodology

MIE will report on the number of CDA formatted documents reconciled via our reconciliation process.

Following each reconcile, if a temporary CDA for the chart is created as part of the process, it will be validated to ensure the reconciled data can be incorporated into a CDA created free of schematic errors (the CDA document will NOT be kept, only the result of the validation). Additionally, any client complaints that data is not being imported correctly from the tool will be tracked, investigated, and reported.

Results

CDA Documents reconciled:

Total Reconciled Documents	Schematically Valid	Schematically Invalid
740	309	431

Discussion

All documents marked as schematically invalid were from a 3rd party interface sent into the system; however, all invalid documents still imported successfully and can be viewed in the system in human-readable format.

Measure 10: Transmission to Immunization Registry: Create Content

Description

This measure will verify that the system can generate a VXU conforming to the HL7 v2.5.1 standard, CDC guidance for communication to Immunization Registries and state/local guidance. The VXU messages shall contain information related to the demographics and vaccination administration record.

Associated Certification Criteria

Certification Criteria	Requirement(s)
§170.315(f)(1): Transmission to immunization registries	(f)(1)(i) - Create Content

Justification

WebChart EHR should be able to generate and send valid VXU messages.

Test Methodology

MIE will report from the database the number of successfully sent VXU messages acknowledged as received by the state immunization registry. MIE will also report from the database on the number of records rejected by the state registry due to error, whether the failure was due to registry internal errors, clinical data entry issues or a not well-formed message. Finally, MIE will report from the database the number of messages which declined to be generated due to data entry issues failing message pre-validation.

Results

Successful	Failures	Data Entry Validation Issue
8258	7	7

Discussion

All 7 of the Failures were related to user data entry. For two of the failures, the user entered an administration location which was then changed to an outside location. The outside location was not a valid reporting location causing the error. The remaining error occurred when the date of administration was after the expiration date of the vaccination and was rejected by the registry.

For the 7 Data Entry Validation issues, 3 were duplicate records which were deleted, two did not have a valid CVX code selected, one did not have an administration date entered, and one vaccination record had a manufacturer not recognized by the CDC.

Measure 11: Transmission to Immunization Registries: Query Records

Description

This measure will verify that the system can generate a QBP conforming to the HL7 v2.5.1 standard, CDC guidance for communication to Immunization Registries and state/local guidance. Furthermore, the system shall be able to retrieve, consume and display to the end user the results of any such query.

Associated Certification Criteria

Certification Criteria	Requirement(s)
§170.315(f)(1): Transmission to immunization registries	f)(1)(ii) - Query Records

Justification

WebChart EHR should be able to request, consume and display an evaluated patient history and forecast.

Test Methodology

MIE will report the number of successful retrievals of evaluated history and forecasting operations from the database. MIE will report the number of failed retrievals, including those resulting from an internal error in the registry resulting in an inability to consume a response from the database. MIE will manually track, resolve and report issues resulting from WebChart EHR application errors as reported by end users.

Results

Successful: 26580

Failure: 501

Discussion

The failure rate is very low with clusters of failures around a few points in time. This is to be expected due to network, server or other technology related issues.

Measure 12: Direct Project: Send

Description

This measure will verify that the system can transmit a Direct project conforming S/MIME to a HISP. The measure will also verify the receipt of those transmissions by verifying the status of the resultant MDN messages.

Associated Certification Criteria

Certification Criteria	Requirement(s)
§170.315(h)(1): Direct Project	(h)(1)(i) - Send
(h)(1)(ii) - Message Disposition Notification: Processed
(h)(1)(ii) - Message Disposition Notification: Failed

Justification

WebChart EHR should be able to generate valid S/MIME messages, transmit them via Direct Project specifications and consume the resulting MDN from the recipient.

Test Methodology

MIE will report from log files the number of messages transmitted. MIE will report from logs the number of messages which failed to be transmitted whether due to internal error, external failures or inability to verify trust of the recipient. MIE will report from logs the number of Processed MDN messages received. MIE will report from logs the number of Failed MDN messages received.

Results

Transmitted: 3

Failed: 20

Processed MDNs: 3

Failed MDNs: 0

Discussion

The messages which failed to send fall into a handful of categories. The majority of the failures were the 17 messages with untrusted recipients. 3 failed due to our own certificate being expired which upon notification of the expiration was updated.

Measure 13: Direct Project: Receive

Description

This measure will verify that the system conforms to Direct Project message receipt requirements for validation.

Associated Certification Criteria

Certification Criteria	Requirement(s)
§170.315(h)(1): Direct Project	(h)(1)(i) - Receive

Justification

WebChart EHR should be able to receive, validate and deliver Direct Project messages transmitted to its HISP.

Test Methodology

MIE will report from logs the number of messages transmitted to the HISP. MIE will report from logs the number of messages failing to conform to Direct Project specifications. MIE will report from logs the number of messages which are successfully delivered to recipients.

Results

Received: 1545

Failed specifications: 3

Delivered messages: 1545

Discussion

All inbound messages were successfully delivered. There were 3 messages received which were delivered successfully but the system was unable to generate an MDN response due to a missing start boundary in the original message.

Measure 14: Patient Portal View

Description

This measure will verify that a patient can view various document types within the patient portal.

Associated Certification Criteria

Certification Criteria	Requirement(s)
§170.315(e)(1): View, download, and transmit to 3rd party	(e)(1)(i)(A)(1),(2),(3),(4),(5)
(e)(1)(i)(D)(1), (2)
(e)(1)(ii)(A)

Justification

WebChart EHR should be able to provide a mechanism for a patient to read documents sent to them within a patient portal as required by § 170.315(e)(1).

Test Methodology

MIE will report a number of measurements surrounding documents, including:

Number of documents made available to patients in the patient portal
Number of documents read by patients in the patient portal
Number of failures in the ability to read messages in the patient portal

Results will be retrieved from database tables and aggregated for reporting. Any failures will be reported from the information found in log files as well as any client reported issues tracked during the testing period.

Results

There were 15 views of 10 CDA documents in 2023 in tracked client systems.

Discussion

Both portal usage and, especially, CDA document viewing by patients are very low. It was anticipated that through further education, testing would be more robust using real patient data. To address this low volume in 2024, MIE will continue to educate for portal usage while also implementing more robust data collection on test data in production environments.

Measure 15: Patient Portal Download

Description

This measure will verify that a patient can download various document types within the patient portal.

Associated Certification Criteria

Certification Criteria	Requirement(s)
§170.315(e)(1): View, download, and transmit to 3rd party	(e)(1)(i)(B)(1), (2), (3)
(e)(1)(i)(D)(1), (2)
(e)(1)(ii)(A)

Justification

WebChart EHR should be able to provide a mechanism for a patient to download documents sent to them within a patient portal.

Test Methodology

MIE will report a number of measurements surrounding documents, including:

Number of documents made available to patients in the patient portal
Number of documents successfully downloaded from the patient portal
Number of documents unsuccessful in being downloaded from the patient portal.

Results will be retrieved from database tables and aggregated for reporting. Any failures will be reported from the information found in log files and third party reports as well as any client reported issues tracked during the testing period.

Results

There were 8 streams of 3 CDA documents in 2023 in tracked client systems.

Discussion

Both portal usage and, especially, CDA document downloading by patients are very low. It was anticipated that through further education, testing would be more robust using real patient data. To address this low volume in 2024, MIE will continue to educate for portal usage while also implementing more robust data collection on test data in production environments.

Measure 16: Patient Portal CCDA Transmit

Description

This measure will verify that a patient can transmit various document types within the patient portal to other entities.

Associated Certification Criteria

Certification Criteria	Requirement(s)
§170.315(e)(1): View, download, and transmit to 3rd party	(e)(1)(i)(C)(1), (2)
(e)(1)(i)(D)(1), (2)
(e)(1)(ii)(A)

Justification

WebChart EHR should be able to provide a mechanism for a patient to transmit documents sent to them within a patient portal to other entities.

Test Methodology

MIE will report a number of measurements surrounding documents, including:

Number of documents made available to patients in the patient portal
Number of documents successfully transmitted from the patient portal
Number of documents unsuccessful in being transmitted from the patient portal.

Results

3 exports of 2 CDA documents were reported in 2023 in tracked client systems.

Discussion

Both portal usage and, especially, CDA document transmission by patients are very low. It was anticipated that through further education, testing would be more robust using real patient data. To address this low volume in 2024, MIE will continue to educate for portal usage while also implementing more robust data collection on test data in production environments.

Measure 17: Send Using Edge Protocol for SMTP / XDM

Description

This measure will verify that the system is able to utilize a SMTP edge protocol for sending and receiving Direct Project messages. As part of receiving messages, XDM shall be handled when applicable.

Associated Certification Criteria

Certification Criteria	Requirement(s)
§170.315(b)(1): Transitions of Care	(b)(1)(i)(A)(Alternative) - Send Using Edge Protocol for SMTP/IXE XDR
(b)(1)(i)(B)(Alternative) - Receive Using Edge Protocol for SMTP/IXE XDR
(b)(1)(i)(C)(Conditional) - XDM Processing

Justification

WebChart EHR should be able to receive and send Direct Project messages to a HISP utilizing a SMTP edge.

Test Methodology

MIE will report from logs the number of messages transmitted to the HISP by SMTP. MIE will report from logs the number of messages received from the HISP by SMTP. MIE will report from logs the number of XDM packages processed. In the case where insufficient real-world data is available, data resulting from regular testing with DirectTrust shall be included in the reporting.

Results

Transmitted to HISP via SMTP: 1545

Transmitted by HISP: 3

XDM packages processed: 505

Discussion

A growing number of incoming Direct Messages contained XDM packages. These were processed successfully by the system. Outbound Direct Message usage is still low. To address this low volume in 2024, MIE will continue to educate for portal usage while also implementing more robust data collection on test data in production environments.

Measure 18: Data Export

Description

This measure will verify that a user can use WebChart EHR’s Data Export Tool to pull down groups of patient data from a WebChart EHR system.

Associated Certification Criteria

Certification Criteria	Requirement(s)
§170.315(b)(6): Data Export	(b)(6)(i)
(b)(6)(ii)(A)-(F)
(b)(6)(iii)(A)-(B)
(b)(6)(iv)

Justification

WebChart EHR should be able to provide a mechanism for a user to download patient chart information via CDA from a large set of patients within the system as outlined in §170.315(b)(6). This tool is publicly available (https://github.com/mieweb/wcexport) .

Test Methodology

MIE will report from the event log database tables a series of occurrences that indicates use of the WebChart EHR Data Export Tool:

Event logs of the report to find all patients for Document Export being called.
Event logs of CDA documents being generated within a certain short time period following the report.

MIE will track customer reports of data expected to be in mass data export downloads that did not download as failures.

Results

A mass data export of 12,563 CDA documents occurred in a client system in early November. The documents generated were known to have been used in another system for import later on in the month. All 12,563 documents were successfully imported.

Discussion

The mass export of CDA documents is a rarely used feature; however, as expected, all documents were successfully generated and successfully imported into the new system.

Measure 19: CDA Validation

Description

This measure will verify that CDAs both created by and received by a WebChart EHR system pass basic CDA validation.

Associated Certification Criteria

Certification Criteria	Requirement(s)
§170.315(b)(1): Transitions of Care	(b)(1)(ii)(A)
§170.315(b)(6): Data Export	(b)(6)(ii), (A)-(F)

Justification

WebChart EHR should be able to validate that CDAs that are stored within WebChart EHR either do or do not conform to basic CDA schema requirements.

Test Methodology

All CDAs stored within a WebChart EHR will be run through schema validation regardless of the document’s origin. Documents may originate within the WebChart EHR system or be imported from a third party application of manual upload. The schema validator will be installed within the MIE production environment to ensure the security of all PHI contained in the documents. Only results of the validation will be made available, document content will not be revealed to developers during testing.

The number of valid vs. invalid CDAs and their sources will be reported.

Results

All Documents:

Valid CDAs: 1079

Invalid CDAs: 2319

Discussion

The bulk of these results are from 3rd party uploaded documents.

For the WebChart EHR generated documents, most errors occurred because of vitals data coming from external systems was stored incorrectly and put into the CCDA document in that incorrect form (ie, a value of “5 ft 4 in”, when PQ values can only be numeric). To prevent such “pass-along” errors in the future, new policies will be investigated and implemented in 2024 to ensure better generated CDA validity.

Measure 20: Patient Data requests VIA API

Description

This measure will verify that the API as outlined in WebChart EHR’s documentation is functional. A valid request for patient information must provide that information.

Associated Certification Criteria

Certification Criteria	Requirement(s)
§170.315(g)(7): Application access – patient selection	(g)(7)(i) - Query processing and response
§170.315(g)(8): Application access – data category request	(g)(8)(i)(A) - Return CCDS data
(g)(8)(i)(B) - Request response
§170.315(g)(9): Application access—all data request	(g)(9)(i)(A)(1) - Demonstrate API
(g)(9)(i)(A)(3) - Data Classes
(g)(9)(i)(B) - Data Return

Justification

WebChart EHR should provide patient information to requesters with the proper access to the information. In production environments of WebChart EHR, the use of the documented API is rare; therefore, MIE will conduct dual level testing of the API first, using automated testing of a test system in a production environment and second, manually tracking any client reported issues with the API functionality against the automatically tracked API requests are made.

Test Methodology

To address the overall automated testing, the following test requests will be made daily against a test system in a production environment.

Issue a request in the browser to search for a patient (patient selection)
Issue a request in the browser to request demographics of a patient (data category request)
Issue a request using the export tool described in the documentation.

All API requests made in production systems are recorded in log files. The number of requests logged will be reported against the number of issues with API functionality that are reported.

Results

Production Exports	5
Total Charts Exported	481258
Total Export Errors	0

Discussion

Five total production API exports occurred in 2023. Three of these exports produced CSV files and two produced PDF files for patient charts. All exports had an initial failure rate of <1%. These initial export errors were rerun error-free and 100% of data was delivered as expected.

Measure 21: Web Content Accessibility

Description

This measure will verify that all certified content in the patient portal will maintain accessibility conformance as outlined in the Web Content Accessibility Guidelines (WCAG) 2.0.

Associated Certification Criteria

Certification Criteria	Requirement(s)
§170.315(e)(1): View, download, and transmit to 3rd party	(e)(1)(i) - Web Content Accessibility

Justification

The certified content of the patient portal should be accessible to all users regardless of abilities or impairments as outlined in the Web Content Accessibility Guidelines (WCAG) 2.0.

Test Methodology

MIE will conduct monthly third-party production accessibility scanning as well as automated nightly internal accessibility scanning of a test system in a production environment.

Results

The internal accessibility scanning of a pre-production test system identified 0 urgent and 0 secondary non-conformance issues in 99.13% of nightly and ad-hoc scans throughout the year. In the remaining 0.87% of scans only secondary issues were uncovered, all of which were eliminated prior to code changes reaching production systems. Production accessibility scanning identified 0 urgent and 0 secondary non-conformance issues for the entire year.

Discussion

As expected no urgent non-conformance issues were identified in either live production or pre-production test systems. In the rare occurrence that a secondary non-conformance issue was identified in testing, it was addressed and eliminated prior to reaching live production systems.

Measure 22: FHIR API Documentation

Description

This measure will verify that WebChart EHR’s FHIR API documentation is publicly and perpetually available. Compliance will be recorded by an external uptime monitor and reported quarterly. Upon request, or in the event of downtime, data can additionally be reported in daily, weekly, or monthly increments.

Associated Certification Criteria

Certification Criteria	Requirement(s)
§170.315(g)(10): Standardized API for patient and population services	(g)(10)(vii) - Documentation

Justification

WebChart EHR should provide public access to all FHIR API documentation, software components, software configurations, registration instructions, and terms of use as outlined in 170.315(g)(10). This documentation should be available at all times throughout the year.

Test Methodology

An external uptime monitor will check the availability of all documentation available at https://docs.webchartnow.com/resources/system-specifications/fhir-application-programming-interface-api/ and the linked subpages. Both up- and downtime will be logged to be reported quarterly. The cause of any downtime and the duration will also be logged In the event of any downtime, the amount of downtime can be reported at daily, weekly, or monthly intervals in addition to the quarterly reports, and the cause of each downtime occurrence will be reported.

Results

The FHIR API documentation was available 99.975% of the time in 2023 representing an overall increase from an availability of 99.967% in 2022.

The FHIR API documentation was available 99.991% of Q1. There was a 6 minute down period on March 8, 2023 and a 1 minute down period on March 9, 2023 due to a connection timeout.

The FHIR API documentation was available 99.914% of Q2. There was a 1 hour 51 minute down period on June 13, 2023 due to a connection timeout.

The FHIR API documentation was available 100% of Q3.

The FHIR API documentation was available 99.995% of Q4. There was a 1 minute down period on November 30, 2023 and a 5 minute down period on December 15, 2023 due to a connection timeout.

Discussion

As expected, the documentation maintained an uptime of greater than 99.9% at 99.975% for the year. During any down time the FHIR API documentation would not have been available, but no effect on end user requests for the information was reported.

Measure 23: CCDA Content

Description

This measure will verify that CCDAs generated in WebChart EHR systems have all USCDI data and other required data.

Associated Certification Criteria

Certification Criteria	Requirement(s)
§170.315(e)(1) View, download, and transmit to 3rd party	(e)(1)(i)(A)(1) - (e)(1)(i)(A)(7)

Justification

WebChart EHR should generate CCDAs that can generate the sections required by USCDI.

Test Methodology

We will have tests that will choose a certain number of random patient CCDAs in specific live systems and test for the given sections to exist in the documents.

Results

On a quarterly basis, USCDI data elements have been entered on 2 test patients in live WebChart EHR systems. Each quarter, 1 CCDA document was generated for each test patient. All 8 CCDA documents contained all USCDI data elements.

Additionally, the 28543 CCDA documents viewed in participating WebChart EHR systems and the 309 schematically valid reconciled CCDA documents were compared to the patients’ charts. In all cases, all USCDI data available in the chart was displayed in the documents.

Discussion

All testing completed with test patients in live systems was successful, demonstrating that all USCDI elements can be included in CCDAs. All live patient data tested was also successful; however, not all patient charts contained all USCDI data elements. More robust testing of all USCDI elements is planned for 2024.

Measure 24: Record and Change Care Plan

Description

This measure will track that users can create and change care plan data.

Associated Certification Criteria

Certification Criteria	Requirement(s)
§170.315(b)(9): Care plan	Record, Change and Access

Justification

WebChart EHR per certification requirements must be able to provide a way to enter and update Care Plan data.

Test Methodology

We will report on the following data elements being created or edited in patient charts:

Goals
Health concerns
Health status evaluations and outcomes
Interventions

Results

On a quarterly basis, CDA Care Plan information was entered on test patients in properly configured live WebChart EHR systems. CCDA documents were then generated for those patients. In all cases the Care Plan information was present in the CCDA.

The following is the number of records of each element in the live systems tested:

Element	Number of Charts with Element Entered per Quarter
Care Plan Goals	4
Health Concerns	3
Health Status	2
Interventions	2

Discussion

Currently, no live clients are using the CCDA related Care Plan fields in their workflows; however, 100% of completed tests were successful.

Measure 25: Create Care Plan CCDA Documents

Description

This measure will track that users can create Care Plan CCDA Documents.

Associated Certification Criteria

Certification Criteria	Requirement(s)
§170.315(b)(9): Care plan	Requirement from matrix

Justification

WebChart EHR per certification requirements must be able to generate Care Plan CCDA Documents.

Test Methodology

We will report on the number of encounters with Care Plan information, and the number of Care Plan CCDAs generated.

Results

On a quarterly basis, CDA Care Plan information was entered on test patients in properly configured live WebChart EHR systems. CCDA documents were then generated for those patients.

Element	Number of Charts with Element Entered per Quarter	Number of CCDAs with Element Present per Quarter	Total Successful Tests
Care Plan Goals	4	4	16
Health Concerns	3	3	12
Health Status	2	2	8
Interventions	2	2	8

Discussion

In 2023, no live clients were generating Care Plan CCDAs as part of their workflows; however, 100% of completed tests were successful.

Measure 26: Receive Care Plan CCDA Documents

Description

This measure will track that the system can receive Care Plan CCDA Documents.

Associated Certification Criteria

Certification Criteria	Requirement(s)
§170.315(b)(9): Care plan	Requirement from matrix

Justification

WebChart EHR per certification requirements must be able to receive Care Plan CCDA Documents.

Test Methodology

We will report on:

* the number of Care Plan CCDAs received from outside sources.
* Pass or fail count on the Care Plan CCDAs received.

Results

On a quarterly basis, CCDA documents with Care Plan information were imported to test patients in live WebChart EHR systems. CCDA documents with Care Plan information were imported for 4 test patients separate from the test patients used in Measures 24 and 25.

4 CCDA documents were uploaded each quarter for a total of 16 documents. All 16 were successful.

Discussion

As expected, all manual testing was successful. Currently, no clients are set up to receive Care Plan CCDA documents as part of their workflow. Whether any live clients will receive CCDA Care Plan documents as a live part of their workflow is not certain, so expanded manual live testing will be implemented in 2024.

Measure 27: Create CCDA Documents with Security Tags

Description

This measure will track that the system can create CCDA Documents with valid security tags.

Associated Certification Criteria

Certification Criteria	Requirement(s)
§170.315(b)(7): Security tags - summary of care - send	(b)(7)

Justification

WebChart EHR per certification requirements must be able to generate CCDA Documents with valid security tags.

Test Methodology

We will have automated tests that run at minimum weekly to test that the software is still able to generate CCDAs with Security Tags.

From discussions with others around the industry who interact with large usage of CDA creation and transmission, there is little to no usage of DS4P within documents created by systems currently. If we determine that we are seeing usage of the security tagging within Production systems, we will report:

* the number of CCDAs received during the RWT period.
* The number of CCDAs with security tags received during the RWT period.

Results

On a quarterly basis, CDA documents were generated on test patients with security set on the encounter in live WebChart EHR systems. Across the year, 23 documents with security tags were generated. All 23 documents were reviewed and contained the appropriate security tags based on the encounter settings. Additionally, all 23 documents were confirmed as valid CCDAs. These documents were then imported into a test system, and all were confirmed to have successfully imported the appropriate security.

Discussion

Since there are no indications that the security tags will be broadly used within live workflows, more robust manual testing will be implemented in 2024 along with client education that this feature is available.

Measure 28: Receive and Display CCDA Documents with Security Tags

Description

This measure will track that the system can receive CCDA Documents with security tags and properly display them to end users.

Associated Certification Criteria

Certification Criteria	Requirement(s)
§170.315(b)(8): Security tags - summary of care - receive	(b)(8)(i) and (ii)

Justification

WebChart EHR per certification requirements must be able to receive CCDA Documents with security tags and properly display them to end users.

Test Methodology

We will have automated tests that run at minimum weekly to test that the software is still able to generate CCDAs with Security Tags.

* the number of CCDAs received during the RWT period.
* The number of CCDAs with security tags received during the RWT period.

Results

Discussion

Measure 29: FHIR Sandbox Testing

Description

This measure will use the Inferno Test suite to validate all types of secure connections and search operations supported by the FHIR API within a publicly available production sandbox system.

Associated Certification Criteria

Certification Criteria	Requirement(s)
§170.315(g)(10): Standardized API for patient and population services	(g)(10)(i) - Data response: USCDI v1 + US Core STU v3.1.1
(g)(10)(ii) - Supported search operations
(g)(10)(iii) - Application registration
(g)(10)(iv) - Secure connection
(g)(10)(v)(A) - Authentication and authorization for patient and user scopes: SMART 1.0.0
(g)(10)(v)(B) - Authentication and authorization for system scopes
(g)(10)(vi) - Patient authorization revocation

Justification

WebChart EHR’s FHIR API is still newly available to clients and has no adoption as of writing this plan. Therefore to cover testing prior to live clients actively using the API, a publicly available production sandbox will be tested using Inferno. FHIR adoption is expected to be slow, but increasing, throughout 2023 leading to improved app support in WebChart EHR as well as increased real world data being available, at which time, Measures 30 and 31 will provide a more complete view of the production FHIR capabilities.

Test Methodology

MIE will run weekly automated testing on the public FHIR R4 sandbox system using Inferno, and using log files stored in a QA database, MIE will report the success rate of the full (g)(10) test suite. Any errors will be tracked, reported, and addressed.

Results

Beginning in Q3, weekly testing of the FHIR Sandbox using Inferno was conducted since no live clients are currently using FHIR. 15 of these tests were fully successful. Weekly tests that were not successful were due to internal issues with test server availability. The FHIR Sandbox environment was always available, but the test system accessing it had unexpected unavailability impacting our ability to test the sandbox.

Discussion

When the test server was available for connection to the FHIR Sandbox, FHIR Inferno testing was successful. MIE plans to both continue with weekly testing beginning from 01/01/2024 and work with clients to robustly test real world scenarios throughout all of 2024.

Measure 30: FHIR Patient Scope

Description

This measure will review WebChart EHR’s ability to connect to an app within a patient scope and provide the user with the requested data.

Associated Certification Criteria

Certification Criteria	Requirement(s)
§170.315(g)(10): Standardized API for patient and population services	(g)(10)(i) - Data response: USCDI v1 + US Core STU v3.1.1
(g)(10)(ii) - Supported search operations
(g)(10)(iii) - Application registration
(g)(10)(iv) - Secure connection
(g)(10)(v)(A) - Authentication and authorization for patient and user scopes: SMART 1.0.0
(g)(10)(vi) - Patient authorization revocation

Justification

WebChart EHR’s FHIR API is still newly available to clients, and has no adoption as of writing this plan. FHIR adoption is expected to be slow, but increasing, throughout 2023 leading to improved app support in WebChart EHR as well as increased real world data being available. Until that time when clients are actively using the FHIR API, MIE will conduct testing using a publicly available production sandbox system and a patient app recommended to our clients. As clients continue adoption of the FHIR API, real patient use of the patient app will be reported.

Test Methodology

MIE will report from de-identified log files an analysis of authentication and data searches using a patient app. Specific rates can be reported from the sandbox system as the automated testing setup will indicate what actions should yield successful authentication or data return. An overall analysis will be reported for the real world patient data since we cannot estimate failures due to patients correctly being denied access.

Results

No client systems have generated FHIR patient connections yet.

Successful Sandbox connections have been made via:

Our Inferno testing tool : 658 successful Patient scope FHIR connections.
The CommonHealth app: 14 successful Patient scope FHIR connections.

Discussion

MIE is in active discussions with the CommonHealth app to connect both the FHIR Sandbox and a live client for portal patients. We also have plans on getting Apple Health Kit connectivity in 2024.

Measure 31: FHIR EHR Provider Scope

Description

This measure will review WebChart EHR’s ability to connect to an app within an EHR provider scope and provide the user with the requested data.

Associated Certification Criteria

Certification Criteria	Requirement(s)
§170.315(g)(10): Standardized API for patient and population services	(g)(10)(i) - Data response: USCDI v1 + US Core STU v3.1.1
(g)(10)(ii) - Supported search operations
(g)(10)(iii) - Application registration
(g)(10)(iv) - Secure connection
(g)(10)(v)(B) - Authentication and authorization for system scopes

Justification

WebChart EHR’s FHIR API is still newly available to clients, and has no adoption as of writing this plan. FHIR adoption is expected to be slow, but increasing, throughout 2023 leading to improved app support in WebChart EHR as well as increased real world data being available. Until that time when clients are actively using the FHIR API, MIE will conduct testing using a publicly available production sandbox system and a provider app recommended to our clients. As clients continue adoption of the FHIR API, real provider use of the provider app will be reported.

Test Methodology

MIE will report from de-identified log files an analysis of authentication and data searches using a patient app. Specific rates can be reported from the sandbox system as the automated testing setup will indicate what actions should yield successful authentication or data return. An overall analysis will be reported for the real world provider data since we cannot estimate failures due to providers correctly being denied access.

Results

111 successful Sandbox connections to our Inferno testing tool were made for EHR provider access.

Discussion

The Live RW sandbox systems have shown our ability to do EHR provider connections. We will be looking for opportunities to connect client systems with EHR provider FHIR endpoints in 2024.

Schedule of Key Milestones

Key Milestone	Care Setting	Date/Timeframe	Status
Release of documentation for the Real World Testing to be provided to ACB and providers	All settings	November 1, 2022	Complete
Begin collection of information as laid out by the plan	All settings	January 1, 2023	Complete
Follow-up with providers and authorized representatives to understand any issues arising with the data collection.	All settings	Quarterly, 2023	Complete
Data collection and review.	All settings	Quarterly, 2023	Complete
New certification of b.10	All settings	Q3, 2023	Complete, Q4
Additional CQM or criteria certification as determined by the developer	All settings	Q3, 2023	f.5 certification complete, Q4
End of Real World Testing period/final collection of all data for analysis	All settings	December 31, 2023	Complete
Data analysis and report creation	All settings	January, 2024	Complete
Submission of Real World Testing Results to ACB	All settings	Jan 29, 2024, resubmitted on Feb 29, 2024

Attestation

This Real World Testing plan is complete with all required elements, including measures that address all certification criteria and care settings. All information in this plan is up to date and fully addresses the health IT developer’s Real World Testing requirements.

Authorized Representative Name	Doug Horner
Authorized Representative Email	horner@mieweb.com
Authorized Representative Phone	260-459-6270
Authorized Representative Signature	Doug Horner
Date	02/29/2024

Enterprise Health Documentation

Last Updated: Thu, 29 Feb 2024 19:50:36 UTC

Last Build: Fri, 26 Jul 2024 16:24:35 UTC
WikiGDrive Version: dd69069d725fca5f553df7ded62e130a49d49ca6