
HTI-5 Changed What Certification Tells Buyers. The Evidence Healthcare Organizations Need Did Not Change.
Two assurance mechanisms are being reduced: transparency about predictive technology and the Real World Testing evidence buyers could point back to.
Less required evidence does not mean less need for evidence. |
|---|
Healthcare organizations are already skeptical of artificial intelligence, predictive models, clinical decision support, and other technology that can influence care. For years, one answer to that skepticism has been the ONC certification framework. It included requirements intended to make certain predictive tools more transparent, and it required certified developers to produce Real World Testing documentation about how certified capabilities performed in real-world use.
HTI-5 changes both assurance mechanisms. It removes key certification requirements that supported transparency and risk-management information for predictive decision support. It also removes annual Real World Testing plans and narrows detailed Real World Testing results reporting.
These are different changes, but they create the same problem for healthcare buyers: less required evidence does not mean less need for evidence.
Certification still matters. Buyers also need to understand what the certification framework will no longer tell them, what documentation may no longer be required, and what they now need vendors to show.
Assurance Loss #1: Predictive Technology Transparency
Healthcare buyers usually hear the language of AI. ONC uses a more specific regulatory term for some of these tools: predictive decision support interventions, or predictive DSIs. The terminology matters for the rule, but the buyer issue is simpler: these are predictive technologies in certified health IT that can influence the information people use to make decisions.
For years, the certification framework required standardized source-attribute information intended to help users look behind certain predictive tools. That information could help expose how a tool was developed, what it was intended to do, information about the data behind it, how it performed, where its limitations were, and other details relevant to evaluating the technology.
HTI-5 removes the source-attribute support requirement, the requirement to provide access to and modification of those source attributes, and the predictive DSI intervention risk-management requirement.
The practical consequence is significant: healthcare buyers can no longer assume ONC certification will surface the same standardized transparency and risk-management information for predictive technology. The technology did not become easier to evaluate. The questions did not become less important. The buyer now has to make sure those questions still get answered.

Assurance Loss #2: Real World Testing Evidence
The second change is broader than AI. Real World Testing gave healthcare organizations something a certification label alone could not: required documentation about certified technology being exercised in real-world settings. That documentation gave buyers evidence they could point back to when the certification label and their own implementation experience did not seem to match.
A buyer could ask: If this capability is certified, what exactly was tested? What did the results show? What conditions, workflows, users, settings, or versions were involved? If the product demonstrated this capability for certification and Real World Testing, show us how that same capability is supposed to work in our EHR.
HTI-5 does not eliminate Real World Testing altogether. But it removes annual Real World Testing plans and narrows detailed results reporting. That means less required documentation for buyers to fall back on when they need to connect a certification claim to actual product behavior.

What buyers should do with both changes
The response to both assurance losses is the same: build an evidence chain that does not depend on certification alone.
Ask what the applicable certification established. Request the vendor documentation supporting the claim. Review available Real World Testing evidence. Require the vendor to demonstrate the capability using your workflows, data realities, exception paths, handoffs, and configuration. Document what requires customization, third-party technology, future development, or additional services. Then validate the deployed capability locally and monitor whether it continues to perform as expected.
A useful buyer question is simple: “If certification says this capability exists, show us the evidence, then show us how we reproduce that capability here.”
Certification can establish a baseline. The buyer still has to establish that the technology works in the organization’s reality.
Certification and buyer assurance are not the same question:
WHAT CERTIFICATION REQUIRES | WHAT OUR ORGANIZATION REQUIRES |
What information must be supplied to satisfy the applicable certification criteria? | What information do our clinicians, leaders, governance committees, security teams, compliance teams, and procurement teams need to make and sustain a responsible decision? |
Those questions overlap, but they are not identical. When the first becomes narrower, the second becomes more important.
What HTI-5 does not make less important
The removal or narrowing of a certification requirement should not be interpreted as proof that the underlying information is unnecessary. It also should not be treated as evidence that:
Predictive technology no longer needs validation.
Bias and subgroup performance no longer matter.
Known limitations no longer need to be disclosed.
Healthcare organizations can skip local testing or governance review.
Model changes do not need oversight.
A vendor demonstration is enough to establish safety, value, or workflow fit.
Every AI capability falls under the same certification, regulatory, or oversight pathway.
A regulatory change may alter what is required through one mechanism. It does not erase the clinical, operational, ethical, contractual, and governance questions that remain.
For predictive technology, buyers still need answers
HTI-5 changes what may be required through certification. It does not make the underlying questions disappear. For predictive technology, buyers should still be able to address the following before implementation and throughout the technology lifecycle.
1. INTENDED USE
What problem is this technology designed to solve?
Who is expected to use the output, and what decision is the output intended to support?
What uses are outside the product’s intended scope?
What human review is expected before action is taken?
2. DATA AND REPRESENTATION
What data were used to develop, train, tune, or test the technology?
Who was represented in that data?
Which populations, settings, diagnoses, geographies, workflows, or care environments were underrepresented or absent?
How similar is the development and validation data to our organization’s patients and operating environment?
3. VALIDATION AND PERFORMANCE
How was performance validated?
Was validation internal, external, prospective, retrospective, or independent?
What outcomes and performance measures were used?
How does performance vary across sites, populations, subgroups, and clinical conditions?
What evidence supports use in settings like ours?
4. LIMITATIONS AND UNCERTAINTY
What limitations are known?
When is the output most likely to be incomplete, misleading, or wrong?
How does the technology communicate uncertainty?
What should a user do when the technology is uncertain or conflicts with clinical judgment?
5. BIAS AND SUBGROUP PERFORMANCE
How is bias assessed?
Which subgroups were evaluated separately?
What material performance differences were found?
What monitoring or mitigation is used when disparities appear?
6. WORKFLOW FIT AND HUMAN FACTORS
Where does the output appear in the real workflow?
Does the technology add cognitive load, alerts, review steps, duplicate work, or documentation burden?
What happens during handoffs, downtime, missing data, or other exception paths?
Can users understand why the technology produced the output?
7. CHANGE GOVERNANCE
How are model, data, threshold, workflow, or user-interface changes governed?
What changes trigger notice, review, testing, or revalidation?
How will the organization know that the deployed capability is different from what was originally evaluated?
Can the organization delay or reject a material change?
8. MONITORING AFTER IMPLEMENTATION
What should be monitored locally?
What baseline should be established before launch?
Who owns performance review, incident review, and escalation?
How will the organization detect drift, unintended consequences, automation bias, or declining performance?
What is the process for pausing or retiring the technology?
Do not replace lost assurance with another polished demo
Less required documentation should not push buyers toward the other weak form of assurance: a polished vendor demonstration. Predictive-technology demos can make complex work look simple. A curated patient produces a clear recommendation. An ideal workflow moves from input to insight without hesitation.
Healthcare rarely behaves that way. Real data may be incomplete, duplicated, delayed, mapped inconsistently, entered in free text, or carried forward from prior encounters. Real patients have comorbidities. Real workflows include handoffs, missing information, conflicting signals, time pressure, and exceptions.
Evaluate the average day, not the best day. Require the demonstration to include the organization’s real scenarios, difficult cases, exception paths, incomplete data, and realistic handoffs. Score what the technology actually demonstrates. Document promised, future, customized, or unvalidated capability separately.
ASK. | DOCUMENT. | VALIDATE. | MONITOR. |
If the information matters, put access to it in the deal
Organizations should decide which information is essential for procurement, implementation, governance, and ongoing use. If the organization’s oversight depends on that information, do not rely on informal assurances or assume it will always remain available.
Depending on the technology and the organization’s risk profile, procurement and contracting discussions may need to address:
Documentation of intended use, validation, known limitations, and relevant subgroup performance.
Access to current technical and user documentation before implementation.
Notice of material changes to models, data sources, thresholds, logic, workflow behavior, or intended use.
Expectations for local validation, acceptance testing, and revalidation after significant change.
Cooperation with monitoring, safety review, incident investigation, and corrective action.
Clear ownership for documentation updates and change communication.
Options to pause, disable, or retire the capability if performance or safety concerns emerge.
Contract language should be reviewed by the organization’s legal, compliance, privacy, security, clinical, and procurement teams. The practical principle is straightforward: if responsible governance requires the information, establish how the organization will obtain and maintain it.
Build governance around the technology lifecycle
A one-time purchase review is not enough. Predictive technology can change because the model changes, the data changes, the workflow changes, the patient population changes, or the way people respond to the output changes.
A durable governance process should connect:
Executive sponsorship and defined accountability.
Clinical, operational, technical, compliance, privacy, security, legal, quality, and procurement review.
Frontline users who understand how the technology behaves in real work.
Pre-implementation validation and clear success measures.
Change control, version awareness, and revalidation triggers.
Ongoing monitoring, incident review, escalation, mitigation, and retirement criteria.
The objective is not to create bureaucracy around every algorithm. The objective is to match the rigor of the review to the consequence of the decision. Technology that influences clinical judgment, prioritization, treatment, access, or resource allocation warrants a governance approach proportionate to that impact.
The bottom line
HTI-5 reduces two things healthcare buyers have relied on: standardized transparency about predictive technology and required Real World Testing plans and broader results reporting.
Neither change makes the underlying questions less important. Buyers still need to know what a predictive capability was designed to do, what shaped it, how it performed, where it can fail, and how its risks are managed. For certified health IT more broadly, buyers still need evidence connecting the certification claim to real product behavior.
The strongest response is not to dismiss certification. It is to use certification as the beginning of the evidence trail instead of the end of the conversation.
Certification requirements changed. The buyer's need for evidence didn't.

