Prototype results
The prototype runs today. All figures below come from published test data, or from counts on a live export with no patient data shown.
On the published test decks
Section titled “On the published test decks”| Engine | Measures | Result |
|---|---|---|
| fqm-execution | CMS122, CMS125, CMS130 | All 185 published test cases reproduced, with every patch disclosed |
| HAPI FHIR Clinical Reasoning | CMS122, CMS125, CMS165 | 116 exact matches and 73 differences across 189 cases, none unexplained |
On the 121 cases both engines ran, fqm-execution matches all 121 with patches disclosed. HAPI matches 105 and explains the other 16.
Throughput is about 0.2 seconds per patient on HAPI against a loaded deck.
Two engine defects that fail silently
Section titled “Two engine defects that fail silently”All 73 HAPI differences trace to two engine defects. Neither produced an error. Both returned a plausible MeasureReport.
- Unit conversion. Converting a quantity returns null when the source unit is a CQL calendar duration. That nulls every days-supply calculation in the shared medication library and fails 17 medication-dependent exclusions across three decks. Patching four call sites restores all 17.
- Measurement-period precision. The measurement period is built at second precision, so resources dated in the period’s final second compare as null. On CMS165 this removes 54 of 68 patients from the initial population.
Both defects are documented for upstream reporting.
On a live eClinicalWorks export
Section titled “On a live eClinicalWorks export”The breast cancer screening measure (CMS125) on a real eClinicalWorks bulk export:
| Run | Performance rate |
|---|---|
| Published measure package, as published | 0%, with no error and no warning |
| With the OpenDQM adapter layer | 68.47% |
| Our production calculator, same patients | 68.15% |
The numerator matched exactly, at 291 patients.
The production calculator used for this comparison is itself validated against a Cypress-certified QRDA Category I path. On a live athenahealth practice the screened rate agreed at 87%, against the QRDA control at 84%. This is the validation chain described in Validating FHIR results.
Explore this run in the example run report.
Limits, stated
Section titled “Limits, stated”- Whether a retrieve should match a resource that declares no profile is an engine setting the measure community has not settled. The calculator conforms the data rather than change the setting.
- No rule has yet been clinically signed.
- The retrieve tracer, fault-injected test decks and bias report are designed but not yet built.
- Every run to date is ambulatory.