This is the first post in a series about how the Data Science team at Claritas Rx uses machine learning across the specialty pharmacy patient journey. We’re starting with the problem we spend the most time on: keeping patients on the therapies that help them.
—
The big issue
Specialty medications and rare disease therapies are complex, often expensive treatments for conditions like cancer, autoimmune disease, and rare genetic disorders. For the patients who depend on them, staying on treatment is rarely straightforward. A prior authorization lapses. A copay becomes unaffordable. A side effect goes unmanaged, or a refill gets missed and never picked back up. Any one of these can quietly end a therapy the patient still needs.
The industry calls this discontinuation, or DC. The people whose job is to prevent the avoidable kind are Patient Service Advocates (PSAs), who help patients get past the access, affordability, and adherence hurdles sitting between a prescription and an actual dose. Their problem is scale: one PSA may be responsible for thousands of patients, and they can’t call everyone. Every morning the real question is which patients need attention first, and why.
That’s a machine learning problem. It’s the one our discontinuation model was built to answer.
What the model predicts
One question, answered for every active patient, every day: how likely is this patient to discontinue therapy in the next 90 days?
Why 90? In our data a discontinuation takes about 60 days to become visible (the missed refill, the reauthorization that stalls), and there’s a 30-day grace period after that before anyone formally counts the patient as stopped. So 90 days is roughly the window in which a PSA can still change the outcome. Much shorter and there’s no time to act on the prediction. Much longer and you’re forecasting someone’s life circumstances a year out.
The model outputs a calibrated probability, which we bucket into High, Medium, and Low so a PSA can triage at a glance. The cutoffs are set separately for each therapy, and that choice matters more than it sounds. Baseline discontinuation rates vary enormously between, say, an oral oncology drug and a chronic biologic; one global threshold would flag nearly every patient on the first and almost nobody on the second. Setting each therapy’s High tier at roughly its own top quarter keeps every worklist at a size a person can get through.
One model, or twenty?
We score patients on more than twenty therapies. Some are chronic conditions with tens of thousands of patients, some are rare-disease launches with a few hundred. People ask whether we build one model for everything or one per therapy, and the answer is that we tried both and liked neither. The single model washed out patterns that were obvious within a given therapy. The per-therapy models were great for big populations and no so good for small ones.
We ended up with three tiers and a router. Therapies with deep history get a dedicated model. Clinically related therapies share a grouped model (for example we run one for hematology, oncology and one for chronic organ-protective therapies ) on the theory that patients on related drugs discontinue for related reasons, which has held up empirically. Everything else, including brand-new launches, falls to a unified portfolio model. So a new therapy gets scored from its first day on the platform, and graduates to something more specific as its history builds.
A grouped model only ships if it beats the unified model on held-out, out-of-time data for every therapy it covers, not just on average. The ones in production sit between 0.83 and 0.90 out-of-time AUC. And if a specialized model ever misbehaves, the router falls back a tier.
From data to decision
Here’s the whole flow at a glance, before we get into the parts that hurt:
Daily flow from patient data through feature engineering, the risk model, SHAP explanations and the uplift re-ranker, to a ranked daily output consumed by Patient Service Advocates
Inputs: demographics, pharmacy claims and refill sequences, insurance coverage and prior authorization status, payer changes, interactions with the healthcare system. Outputs: a risk tier, the factors driving it, a short list of suggested interventions with talking points, and a priority rank reflecting how much a call is likely to help. The package is deliberately more than a score, because a score on its own doesn’t tell a PSA what to do at 9am.
What makes this harder than it sounds
A version of this model that looks great in a notebook and falls apart in production is easy to build. Four considerations
Leakage. The classic failure is training on information that won’t exist at prediction time. Our favorite example is “days on therapy.” It looks like a perfectly reasonable feature until you notice it only keeps growing while the patient hasn’t discontinued, so it more or less contains the answer. A model trained on it aces validation and is useless in the field. Every feature we compute now comes from a strict as-of view of history, and anything suspiciously predictive gets interrogated before it’s allowed in. We also cut calendar features like sample month (correlated with outcomes, causal of nothing), and we drop training rows sampled within 90 days of our data cutoff, because those patients’ outcomes aren’t observable yet and there’s no label there to train on.
Late-arriving data. Pharmacy claims aren’t real-time. A shipment from yesterday might not reach our feeds for days or weeks, but a retrospective training set has all of it sitting there, neatly filled in. Train on that and the model learns to lean on recent-claims signals it will never have on the morning it scores, and its live accuracy sags below the offline numbers with nothing to tell you why. Our fix: the feature builders hide the most recent 30 days of claims at training time, so the model trains in the same fog it will operate in. The 30 wasn’t a guess, by the way. We started at a conservative 60, backtested against what the pipeline had historically been able to see on each scoring day, and 30 matched reality.
Sampling. Our first training set pooled every patient at many points in their history — a “panel,” in the jargon. Sounds thorough. What it does in practice is over-represent long-tenured patients, since they contribute the most rows, and months after launch our drift monitors caught the scored population pulling steadily away from the training population. The rebuild used point-in-time snapshots instead: reconstruct the active population exactly as it stood on each of eleven monthly dates, and train on those. Training and inference are now aligned by construction. Drift settled almost immediately, and every model we’ve built since starts from snapshots.
Explanations. No PSA is going to act on “Patient #84213 is high risk,” and frankly they shouldn’t. For every prediction we compute which factors drove it, using SHAP values (a game-theoretic method for splitting a prediction into per-feature contributions), then translate those into plain language and map them to a library of interventions with talking points that adapt to the patient’s therapy stage, coverage, and urgency. What lands in front of the PSA reads more like: high risk, mostly because the prior auth expires soon and refills are slipping — make these two calls first. We didn’t bolt explainability on for presentation. Without it, the scores simply don’t get used.
From “who is at risk” to “who we can help”
Risk and actionability are different things
Some high-risk patients will discontinue no matter what anyone does. Plenty of low-risk patients are low-risk precisely because they’re stable and don’t need a call. The patients worth a PSA’s limited hours are the ones whose outcome a call can change, and nothing about a risk score identifies them.
So there’s a second model, an uplift model, that estimates how much a PSA’s outreach would lower a given patient’s risk. (For the causal-inference readers: it’s a doubly robust learner.) We use it to re-rank the high-priority list so the hours go where they change outcomes.
Fair warning if you ever build one of these on observational data: our first naive estimates said outreach “helped” about three quarters of patients. That number says nothing about phone calls and a lot about who got called. Outreach was never randomly assigned; it went to patients someone had already decided were worth helping, and the data inherits that bias. Doubly robust methods correct for much of it, but we still keep the raw effect estimates internal. PSAs see a bounded benefit score and a priority rank within their therapy and tier. We can defend the ranking; the magnitudes are shakier, so we don’t show them.
Uplift modeling gets a full post later in the series. It’s earned one.
Getting into production
The pipeline runs itself every morning. Before the outreach teams sit down, it has pulled the current active population, rebuilt every feature as of that date, scored patients across the portfolio, generated the explanations and recommendations, and written everything to the warehouse.
What made this reliable was a mindset shift: the model is a system we operate, not a thing we shipped. The loop looks like this:
The MLOps loop: train on point-in-time snapshots, validate, deploy versioned models, score daily, and monitor for drift and calibration — feeding back into retraining when decay is detected
A few of the pieces, concretely. Model artifacts are versioned and stored centrally, and the morning job pulls whatever version is marked current, so rolling back means moving a pointer rather than cutting an emergency release. Input drift is watched with population-stability metrics, applied with some judgment — a couple of our legitimately sparse features would flag “critical drift” every day forever if we let them, and an alarm everyone has learned to ignore protects nothing. Calibration is checked continuously against observed outcomes, with per-therapy recalibration when the probabilities wander.
Even the recalibration has guardrails now: corrections too small to matter get skipped, and the maximum shift is capped. We added those after watching a calibrator keep “correcting” a therapy whose underlying model had drifted back toward the truth on its own. The fix had become the bug.
Most of what goes wrong in production is mundane, and our most instructive outage had nothing to do with modeling. Every new image gets a validation run before the next scheduled job is allowed to rely on it, and every failure path pages somebody.
Tech stack
Nothing here is exotic, and that’s on purpose. Python throughout. XGBoost (gradient-boosted decision trees) for the risk models, SHAP for the explanations, and a doubly robust learner from the scikit-learn ecosystem for uplift. The daily job runs in Docker on AWS managed compute on a fixed schedule with no servers to babysit, reading from and writing to the same Snowflake warehouse the downstream teams already use. Dashboards and alarms live in the cloud platform’s own observability stack: one for pipeline health, one for model outputs, one for drift. It’s a dull stack, wired together carefully, and that’s most of the trick.
Why we work this way
We could optimize a single accuracy number and call it done. But everything that has moved patient outcomes for us has lived in the less glamorous corners: temporal logic that’s exactly right, training data sampled the way the world arrives, predictions that someone without a statistics degree can act on, and honesty about the difference between who’s at risk and who we can help.
That’s data science at Claritas Rx, and the discontinuation model is one example of it. Later in the series: uplift modeling, explainability in practice, and the engineering that keeps these systems trustworthy after the launch excitement wears off.
*Interested in this kind of work, as a partner or a future teammate? We’d like to hear from you.*
