← Anuja Saini

Project

Usage → Success Score → Renewal Risk → Next Best Step

2021 to 2024

A health score for every Salesforce customer, a forecast of who renews, and a recommender for what each should do next.

01The question

Which customers are about to leave, and what should each of them do next?

Salesforce sells subscriptions, so renewal is the moment that matters. Every customer leaves a monthly trail of adoption, learning and technical health. Turn it into a forecast, a score, and a next step.

Diagram: monthly product usage, learning and technical signals feed an LSTM renewal forecast; SHAP attributes the forecast to each signal; the attributions roll up into three weighted per-metric scores (adoption 50%, expertise 20%, technical health 30%) and one 0 to 100 score, which goes to the customer's Help portal, the account team and the recommender.Diagram: monthly product usage, learning and technical signals feed an LSTM renewal forecast; SHAP attributes the forecast to each signal; the attributions roll up into three weighted per-metric scores (adoption 50%, expertise 20%, technical health 30%) and one 0 to 100 score, which goes to the customer's Help portal, the account team and the recommender.
Where the score comes from, and where it goes.

02Part one · Will they renew?

The score

The Customer Success Score: one 0 to 100 number per customer, from product adoption, customer expertise and technical health, refreshed monthly. Customers see it in the Help portal; account teams plan around it. Winner of the 2023 TSIA STAR Award for Innovation in Customer Success.

The Customer Success Score dashboard for a demo org: a gauge reading 61, Excellent, up 8 from last month, beside the three category scores: product adoption 82, customer expertise 58, technical health 29.
The dashboard, on Salesforce's demo org: one gauge, three categories, change since last month.
Trending signals grouped under product adoption, customer expertise and technical health, each with a score and its month-over-month change.
Underneath, the signals behind each category. This is where the account team looks next.
Customer Success Scoreadoption 50% · expertise 20% · technical health 30%fair · good · excellentTSIA STAR Award 2023

The predictor

Renewals are seasonal and every history is a different length; a tabular model reading one month at a time misses the shape. A two-layer LSTM reads the whole sequence. Three iterations: a fixed six-month window (dropped 23 percent of opportunities), padded variable windows (dropped none), then one model per distance to renewal. The third held.

two-layer LSTM, 100 + 50 unitsdropout 0.2 / 0.1MSE, Adampadded and masked sequencesone model per months-to-renewalXGBoost baseline

Results

MAE on normalised renewal amount6 months out3 months out1 month out
XGBoost baseline0.380.370.13
LSTM, one model per horizon0.190.200.17
Held-out renewals. The baseline is only good in the last month, when it is too late to act; the LSTM holds six months out.
Line chart of mean absolute error against months before renewal for four models. The XGBoost baseline and the single-window LSTMs swing between 0.09 and 0.40 depending on horizon; the per-horizon LSTM stays flat near 0.19.Line chart of mean absolute error against months before renewal for four models. The XGBoost baseline and the single-window LSTMs swing between 0.09 and 0.40 depending on horizon; the per-horizon LSTM stays flat near 0.19.
All four models tried. The flat line ships.

From forecast to score

A forecast says how much, not why. SHAP gives every attribute its signed share of each prediction: renewal looks low because licence utilisation fell, despite strong certifications. Shares are summed into the metric each attribute belongs to, and those per-metric scores are product adoption, customer expertise and technical health. Causal analysis then separates the factors that move attrition from the ones that merely travel with it.

  1. forecast
  2. SHAP per attribute
  3. roll up per metric
  4. success score
SHAPsigned, per customer, per monthattribute → metric roll-upcausal inference

In production

Seasonality lives in the model, so retraining dropped from monthly to twice a year. Retraining, scoring and attribution run end to end in Airflow.

1B+ forecasts a monthAirflowrefresh: days → hoursdrift checked quarterly

03Part two · What should they do next?

The problem

A low adoption score says a customer is behind, not what to read. The first recommender was a classifier with one class per piece of content: new content meant retraining, and new customers got nothing.

Two towers

A two-tower model embeds customer and content in one space and scores a pair by dot product. New content needs an embedding, not a retrain; a new customer is placed by attributes alone.

Two-tower architecture: customer features and content features each pass through three dense layers to an embedding; the two embeddings meet in a dot product and sigmoid giving the probability the customer engages with the content. The loss is cross-entropy plus four penalties.Two-tower architecture: customer features and content features each pass through three dense layers to an embedding; the two embeddings meet in a dot product and sigmoid giving the probability the customer engages with the content. The loss is cross-entropy plus four penalties.
Two towers, one dot product.
two-tower retrievalnegative sampling 5:1 to 2:1explicit negative feedbackarticles · Trailhead modules · coaching sessions

Teaching it judgement

Plain cross-entropy recommends the popular thing to everyone. Four penalties fix that: repeating a product, pushing products already well adopted, starving rare content types, suggesting products the customer does not own. A holistic layer then blends model and history by account age and guarantees a mix.

diversity penaltyhigh-adoption penaltycontent-imbalance weightingno-score penaltyholistic layer
One customer through the holistic layer: history of 60% articles, 30% modules, 10% coaching; the model says 80% modules; blended by account age to 55/40/5; a diversity rule guarantees at least one of each type; a thumbs-down on articles shifts future recommendations to advanced modules.One customer through the holistic layer: history of 60% articles, 30% modules, 10% coaching; the model says 80% modules; blended by account age to 55/40/5; a diversity rule guarantees at least one of each type; a thumbs-down on articles shifts future recommendations to advanced modules.
One customer through the holistic layer.

Results

  • +15 to 20%recall and precision vs the classifier
  • +13%coverage
  • +10%recall@20
  • +6%diversity

04Impact

  • $15M+revenue retained
  • 97K+hours saved for account teams
  • One number both sides see, a forecast that holds six months out, and the reasons behind it.
  • A next step for every customer, including the ones with no history yet.
  • A manual monthly refresh became a scheduled pipeline, retrained twice a year.

05Reading