Vendor monitoring programs rarely fail because the data is wrong. They fail because the alerting layer treats every observable change as worth a human's attention. Within a quarter, the team has learned that alerts are noise, and the one signal that mattered scrolls past unread. Threshold design is therefore not a tuning detail. It is the difference between a monitoring program and a subscription you ignore.
Start from the decision, not the data
Every alert should map to a decision someone is expected to make. If no decision changes, no alert should fire. Before setting a single threshold, write down the only four decisions vendor monitoring can actually trigger:
- Log it and watch (no human action now)
- Ask the vendor a question (targeted inquiry, days)
- Pull the reassessment forward or restrict usage (formal action, this week)
- Invoke incident procedures (contract notices, containment, today)
Your severity tiers should be these four decisions wearing labels. Everything below is a mechanism for routing observations to the right one.
Severity tiers that mean something
Three user-facing tiers are enough. More than three and the boundaries blur; fewer and you cannot separate "ask a question" from "act now."
| Tier | Definition | Illustrative examples | Decision it triggers |
|---|---|---|---|
| Minor | Deviation from baseline with no plausible near-term exploitation path | Single certificate nearing expiry, new subdomain, small score dip that self-corrects | Log and watch; roll up in weekly digest |
| Moderate | Sustained or compounding drift that contradicts assessed posture or expands attack surface | Score decline persisting past the confirmation window, newly exposed admin interface, SPF/DMARC regression on a vendor that attested to email hardening | Targeted vendor inquiry within 5 business days |
| Severe | Evidence of active compromise, public breach disclosure, or collapse of a control your risk acceptance depended on | Credential dump containing vendor domains, ransomware group claim, critical vulnerability confirmed on an internet-facing asset holding your data | Escalate same day; trigger incident and reassessment workflow |
Note what is absent: raw score values. A score of 72 is not a severity. Severity comes from the delta, the persistence, and the vendor's criticality. Grading drift into tiers before anything reaches a human is the single highest-leverage design choice in the whole system; it is also how platforms like ThirdSentry handle it natively, tiering drift as Minor, Moderate, or Severe before any user-facing alert is created.
Per-tier thresholds, scaled by vendor criticality
The same observation deserves different urgency on different vendors. Anchor thresholds to your vendor tiering framework so that criticality does the scaling for you. An illustrative starting matrix, assuming a 0 to 100 score scale:
| Signal | Tier 1 (critical vendors) | Tier 2 (important) | Tier 3 (standard) |
|---|---|---|---|
| Score drop from assessed baseline | 5+ points sustained 7 days = Moderate; 10+ = Severe review | 10+ points sustained 14 days = Moderate | 15+ points sustained 30 days = Moderate |
| New internet-facing service | Alert on any | Alert on risky categories (admin panels, databases, remote access) | Digest only |
| Breach or leak intelligence | Severe, same day | Severe, same day | Moderate, 5-day inquiry |
| Certificate or email auth regression | Moderate | Minor, escalate if unresolved 30 days | Minor |
| Contradiction of an attested control | Moderate minimum, regardless of score impact | Moderate | Minor with digest flag |
Treat these numbers as a calibration starting point, not gospel. The tuning rule that matters: review threshold performance quarterly, and adjust until roughly 80 percent of Moderate-and-above alerts result in a real action. If most alerts end in "closed, no action," the thresholds are too tight to the noise floor.
Baseline discipline: the part everyone gets wrong
Drift is only meaningful relative to a baseline, and the baseline must be an assured state, not a moving average. Three rules keep it honest:
- Set the baseline at assessment completion. The moment a reassessment closes, the observed external posture at that time becomes the new reference. You now know what "good" looked like when the vendor last proved it.
- Never reset on score recovery. If a vendor's score sags 12 points and crawls back over six weeks, that round trip is information. Silently re-baselining at the bottom, or on recovery, erases it. The baseline moves only when a completed reassessment says the new state has been examined.
- Record baseline provenance. Every baseline should carry the assessment ID and date that produced it. When an auditor asks why an alert did or did not fire in March, this is the answer.
Suppression rules that prevent fatigue without hiding risk
Suppression is where fatigue is actually solved, and where risk gets hidden if you are careless. Safe suppression follows four patterns:
- Deduplicate by finding, not by event. One expired certificate observed daily is one finding with an age counter, not thirty alerts.
- Use confirmation windows for score-based signals. Require the condition to persist (7, 14, or 30 days by tier, per the matrix above) before promoting it beyond a log entry. Transient scanner noise dies in the window; real drift survives it.
- Suppress downward, never across. It is acceptable to hold Tier 3 cosmetic findings in a digest. It is never acceptable to suppress a Severe signal on any tier, or to let a vendor-level mute swallow breach intelligence.
- Time-box every suppression. A mute without an expiry date is a permanent blind spot. Cap suppressions at 90 days with mandatory re-review, and log who approved each one.
Escalation paths: who hears what, and when
A numbered escalation procedure removes the "who owns this" delay that costs the most time in real events:
- Minor: auto-logged to the vendor record; weekly digest to the vendor risk analyst. No tickets, no pages.
- Moderate: ticket assigned to the vendor's relationship owner and the risk analyst, 5-business-day SLA to either close with rationale or convert to a vendor inquiry. Unanswered inquiries auto-escalate at day 10.
- Severe: immediate notification to the TPRM lead and security on-call; vendor owner engaged same day; decision within 24 hours on whether to invoke the incident process, contractual notice clauses, or an out-of-cycle reassessment. Every Severe alert gets a written disposition, even if the disposition is "false positive," because that record is what tunes the system and satisfies the audit.
Close the loop
Threshold design is not a one-time setup task. Feed disposition data back monthly: which alerts produced action, which were noise, which real events produced no alert at all. The last category, the silent misses, matters most and is only visible if you compare monitoring output against what reassessments and divergence between assessed and live posture later uncover. A program that reviews its own alert quality quarterly will converge on a signal-to-noise ratio the team actually trusts, which is the entire point of a continuous vendor monitoring program.
If you would rather not build the tiering, baselining, and suppression machinery by hand, ThirdSentry ships it as the default: drift graded Minor, Moderate, or Severe before anyone is alerted, baselines that reset only on completed reassessments, and escalations wired to the same vendor record your assessments live on.
