Signal Detection at Scale: Methods and Validation

Safety signal detection represents the earliest opportunity to identify previously unknown or incompletely characterized adverse drug reactions. A signal, as defined by the WHO Uppsala Monitoring Centre, is "information arising from one or multiple sources which suggests a new potentially causal association, or a new aspect of a known association, between an intervention and an event or set of related events."

Traditional signal detection relies on disproportionality analysis—statistical methods that identify drug-event combinations reported more frequently than expected based on the background reporting rate. The four primary methods employed globally include:

• Proportional Reporting Ratio (PRR) — Used by UK MHRA and many industry groups. Compares the proportion of reports for a drug-event pair to the proportion for all other drugs.
• Reporting Odds Ratio (ROR) — Used by Netherlands Pharmacovigilance Centre. Similar to PRR but uses odds ratio calculation.
• Multi-item Gamma Poisson Shrinker (MGPS) — Used by FDA's FAERS analysis. Bayesian approach that adjusts for multiple comparisons.
• Bayesian Confidence Propagation Neural Network (BCPNN) — Used by WHO VigiBase. Neural network-inspired Bayesian method.

Each method has demonstrated value in identifying safety signals. However, each also operates within significant constraints: analysis of a single data source, inability to account for cross-database reporting patterns, and high false positive rates due to confounding by indication, media attention, and reporting stimulation.

The fundamental limitation is information fragmentation. FAERS captures North American spontaneous reports. EudraVigilance captures European reports. VigiBase captures global WHO network reports. Real-world evidence sources capture clinical practice patterns. No existing approach synthesizes these sources simultaneously to create a comprehensive view of emerging safety signals.

Classical disproportionality methods excel at identifying statistical anomalies in reporting patterns. However, they struggle to distinguish true signals from noise generated by confounding factors, media attention, and reporting stimulation. Deep learning enhances classical methods by learning patterns that separate signal from noise.

ArcaScience's hybrid approach uses a three-stage architecture:

1. Classical Signal Generation — PRR, ROR, MGPS, and BCPNN run independently on each data source, generating initial signal candidates with statistical thresholds.
2. Feature Extraction — For each signal candidate, extract 127 features including temporal patterns, geographic distribution, reporter characteristics, MedDRA hierarchy relationships, drug class effects, and RWE incidence rates.
3. Neural Network Classification — A gradient-boosted decision tree ensemble trained on 247 validated drug-event pairs predicts signal validity probability and priority score.

The neural network architecture incorporates domain knowledge through feature engineering rather than end-to-end learning. This approach maintains interpretability—critical for regulatory validation—while achieving the pattern recognition capabilities of deep learning.

Key features that distinguish true signals from false positives include:

• Consistency across multiple data sources (signals present in FAERS, EudraVigilance, and VigiBase simultaneously)
• Temporal stability (signals that persist rather than spike and disappear)
• Biological plausibility based on known mechanism of action and pharmacology
• Absence of strong confounding by indication (event not expected from underlying disease)
• RWE corroboration (elevated incidence in claims/EHR data compared to unexposed population)

Validation against known signals demonstrates 98.3% sensitivity (correctly identifying 243 of 247 validated drug-event pairs) with 40% reduction in false positive rate compared to classical methods alone. This performance enables proactive signal detection at scale without overwhelming safety teams with false alerts.