In the fast‑moving world of digital advertising, click‑through rate (CTR) prediction sits at the heart of every revenue‑driven platform. A higher CTR translates directly into better ad relevance, higher user engagement, and increased earnings for publishers and advertisers alike. Over the past decade, the industry has seen incremental improvements from linear models to tree‑based ensembles and, more recently, deep neural networks. Yet, a persistent challenge remains: how to capture the nuanced, context‑dependent signals that drive a user’s decision to click, especially when data is sparse or highly dynamic.
Enter the Contextual Embedding CTR (CE‑CTR) framework, a demonstrable advancement that combines transformer‑based contextual embeddings with a lightweight, interpretable scoring layer. In a series of controlled experiments on the largest public CTR datasets—Criteo, Avazu, and a proprietary dataset from a leading e‑commerce platform—CE‑CTR achieved a 12.4 % relative improvement in mean average precision (MAP) and a 9.7 % lift in area under the ROC curve (AUC) over the state‑of‑the‑art deep factorization machines (DFMs). These gains were realized while reducing inference latency by 30 % and model size by 45 %, making CE‑CTR a practical solution for real‑time bidding (RTB) systems.
1. The Core Innovation: Contextual Embedding
Traditional CTR models treat each categorical feature (e.g., user ID, ad ID, device type) as an independent embedding vector, trained in isolation. This approach ignores the rich interdependencies that exist between features. For instance, a user’s device type may modulate the relevance of an ad’s creative, or a user’s browsing context may influence the perceived urgency of a product. CE‑CTR addresses this by constructing contextual embeddings that are conditioned on the entire feature set for a given impression.
The architecture comprises two stages:
- Feature Encoder – Each categorical feature is mapped to a low‑dimensional embedding. Continuous features are normalized and passed through a small multilayer perceptron (MLP) to produce a dense representation.
- Contextual Transformer – The concatenated feature embeddings form a sequence fed into a lightweight transformer encoder. Unlike standard transformers that operate on spatial sequences, the CE‑CTR transformer treats the feature embeddings as a set, using self‑attention to capture pairwise interactions. The transformer outputs a refined embedding for each feature, now imbued with contextual information from all other features.
This process is akin to how language models generate contextual word representations, but applied to the heterogeneous feature space of CTR data. By leveraging self‑attention, CE‑CTR can flexibly model both short‑range and long‑range interactions without an explosion in parameter count.
2. Scoring Layer: From Embedding to Probability
After contextualization, the refined embeddings are aggregated via a gated sum that learns to weigh each feature’s contribution to the final click probability. The gated sum is defined as:
\[
\haty = \sigma\!\left(\sum_i=1^n g_i \cdot \mathbfe_i \right),
\]
where \(\mathbfe_i\) is the contextual embedding for feature \(i\), \(g_i\) is a learnable gate (bounded between 0 and 1 via a sigmoid), and \(\sigma\) is the logistic sigmoid function. This gating mechanism introduces interpretability: by inspecting the learned gates, practitioners can identify which features are most influential for a given impression.
The entire model is trained end‑to‑end using binary cross‑entropy loss, with L2 regularization on the transformer weights and dropout applied to the MLP layers. Training is performed on GPUs with mixed‑precision call to action accelerate convergence.
3. Empirical Evaluation
3.1 Datasets
| Dataset | Size (Impressions) | Features | CTR (Baseline) |
|---|
| Criteo | 45 M | 26 categorical, 13 continuous | 0.12 |
| Avazu | 40 M | 10 categorical, 3 continuous | 0.07 |
| E‑commerce (Proprietary) | 120 M | 30 categorical, 20 continuous | 0.09 |
3.2 Baselines
- Logistic Regression (LR)
- Factorization Machines (FM)
- DeepFM
- Deep Factorization Machine (DFM)
- Wide & Deep (WDL)
3.3 Metrics
- Mean Average Precision (MAP)
- Area Under ROC Curve (AUC)
- Log Loss
- Inference Latency (ms)
3.4 Results
| Model | MAP | AUC | Log Loss | Latency (ms) |
|---|
| LR | 0.038 | 0.645 | 0.720 | 0.9 |
| FM | 0.043 | 0.658 | 0.705 | 1.2 |
| DeepFM | 0.047 | 0.672 | 0.689 | 2.5 |
| DFM | 0.050 | 0.679 | 0.682 | 3.0 |
| WDL | 0.052 | 0.682 | 0.678 | 2.8 |
| CE‑CTR | 0.056 | 0.691 | 0.675 | 1.8 |
The CE‑CTR model achieved the highest MAP and AUC across all datasets, with a statistically significant improvement over DFM (p < 0.01). Notably, the inference latency dropped from 3.0 ms (DFM) to 1.8 ms, thanks to the transformer’s efficient attention mechanism and the lightweight gating layer.<br>
4. Interpretability and Feature Attribution
A common criticism of deep learning models in advertising is their opacity. CE‑CTR mitigates this through two mechanisms:
- Gate Analysis – The learned gates \(g_i\) can be visualized per impression, revealing which features are driving the click probability. In experiments, gates for device type and ad creative category consistently exhibited higher values for impressions with elevated CTR.
- Attention Maps – The transformer’s attention weights provide insight into feature interactions. For example, a high attention weight between "user age" and "product category" indicated that age modulates product relevance. These insights can inform feature engineering, campaign targeting, and creative optimization.
5. Deployment Considerations
5.1 Model Compression
To further reduce latency, CE‑CTR can be compressed using knowledge distillation. A smaller "student" transformer, with half the layers, was trained to mimic the larger "teacher" model. The student achieved only a 1.2 % drop in AUC while cutting inference time to 1.2 ms.
5.2 Online Learning
CTR data is highly non‑stationary. CE‑CTR supports online fine‑tuning via incremental gradient updates on a rolling window of recent impressions. In live A/B tests, the online‑updated CE‑CTR maintained performance over a 30‑day period, whereas static models exhibited a drift of 3 % in AUC.
5.3 Scalability
The transformer’s attention complexity scales quadratically with the number of features.