Rethinking Uncertainty in Machine Learning
Hi! PARIS Summer School 2025
Speaker Insight – Aymeric Dieuleveut, École polytechnique
As machine learning systems become embedded in critical decisions, from finance to infrastructure, the need for trustworthy, interpretable predictions has never been greater. Aymeric Dieuleveut, Professor of Statistics and Machine Learning at École polytechnique and scientific co-director of the Hi! PARIS Center, believes the key lies not in the models themselves, but in how we communicate their uncertainty.
At this year’s Hi! PARIS Summer School, Dieuleveut introduced participants to conformal prediction a statistical framework designed to make machine learning outputs more transparent, reliable, and ready for real-world deployment.
Key Takeaways
- Conformal prediction provides a flexible way to quantify uncertainty around machine learning predictions, offering guarantees that are easy to interpret.
- Rather than replacing existing models, conformal methods build on top of any trained predictor, including black-box models, probabilistic forecasts, or quantile regressors.
- Several trade-offs structure the design of conformal methods, especially between computational efficiency and statistical robustness.
- This approach has already been deployed in real-world applications, such as energy price forecasting at EDF.
- Conformal prediction is part of a broader ecosystem of methods, alongside privacy, decentralization, and robustness, that aim to build public trust in AI systems.
Aymeric Dieuleveut at the Hi! PARIS Summer School 2025
Moving beyond the single prediction
At its core, conformal prediction challenges a basic assumption in machine learning: that a model should produce a single best guess. Instead, it offers prediction sets, ranges or intervals, with statistical guarantees that the true value lies within.
For Dieuleveut, this marks a shift not only in method, but in mindset. “When we make predictions with black-box models, we often don’t know how reliable the outputs are,” he explained. “Conformal prediction helps us go beyond that, to actually measure the uncertainty in a principled way.”
Exploring the trade-offs
During his tutorial, Dieuleveut walked participants through the two key trade-offs involved in designing conformal prediction methods. The first is between computational cost and statistical efficiency. Some variants, such as split conformal prediction, are simple and fast. Others offer stronger guarantees but require more intensive computation.
The second trade-off concerns the strength of the guarantee. Most conformal methods ensure what’s known as marginal validity, meaning the coverage guarantee holds on average. But newer methods are moving toward conditional validity, where the coverage depends on specific conditions or inputs.
“This is a subtle but important evolution,” Dieuleveut noted. “It brings us closer to more personalized, context-aware uncertainty estimates.”
Aymeric Dieuleveut at the Hi! PARIS Summer School 2025
From energy markets to model deployment
Conformal prediction isn’t just a theoretical construct, it’s already in use. One example Dieuleveut highlighted comes from the PhD work of Margaux Zaffran, conducted with EDF. By applying conformal methods to electricity price forecasts, her work helped quantify uncertainty in a domain where stakes are high and volatility is common.
As Dieuleveut emphasized, this is one of the most compelling strengths of conformal prediction: it’s model-agnostic and ready to plug into existing systems. “People don’t want to retrain their entire model pipeline just to estimate uncertainty,” he said. “Conformal prediction allows them to add this layer on top.”
Part of a broader trust ecosystem
In a broader sense, conformal prediction is one piece of a larger puzzle. Alongside techniques focused on privacy, robustness, and decentralization, it contributes to building trust in AI systems.
Each of these methods tackles a different dimension, privacy protects data, robustness handles adversaries, decentralization enables learning across networks, but all share a common goal: making machine learning models more reliable and aligned with real-world constraints.
Dieuleveut also noted that, methodologically, these areas are deeply connected. Many draw from shared optimization principles and can be applied using overlapping toolkits.
Compatible, not competitive
One misconception Dieuleveut addressed during his session is the idea that conformal prediction is at odds with Bayesian or probabilistic approaches. In fact, the opposite is true. Conformal methods are often complementary, enhancing existing models rather than replacing them.
“You can apply conformal prediction to virtually any trained model,” he explained. “That’s why it’s so powerful, it doesn’t throw away years of progress in other domains. It builds on them.”
In a landscape where model reuse is critical and deployment pipelines are complex, that kind of adaptability isn’t just convenient, it’s essential.