I am an Associate Professor (with tenure) in the Department of Informatics at the University of Oslo (UiO), a Principal Investigator at the Norwegian Center for Knowledge-driven Machine Learning (Integreat), and the SFI Visual Intelligence, and a member of the European Laboratory for Learning and Intelligent Systems (ELLIS) Society. I serve as an Area Chair of NeurIPS and AISTATS and Action Editor for Transactions on Machine Learning Research.

Before joining UiO, I was a Senior Scientific Collaborator at EPFL, working with Prof. Volkan Cevher in Laboratory for Information and Inference Systems (LIONS). Before joining LIONS, I was an NSERC Postdoctoral Fellow at the Vector Institute in Toronto working with Prof. Daniel M. Roy. I received my Ph.D. from the University of Toronto where I was very fortunate to be advised by Prof. Ben Liang and Prof. Min Dong.

My current research is focused on understanding how the input data distribution is encoded within layers of popular neural networks and developing theoretical concepts and practical tools to minimize the statistical risk under resource constraints and realistic settings referring to statistical and system characteristics contrary to an ideal learning setting. I am interested in a broad range of applications including but not limited to emotion recognition, marine data, and neuroscience.

Recent News

• 11/2024: I will serve as an Action Editor for Transactions on Machine Learning Research.
• 10/2024: I will serve as an Area Chair of AISTATS 2025.
• 8/2024: Integreat is officially launched.
• 4/2024: I have PhD Position in ML.
• 3/2024: I will serve as an Area Chair of NeurIPS 2024.
• 2/2024: We have one postdoc position in “Joint Physics-informed and Data-driven Complex Dynamical System Solvers”.
• 2/2024: Our paper Mixed Nash for Robust Federated Learning has been accepted to the Transactions on Machine Learning Research.
• 2/2024: The Research Excellence Center SFI Visual Intelligence has recognized our work Federated Learning under Covariate Shifts with Generalization Guarantees as Spotlight Publications from 2023!
• 1/2024: Our paper On the Generalization of Stochastic Gradient Descent with Momentum has been accepted to the Journal of Machine Learning Research.
• 1/2024: We will have Multiple Exciting Ph.D. Positions through Integreat joint with wonderful Computer Scientists, Statisticians, and Mathematicians across Norway!
• 1/2024: I am the main organizer of Integreat Discrepancies Seminar Series.
• 1/2024: I am a Program Chair at the Northern Lights Deep Learning Conference (NLDL 2024).
• 12/2024: Our new PhD research fellow Amir Arfan has started the PhD through SUURPh programme. Welcome Amir!
• 10/2023: I gave a talk titled Learning under Resource Constraints in Real World at the University of Toronto.
• 9/2023: I gave a talk titled Learning under Resource Constraints in Real World at the University of Copenhagen.
• 9/2023: I gave a talk titled Learning under Resource Constraints in Real World at Integreat.
• 6/2023: Our paper Federated Learning under Covariate Shifts with Generalization Guarantees has been published in Transactions on Machine Learning Research.
• 4/2023: I am now a PI at the Visual Intelligence!
• 3/2023: I gave a talk titled Scalable and Robust Deep Learning at the Visual Intelligence.
• 1/2023: We have an open Ph.D. position. Deadline: 28 Feb, 2023.
• 1/2023: Our paper Distributed Extra-gradient with Optimal Complexity and Communication Guarantees has been accepted to ICLR 2023.
• 1/2023: I am now a member of the ELLIS Society!
• 1/2023: I joined the Department of Informatics at the University of Oslo!
• 12/2022: I gave a talk titled Randomization Improves Deep Learning Security at the Annual Workshop of the VILLUM Investigator Grant at Aalborg University.
• 10/2022: Our paper MixTailor: Mixed Gradient Aggregation for Robust Learning Against Tailored Attacks has been published in Transactions on Machine Learning Research.
• 8/2022: I gave a talk titled How Did DL Dominate Today’s ML? What Challenges and Limitations Remain? at the University of Oslo.
• 6/2022: I gave a talk titled Scalable ML: Communication-efficiency, Security, and Architecture Design at the University of Edinburgh.
• 2/2022: I gave a talk titled Scalable ML: Communication-efficiency, Security, and Architecture Design at the University of Liverpool.
• 09/2021: Our paper Subquadratic Overparameterization for Shallow Neural Networks has been accepted to NeurIPS 2021.

Selected Publications

We have shown it is crucial to establish an appropriate balance between the optimization error associated with the empirical risk and the generalization error when accelerating SGD with momentum and established generalization error bounds and explicit convergence rates for SGD with momentum under a broad range of hyperparameters including a general step-size rule. For smooth Lipschitz loss functions, we analyze SGD with early momentum (SGDEM) under a broad range of step-sizes, and show that it can train machine learning models for multiple epochs with a guarantee for generalization.

Ali Ramezani-Kebrya, Kimon Antonakopoulos, Volkan Cevher, Ashish Khisti, and Ben Liang, On the Generalization of Stochastic Gradient Descent with Momentum, Journal of Machine Learning Research, vol. 25, pp. 1-56, Jan. 2024.
pdf bib arXiv

Even for a single client, the distribution shift between training and test data, i.e., intra-client distribution shift,has been a major challenge for decades. For instance, scarce disease data for training and test in a local hospital can be different. We focus on the overall generalization performance on multiple clients and modify the classical ERM to obtain an unbiased estimate of an overall true risk minimizer under intra-client and inter-client covariate shifts, develop an efficient density ratio estimation method under stringent privacy requirements of federated learning, and show importance-weighted ERM achieves smaller generalization error than classical ERM.

Ali Ramezani-Kebrya*, Fanghui Liu*, Thomas Pethick*, Grigorios Chrysos, and Volkan Cevher, Federated Learning under Covariate Shifts with Generalization Guarantees, Transactions on Machine Learning Research, June 2023.
pdf code openreview

Beyond supervised learning, we accelerate large-scale monotone variational inequality problems with applications such as training GANs in distributed settings. We propose quantized generalized extra-gradient (Q-GenX) family of algorithms with the optimal rate of convergence and achieve noticeable speedups when training GANs on multiple GPUs without performance degradation.

Ali Ramezani-Kebrya*, Kimon Antonakopoulos*, Igor Krawczuk*, Justin Deschenaux*, and Volkan Cevher, Distributed Extra-gradient with Optimal Complexity and Communication Guarantees, ICLR 2023.
pdf bib code openreview

ML models are vulnerable to various attacks at training and test time including data/model poisoning and adversarial examples. We introduce MixTailor, a scheme based on randomization of the aggregation strategies that makes it impossible for the attacker to be fully informed. MixTailor: Mixed Gradient Aggregation for Robust Learning Against Tailored Attacks increases computational complexity of designing tailored attacks for an informed adversary.

Ali Ramezani-Kebrya*, Iman Tabrizian*, Fartash Faghri, and Petar Popovski, MixTailor: Mixed Gradient Aggregation for Robust Learning Against Tailored Attacks, Transactions on Machine Learning Research, Oct. 2022.
pdf bib code arXiv openreview

Overparameterization refers to the important phenomenon where the width of a neural network is chosen such that learning algorithms can provably attain zero loss in nonconvex training. In Subquadratic Overparameterization for Shallow Neural Networks, we achieve the best known bounds on the number of parameters that is sufficient for gradient descent to converge to a global minimum with linear rate and probability approaching to one.

Chaehwan Song*, Ali Ramezani-Kebrya*, Thomas Pethick, Armin Eftekhari, and Volkan Cevher, Subquadratic Overparameterization for Shallow Neural Networks, NeurIPS 2021.
pdf bib code arXiv openreview

In training deep models over multiple GPUs, the communication time required to share huge stochastic gradients is the main performance bottleneck. We closed the gap between theory and practice of unbiased gradient compression. NUQSGD is currently the method offering the highest communication-compression while still converging under regular (uncompressed) hyperparameter values.

Ali Ramezani-Kebrya, Fartash Faghri, Ilya Markov, Vitalii Aksenov, Dan Alistarh, and Daniel M. Roy, NUQSGD: Provably Communication-Efficient Data-Parallel SGD via Nonuniform Quantization, Journal of Machine Learning Research, vol. 22, pp. 1-43, Apr. 2021.
pdf bib code arXiv

Communication-efficient variants of SGD are often heuristic and fixed over the course of training. In Adaptive Gradient Quantization for Data-Parallel SGD, we empirically observe that the statistics of gradients of deep models change during the training and introduce two adaptive quantization schemes. We improve the validation accuracy by almost 2% on CIFAR-10 and 1% on ImageNet in challenging low-cost communication setups.

Fartash Faghri*, Iman Tabrizian*, Ilya Markov, Dan Alistarh, Daniel M. Roy, and Ali Ramezani-Kebrya, Adaptive Gradient Quantization for Data-Parallel SGD, NeurIPS 2020.
pdf bib code arXiv

Students

• Sigurd Holmsen, Ph.D. in progress, University of Oslo.
• Daniel Kaiser, Ph.D. in progress, University of Tromsø.
• Chau Thi Thuy Tran, Ph.D. in progress, University of Oslo.
• Amir Arfan, Ph.D. in progress, University of Oslo.

• Zhiyuan Wu, Ph.D. in progress, University of Oslo.

• Frida Marie Engøy Westby, MS in progress, University of Oslo.
• Trude Halvorsen, MS in progress, University of Oslo.

• Kjetil Karstensen Indrehus, MS in progress, University of Oslo.

• Oskar Høgberg Simensen, summer intern, University of Oslo.
• Guri Marie Svenberg, summer intern, University of Oslo.

• Karen Stølan Nielsen, summer intern, University of Oslo.

• Co-supervision at the University of Toronto and EPFL
  • Anh Duc Nguyen, undergraduate intern, EPFL.
  • Thomas Michaelsen Pethick, Ph.D. in progress, EPFL.
  • Wanyun Xie, MSc KTH, first job after graduation: Ph.D. at EPFL.
  • Fartash Faghri, Ph.D. UoT, first job after graduation: research scientist at Apple.
  • Iman Tabrizian, M.A.Sc. UoT, first job after graduation: full-time engineer at NVIDIA.

Major Collaborators

• Dan Alistarh
• Volkan Cevher
• Arnoldo Frigessi
• Carsten Griwodz
• Robert Jenssen
• Ole Christian Lingjærde
• Benjamin Ricaud
• Sylvia Richardson
• Daniel M. Roy
• Kristoffer Wickstrøm