Corentin Dancette

Hi! I am a french computer science student, I gratuated from Ecole Centrale Paris, France, and Georgia Institute of Technology, USA.

I am currently a PhD student at the MLIA team, at the LIP6 lab, in Sorbone Université, Paris. I am mainly working on visual reasoning tasks, such as Visual Question Answering. My research is funded by the VISADEEP ANR chair

I was previously intern at the Cognitive Machine Learning team at Ecole Normale Supérieure, in Paris, as a machine learning research intern.

I’m very interested in machine learning and deep learning, more specifically for robotics, computer vision, and speech processing, and its application for AI research.

Education

PhD Student, Sorbonne Université, Paris I am currently a PhD candidate at Sorbonne Université. My PhD subject is “Deep Learning for Visual Reasoning”.

Master degree, Computer Science, Georgia Institute of Technology I graduated from Georgia Tech, with a specialization in machine learning and interactive intelligence.

Engineering student, Ecole Centrale Paris, France I studied Engineering and Computer Science at Ecole Centrale Paris.

Internships and projects

Research Internship, Meta AI / FAIR, Summer 2022
Research intern at Meta AI, San Francisco, under the supervision of Marcus Rorhbach. I worked on reliability of deep learning models in Visual Question Answering.

Research Internship, CoML, Spring 2018
I interned for 6 months at the CoML team, a research lab in École Normale Supérieure in Paris. I worked on unsupervised machine learning for words and phoneme discovery in speech data.

I contributed to the package ABNet3, a siamese neural network for speech embedding. The package is available on github: https://github.com/bootphon/abnet3

Study of the Variational Auto Encoder for speech subword modeling
For a class project, at Georgia Tech, I studied a variational Auto Encoder architecture for the Zerospeech Challenge, Track 1: unsupervised subword modeling. You can find my work here: https://cdancette.fr/zerospeech-vae/

Deep learning for optical flow estimation, Fall 2017
As a graduate student at Georgia Tech, I worked on a project to estimate optical flows in a natural environment dataset. I used the FlowNet2 architecture. You can find my code on github

Software engineering internship, Datadog, Fall 2016
I worked in the data engineering team, managing data pipelines with Spark and Hadoop.

Publications

2022

Fishr: Invariant gradient variances for out-of-distribution generalization Alexandre Rame, Corentin Dancette, and Matthieu Cord ICML 2022 [Abs] [BibTex] [PDF] [Code]
Learning robust models that generalize well under changes in the data distribution is critical for real-world applications. To this end, there has been a growing surge of interest to learn simultaneously from multiple training domains – while enforcing different types of invariance across those domains. Yet, all existing approaches fail to show systematic benefits under controlled evaluation protocols. In this paper, we introduce a new regularization – named Fishr – that enforces domain invariance in the space of the gradients of the loss: specifically, the domain-level variances of gradients are matched across training domains. Our approach is based on the close relations between the gradient covariance, the Fisher Information and the Hessian of the loss: in particular, we show that Fishr eventually aligns the domain-level loss landscapes locally around the final weights. Extensive experiments demonstrate the effectiveness of Fishr for out-of-distribution generalization. Notably, Fishr improves the state of the art on the DomainBed benchmark and performs consistently better than Empirical Risk Minimization.
```
@article{rame2021fishr,
  title = {Fishr: Invariant gradient variances for out-of-distribution generalization},
  author = {Rame, Alexandre and Dancette, Corentin and Cord, Matthieu},
  journal = {ICML},
  year = {2022},
  pdf = {https://arxiv.org/abs/2109.02934}
}
```

2021

Beyond question-based biases: Assessing multimodal shortcut learning in visual question answering Corentin Dancette, Remi Cadene, Damien Teney, and Matthieu Cord ICCV 2021 [Abs] [BibTex] [PDF] [Code]
We introduce an evaluation methodology for visual question answering (VQA) to better diagnose cases of shortcut learning. These cases happen when a model exploits spurious statistical regularities to produce correct answers but does not actually deploy the desired behavior. There is a need to identify possible shortcuts in a dataset and assess their use before deploying a model in the real world. The research community in VQA has focused exclusively on question-based shortcuts, where a model might, for example, answer "What is the color of the sky" with "blue" by relying mostly on the question-conditional training prior and give little weight to visual evidence. We go a step further and consider multimodal shortcuts that involve both questions and images. We first identify potential shortcuts in the popular VQA v2 training set by mining trivial predictive rules such as co-occurrences of words and visual elements. We then introduce VQA-CounterExamples (VQA-CE), an evaluation protocol based on our subset of CounterExamples i.e. image-question-answer triplets where our rules lead to incorrect answers. We use this new evaluation in a large-scale study of existing approaches for VQA. We demonstrate that even state-of-the-art models perform poorly and that existing techniques to reduce biases are largely ineffective in this context. Our findings suggest that past work on question-based biases in VQA has only addressed one facet of a complex issue. The code for our method is available at https://github.com/cdancette/detect-shortcuts
```
@article{dancette2021beyond,
  title = {Beyond question-based biases: Assessing multimodal shortcut learning in visual question answering},
  author = {Dancette, Corentin and Cadene, Remi and Teney, Damien and Cord, Matthieu},
  journal = {ICCV},
  year = {2021},
  pdf = {https://arxiv.org/abs/2104.03149}
}
```
Overcoming Statistical Shortcuts for Open-ended Visual Counting Corentin Dancette, Remi Cadene, Xinlei Chen, and Matthieu Cord Visual Question Answering workshop, CVPR 2021 [Abs] [BibTex] [arXiv] [PDF] [Code]
Machine learning models tend to over-rely on statistical shortcuts. These spurious correlations between parts of the input and the output labels does not hold in real-world settings. We target this issue on the recent open-ended visual counting task which is well suited to study statistical shortcuts. We aim to develop models that learn a proper mechanism of counting regardless of the output label. First, we propose the Modifying Count Distribution (MCD) protocol, which penalizes models that over-rely on statistical shortcuts. It is based on pairs of training and testing sets that do not follow the same count label distribution such as the odd-even sets. Intuitively, models that have learned a proper mechanism of counting on odd numbers should perform well on even numbers. Secondly, we introduce the Spatial Counting Network (SCN), which is dedicated to visual analysis and counting based on natural language questions. Our model selects relevant image regions, scores them with fusion and self-attention mechanisms, and provides a final counting score. We apply our protocol on the recent dataset, TallyQA, and show superior performances compared to state-of-the-art models. We also demonstrate the ability of our model to select the correct instances to count in the image. Code and datasets are available: this https URL
```
@article{dancette2020overcoming,
  title = {Overcoming Statistical Shortcuts for Open-ended Visual Counting},
  author = {Dancette, Corentin and Cadene, Remi and Chen, Xinlei and Cord, Matthieu},
  journal = {Visual Question Answering workshop, CVPR},
  year = {2021},
  arxiv = {2006.10079},
  pdf = {https://arxiv.org/pdf/2006.10079.pdf}
}
```

2020

2019

RUBi: Reducing Unimodal Biases for Visual Question Answering Remi Cadene, Corentin Dancette, Matthieu Cord, Devi Parikh, and others In NeurIPS 2019 [Abs] [BibTex] [arXiv] [PDF] [Code]
Visual Question Answering (VQA) is the task of answering questions about an image. Some VQA models often exploit unimodal biases to provide the correct answer without using the image information. As a result, they suffer from a huge drop in performance when evaluated on data outside their training set distribution. This critical issue makes them unsuitable for real-world settings. We propose RUBi, a new learning strategy to reduce biases in any VQA model. It reduces the importance of the most biased examples, i.e. examples that can be correctly classified without looking at the image. It implicitly forces the VQA model to use the two input modalities instead of relying on statistical regularities between the question and the answer. We leverage a question-only model that captures the language biases by identifying when these unwanted regularities are used. It prevents the base VQA model from learning them by influencing its predictions. This leads to dynamically adjusting the loss in order to compensate for biases. We validate our contributions by surpassing the current state-of-the-art results on VQA-CP v2. This dataset is specifically designed to assess the robustness of VQA models when exposed to different question biases at test time than what was seen during training. Our code is available: http://github.com/cdancette/rubi.bootstrap.pytorch
```
@inproceedings{cadene2019rubi,
  title = {RUBi: Reducing Unimodal Biases for Visual Question Answering},
  author = {Cadene, Remi and Dancette, Corentin and Cord, Matthieu and Parikh, Devi and others},
  booktitle = {NeurIPS},
  pages = {841--852},
  year = {2019},
  arxiv = {1906.10169},
  pdf = {https://arxiv.org/pdf/1906.10169.pdf}
}
```

2018

A K-nearest neighbours approach to unsupervised spoken term discovery Alexis Thual, Corentin Dancette, Julien Karadayi, Juan Benjumea, and Emmanuel Dupoux In 2018 IEEE Spoken Language Technology Workshop (SLT) 2018 [Abs] [BibTex]
Unsupervised spoken term discovery is the task of finding recurrent acoustic patterns in speech without any annotations. Current approaches consists of two steps: (1) discovering similar patterns in speech, and (2) partitioning those pairs of acoustic tokens using graph clustering methods. We propose a new approach for the first step. Previous systems used various approximation algorithms to make the search tractable on large amounts of data. Our approach is based on an optimized k-nearest neighbours (KNN) search coupled with a fixed word embedding algorithm. The results show that the KNN algorithm is robust across languages, consistently out-performs the DTW-based baseline, and is competitive with current state-of-the-art spoken term discovery systems.
```
@inproceedings{thual2018k,
  title = {A K-nearest neighbours approach to unsupervised spoken term discovery},
  author = {Thual, Alexis and Dancette, Corentin and Karadayi, Julien and Benjumea, Juan and Dupoux, Emmanuel},
  booktitle = {2018 IEEE Spoken Language Technology Workshop (SLT)},
  pages = {491--497},
  year = {2018},
  organization = {IEEE}
}
```
Sampling Strategies in Siamese Networks for Unsupervised Speech Representation Learning Rachid Riad, Corentin Dancette, Julien Karadayi, Neil Zeghidour, Thomas Schatz, and Emmanuel Dupoux In Proc. Interspeech 2018 [Abs] [BibTex]
Recent studies have investigated siamese network architectures for learning invariant speech representations using same-different side information at the word level. Here we investigate systematically an often ignored component of siamese networks: the sampling procedure (how pairs of same vs. different tokens are selected). We show that sampling strategies taking into account Zipf’s Law, the distribution of speakers and the proportions of same and different pairs of words significantly impact the performance of the network. In particular, we show that word frequency compression improves learning across a large range of variations in the number of training pairs. This effect does not apply to the same extent to the fully unsupervised setting, where the pairs of same-different words are obtained by spoken term discovery. We apply these results to pairs of words discovered using an unsupervised algorithm and show an improvement on the state-of-the-art in unsupervised representation learning using siamese networks.
```
@inproceedings{Riad2018sampling,
  author = {Riad, Rachid and Dancette, Corentin and Karadayi, Julien and Zeghidour, Neil and Schatz, Thomas and Dupoux, Emmanuel},
  title = {Sampling Strategies in Siamese Networks for Unsupervised Speech Representation Learning},
  year = {2018},
  booktitle = {Proc. Interspeech},
  pages = {2658--2662},
  doi = {10.21437/Interspeech.2018-2384},
  url = {http://dx.doi.org/10.21437/Interspeech.2018-2384}
}
```

Talks

Apr 2022	Detecting and reducing data biases Sicara, Paris
Apr 2022	Beyond Question-Based Biases: Assessing Multimodal Shortcut Learning in Visual Question Answering ICCV 2021
Nov 2020	Detecting and reducing data biases Valeo.ai, Paris
Feb 2020	RUBi : Reducing Unimodal Biases in Visual Question Answering ENS Ulm, LSCP, CoML Team, Paris
Dec 2019	RUBi : Reducing Unimodal Biases in Visual Question Answering NeurIPS 2019