Learning Quantized Neural Networks and Discrete Representations

Description:

The topic is on the intersection of modern machine learning and computer vision. Weights and activations of neural networks can be quantized to be represented with a few bits only. This offers huge savings in terms of computation cost and energy and allows larger models to run in simpler devices. The challenge is to learn such quantized models to achieve high efficiency and accuracy. The research focuses on stochastic relaxation methods. To quantize modern architectures, we need to develop suitable discretizations of  intermediate representations such as queries and keys in the attention model underlying powerful transformer models. Finally, it can be desirable to learn discrete representations on the output of a neural network. For example, for the image retrieval application we want to learn compact binary descriptors, which are efficient to store and fast to compare, such that similar descriptors correspond to semantically similar objects (contrastive learning). 

What are you going to do?

Your tasks will be to:

• Perform novel research towards more efficient learning of discrete representations and their application;
• Present research results at international conferences and journals;
• Actively collaborate within the group and with researchers worldwide;
• Assist in teaching activities such as lab assistance and student supervision;
• Pursue and complete a PhD thesis within the planned duration of four years.

Group:

Visual Recognition Group (VRG – Visual Recognition Group (cvut.cz)), Department of Cybernetics, Czech Technical University in Prague, Czech Republic. CTU ranks highly in the computer vision area. (https://csrankings.org/#/fromyear/2018/toyear/2023/index?vision&europe). The VRG group together with other groups at the department form a large and open international scientific environment, performing research in many topics related to deep learning and AI. There are plenty of events such as reading groups, scientific seminars, invited talks by international speakers, etc. The department has a sufficient computation infrastructure, including many GPU servers. VRG, led by Prof. Jiri Matas (https://scholar.google.com/citations?hl=en&user=EJCNY6QAAAAJ), focuses on basic research and applications of computer vision and machine learning. The main research areas are object recognition and retrieval, representation learning, tracking, text recognition, and minimal problems in computer vision.

Supervisor and Project:

The supervisor for this position is Sasha (Oleksandr / Alexander) Shekhovtsov. 

https://scholar.google.com/citations?hl=en&user=6Ty5Md4AAAAJ

https://cmp.felk.cvut.cz/~shekhovt/

I am broadly interested in statistical methods for machine learning, which includes methods for learning, handling uncertainties, analysis of stochastic and generative models, etc. Current main direction is “Learning Quantized Neural Networks, Discrete Choices and Representations”. Recently, this project proposal was selected for funding by the Czech Science Foundation (GACR). In my research I focus on analysis and development of new general methods, albeit motivated by practical applications, in particular in computer vision. In the project we will keep the focus on the methods, but will need to demonstrate the impact in applications as well. 

Your Experience and Profile:

• A relevant master’s degree from a technical or mathematical school.

The following qualities of the candidate are a plus:

• education in subjects relevant for machine learning: statistics, optimization, data analysis, signal processing, artificial neural networks, deep learning, etc;
• solid skills in programming (python, pytorch, C++ could be helpful) and software (unix shell, VS code, git);
• motivation to explore and dig into problems;
• critical thinking to identify weak points in the experiments and theory;
• ability to use math for analysis and building understanding;
• fluent English;