Seminar: Efficient and Robust Natural Language Processing
Wed 14:15–15:45, Room -1.05
Dr. Simon Ostermann, Tatiana Anikina, Natalia Skachkova
Deutsches Forschungszentrum für Künstliche Intelligenz (DFKI)
If you would like to participate, please drop an email to efficient-nlp-seminar@dfki.de
until April 6th (23:59).
In your email, please:
- Give your name, semester, study program
- Write some words on why you want to take part in this course
- List some of your previous experience:
- your background in deep learning or machine learning
- your background in natural language processing in general
Prerequisites
This seminar is primarily targeted at Master students, but is also open to advanced Bachelor students. We expect you to have a curious mind and advanced familiarity with large language models. At the very least, we expect all students to have read (and understood :-)) the BERT paper and the Transformer paper.
Seminar Content
Nowadays, Large Language Models (LLMs) achieve impressive results on a variety of NLP tasks and languages. This increase in performance comes with a specific cost: Better performance is typically attributed to an increased number of parameters and training data. This means that better and larger models also need more computational resources, more time, more memory, and more energy, which makes training processes prohibitively expensive.
How can we solve the problem of ever larger and ever hungrier models? Efficient NLP is an umbrella term that includes a lot of different approaches to address these problems in terms of model design and data efficiency, by introducing more efficient fine-tuning, inference, or hardware utilization. In this seminar we will mostly focus on model and data-level efficiency and will explore various approaches including efficient prompting methods, adapters, transfer learning, data augmentation and active learning.
A preliminary list of papers to choose from can be found here.
Some words on grading: This seminar is meant to be as interactive as possible. Final grades will be based on students' presentations, term papers (optional), but also on participation and discussion in class.
The participants are expected to prepare for classes accordingly, by reading the relevant papers and also doing background reading, if necessary. Based on this preparation, the participants should be able to discuss the presented papers in depth and to understand relevant context during the discussion.