Chapter 1: Introduction to the course

This chapter introduces the people responsible for the course, aims to answer all open question and should give an impression of the expected workload.

This chapter introduces different learning paradigms, such as embeddings, prompting and pre-training and finetuning a model. These are all very important concepts in the context of Deep Learning for NLP. The purpose of this chapter is to give an overview over these concepts.

Here you will learn about the distinctions between low-level and high-level tasks, as well as the differences between linguistic tasks and broader classification tasks.

In this chapter, take your first steps into the world of Deep Neural Networks for language modeling with the Neural Probabilistic Language Model (NPLM) [1]. Explore how NPLM represents a foundational approach in utilizing deep learning techniques for understanding and generating natural language. Uncover the principles behind NPLM and its significance in paving the way for more sophisticated language models.

Embeddings allow us to represent tokens with a vector representation, enabling computers to efficiently understand the meaning and context of different words. In this chapter we will first explain the general concept of embeddings and then introduce two popular approaches, namely Word2Vec [1] and FastText [2]. Word2vec is a neural network model that learns distributed representations of words in a continuous vector space based on their contextual usage in a corpus of text. Word2vec achieves this by training on a large dataset to predict the surrounding words given a target word, embedding each word as a dense vector where similar words are closer together in the vector space. FastText is an extension of Word2vec that represents words as bags of character n-grams, enabling it to capture morphological information alongside semantics. It accomplishes this by breaking down each word into character n-grams, creating embeddings for these subword units, and then summing or averaging these embeddings to obtain the representation for the whole word.