Posts

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks(2019)

Sentence-BERT derives semantically meaningful sentence embedding that can be compared using cosine-similarity. BERT achieved new state-of-the art performance on various sentence-pair regression tasks using a cross-encoder. A cross-encoder accepts two sentences as input to the transformer network and the target value is predicted. Semantic textual similarity is one of the sentence-pair regression tasks. However, this setup is often not scalable for various pair regression tasks due to many possible combinations. The semantic search that maps each sentence to a vector space where semantically similar sentences are close alleviates the combinatorial explosion. Sentence-BERT uses a siamese network in which the two BERT networks have tied weights such that the produced sentence embeddings can be semantically compared using cosine-similarity.

September 23, 2023

A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning(2008)

A Unified Architecture for Natural Language Processing is an instance of multitask learning. The first layer is a lookup table that stores embeddings of a fixed dictionary and size. The second layer is a Time-Delay Neural Networks layer. It extracts features from the sentence treating it as a sequence with local structure. The third layer takes the maximum value for each of the output features of the second layer over time. The following layers are classical NN layers. The lookup-table is shared among the tasks, and the other layers can be task specific to each task.

#Multitask learning

September 16, 2023

Transmission of Information(1927)

In Transmission of Information, Hartley developed a quantitative measure of “information” in 1927. Hartley claims that information is the outcome of a selection among a finite set of possible messages. Shannon’s “A mathematical theory of communication”, which is based in part on Hartley’s ideas, published in 1947. Hartley did not model the source of information probabilistically. Shannon modeled the source of information as a random process. Hartley stated that information should be proportional to the number of selections for practical engineering value.

#Entropy

September 9, 2023

RoBERTa: A Robustly Optimized BERT Pretraining Approach(2019)

RoBERTa(Robusty optimized BERT approach) is an improved recipe for training BERT models. BERT uses two objectives, masked language modeling and next sequence prediction (NSP), during pretraining. RoBERTa uses masked language modeling only. The authors increased the batch size and Byte-Pair Encoding (BPE) vocabulary size. RobERTa is trained with byte-level BPE, which uses bytes instead of unicode characters as the base subword units.

August 27, 2023

Go To Statement Considered Harmful(1968)

Edgar Djkstra criticized the excessive use of the go to statement in Go To Statement Considered Harmful. While the source code is static, the process taking place under the control of the source code is dynamic. The programmers should aim to shorten the conceptual gap between the source code and its process to describe the progress.

#Early Programming

August 26, 2023

Self-Attention with Relative Position Representations(2018)

The authors of Self-Attention with Relative Position Representations presented a way of injecting relative position representations in the self-attention mechanism of the Transformer. In contrast to recurrent and convolutional neural networks, Transformer does not explicitly model position information in its structure. The original position encoding employs sine and cosine functions of different frequencies. The authors of Transformer hypothesized that sinusoidal position encodings would help Transformer to generalize to sequence lengths unseen during training. Positional encodings are added to the input embeddings at the bottoms of the encoder and decoder stacks of Transformer. This hypothesis was shared by the relative position representations. In contrast to absolute position representations, the relative position representations are invariant to the total sequence length.

August 19, 2023

Self-Adjusting Binary Search Trees(1985)

The splay tree, a self-adjusting form of a binary search tree, is a binary search tree that moves an accessed node to the root after each access. On an \(n\)-node splay tree, accessing, inserting and deleting have an amortized time bound of \(O(\log n)\) per operation. In addition, for sufficiently long access sequences, splay trees are as efficient, to within a constant factor, as static optimum search trees.

#Splay tree

August 12, 2023

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer(2020)

The authors of Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer performed experiments with Text-to-Text Transfer Transformer(T5), a unified framework for NLP. The basic idea underlying T5 is to treat various NLP problems as taking text as input and producing new text as output. Their goal is to explore general language learning abilities instead of providing new methods. They are interested in exploring the limits of transfer learning for NLP by scaling up models and data sets beyond what has previously been considered.

#Transformer

August 5, 2023

Catalan Numbers(2016)

The Catalan numbers are a sequence of natural numbers that occur in various problems in combinatorial mathematics. For example, the number of expressions containing \(n\) valid pairs of parentheses is the \(n\)th Catalan number. Suppose you have a grid of \(n \times n\) squares, the \(n\)th Catalan number represents the number of paths with a length of \(2n\) that lead from the upper left corner to the lower right corner without intersecting the diagonal dotted line running from the upper left to the lower right.

#Catalan numbers

July 29, 2023