Attention Is All You Need

6 Steps

1. 1

   Understand Self-Attention's Role: Recognize that self-attention allows a model to weigh the importance of different words in an input sequence when processing each word, establishing relationships within the sequence itself.

2. 2

   Define Query, Key, Value: Conceptualize Query (Q), Key (K), and Value (V) vectors. Q asks 'what am I looking for?', K answers 'what do I have?', and V provides 'what information do I give if matched?'.

3. 3

   Calculate Raw Attention Scores: Compute the dot product between the Query and Key matrices (Q * K^T). This yields a matrix where each entry indicates the 'compatibility' or 'relevance' between a query item and a key item.

4. 4

   Scale and Normalize Scores: Divide the raw scores by the square root of the key's dimension (dk) to stabilize gradients, then apply a softmax function row-wise to obtain attention weights, ensuring they sum to 1.

5. 5

   Compute Weighted Sum: Multiply the attention weights matrix by the Value matrix. Each row in the output represents a weighted sum of the Value vectors, with weights determined by the attention scores.

6. 6

   Implement Scaled Dot-Product Attention: Write a Python function using NumPy to perform the scaled dot-product attention mechanism from scratch.