Lecture 23: Natural Language Processing

Lecture 23: Natural Language Processing¶

AIMA Chapter 23 — 1 hour¶

Learning Objectives¶

• Build language models (n-grams)

• Understand grammar and parsing

• Apply part-of-speech tagging

• Handle semantic interpretation

Language Models¶

• P(w₁,...,wₙ): Probability of word sequence

• N-gram: P(wᵢ|wᵢ₋ₙ₊₁,...,wᵢ₋₁)

• Smoothing: Laplace, backoff

N-Gram Models¶

• Unigram: P(wᵢ)

• Bigram: P(wᵢ|wᵢ₋₁)

• Trigram: P(wᵢ|wᵢ₋₂,wᵢ₋₁)

• Sparsity: Smoothing essential

Word Representations¶

• One-hot: High-dimensional, sparse

• Embeddings: Dense vectors (Word2Vec, GloVe)

• Distributional: Similar words → similar vectors

Part-of-Speech Tagging¶

• Tags: Noun, verb, adjective, ...

• HMM: States = tags, emissions = words

• Viterbi: Most likely tag sequence

Grammar¶

• Context-free grammar: Rules like S → NP VP

• Parsing: Find parse tree

• Ambiguity: Multiple parses

Parsing¶

• CKY: Bottom-up, O(n³)

• Dependency parsing: Head-modifier relations

• Transition-based: Shift-reduce

Semantic Interpretation¶

• Compositional: Meaning from parts

• Lambda calculus: Represent meaning

• Knowledge base: Query from natural language

Summary¶

• N-grams: Language modeling

• POS: HMM, Viterbi

• Parsing: CKY, dependency

• Semantics: Compositional

References¶

• AIMA Ch. 23

• Russell & Norvig, AIMA 4e, Ch. 23

• Chapter PDF: chapters/chapter-23.pdf

• aima-python: nlp.ipynb

Questions?¶

Next lecture: Deep Learning for NLP (Chapter 24)