Lesson 51
Natural-Language Processing and Transformers for Economics
Big question
How can text be converted into numerical representations while preserving enough context to classify or retrieve meaning?
Lesson progress
Complete checkpoints as you learn
Learning objectives
- Tokenize and vectorize text.
- Use TF-IDF and word embeddings.
- Explain cosine similarity, attention, and transformers.
- Evaluate text classifiers and language limitations.
- Prerequisites: Chapters 28 and 30.
- Key terms: tokenization, TF-IDF, embedding, cosine similarity, attention, transformer.
Simple explanation
Text must be divided into tokens before a model can process it. Tokens may be words, subwords, characters, or byte-level units. The choice affects vocabulary size, rare words, multilingual coverage, and sequence length. Cleaning decisions such as lowercasing or punctuation removal can discard useful information. A reproducible text pipeline stores both preprocessing and model parameters.
Key terms
- Tokenize and vectorize text
- A core idea in Chapter 51 that students apply carefully in economic analysis.
- TF-IDF and word embeddings
- A core idea in Chapter 51 that students apply carefully in economic analysis.
- cosine similarity, attention, and transformers
- A core idea in Chapter 51 that students apply carefully in economic analysis.
- Evaluate text classifiers and language limitations
- A core idea in Chapter 51 that students apply carefully in economic analysis.
- Prerequisites: Chapters 28 and 30
- A core idea in Chapter 51 that students apply carefully in economic analysis.
- Key terms: tokenization, TF-IDF, embedding, cosine similarity, attention, transformer
- A core idea in Chapter 51 that students apply carefully in economic analysis.
Analytical workflow
Interpret the expression in words and units before using it in a claim.
Example
Interpretation. The two inflation documents share weighted vocabulary, while the unrelated sports sentence has zero overlap in this tiny corpus.
Prerequisites
- Complete the preceding course chapters or review their summaries as needed.
Full theory and examples
51.2
Tokenization chooses the model’s alphabet
Text must be divided into tokens before a model can process it. Tokens may be words, subwords, characters, or byte-level units. The choice affects vocabulary size, rare words, multilingual coverage, and sequence length. Cleaning decisions such as lowercasing or punctuation removal can discard useful information. A reproducible text pipeline stores both preprocessing and model parameters.
51.3
Representations move from counts to geometry
Bag-of-words and TF-IDF represent documents by weighted token counts. They are strong, interpretable baselines for many classification and retrieval tasks. Embeddings place tokens or documents in a continuous vector space where geometric proximity can encode usage similarity. Cosine similarity compares direction rather than magnitude, but similarity does not guarantee factual equivalence or social neutrality.
51.4
Attention connects tokens conditionally
Attention computes how strongly each token should use information from other tokens. Queries, keys, and values produce weighted combinations, and multi-head attention learns several relation patterns. Transformers combine attention with feed-forward layers, residual connections, and normalization (Vaswani et al. 2017). They can model long dependencies more directly than recurrent networks, but computation grows with sequence length and outputs inherit training-data limitations.
51.5
Core equations
Cosine similarity
The measure compares vector direction and ranges from -1 to 1 for nonzero real vectors.
Scaled attention
Token values are combined using similarity-based weights.
51.6
Python demonstrations
51.6.1
Demonstration 32.1: TF-IDF document similarity
Verified output
Interpretation. The two inflation documents share weighted vocabulary, while the unrelated sports sentence has zero overlap in this tiny corpus.
51.6.2
Demonstration 32.2: A text-classification baseline
Verified output
Interpretation. The example demonstrates workflow, not reliable disaster detection. Four training texts are far too few for deployment.
51.7
Visual evidence
51.8
Reference table
Why This Matters
NLP representations shape what textual evidence a model can use and what distinctions it can miss.
Common Mistake
Treating attention weights as a definitive causal explanation of a model’s decision.
Ceteris LAB Tip
Establish a TF-IDF baseline before adding a transformer. It reveals whether complexity produces real improvement.
R-to-Python / Source Bridge
The chapter adapts IntroAI session 9 on tokenization, embeddings, cosine similarity, attention, transformers, and disaster-tweet classification, with explicit baseline and evaluation guidance (Sharifi-Zarchi and contributors 2026).
| Representation | Strength | Limitation |
|---|---|---|
| bag of words | simple and interpretable | ignores order |
| TF-IDF | strong sparse baseline | limited semantics |
| static embedding | compact semantic geometry | one vector per word |
| contextual embedding | context-sensitive | computational and opaque |
| attention map | relation visualization | not a complete explanation |
Visual evidence
Additional Python demonstrations
Live Python
Source demonstration 2
Source demonstration 2
Stdout
Run Python to see results here.
Status / stderr
Ready to run Python in your browser.
Line-by-line guide
- Line 1Load a Python library needed for data work or regression.
- Line 2Load a Python library needed for data work or regression.
- Line 3Load a Python library needed for data work or regression.
- Line 5Create or update a Python object used in the analysis.
- Line 6Create or update a Python object used in the analysis.
- Line 7Create or update a Python object used in the analysis.
- Line 8Display a result so students can inspect the output.
Verified source output
[1.0, 0.25, 0.0]
[0]
Interpretation. The two inflation documents share weighted vocabulary, while the unrelated sports sentence has zero overlap in this tiny corpus.
Interpretation. The example demonstrates workflow, not reliable disaster detection. Four training texts are far too few for deployment.
Guided practice
- 1Re-run Demonstration 32.1 and change one input while keeping the analytical question fixed.
- 2Explain in two sentences how the output supports, or fails to support, the chapter opening question.
- 3Add one validation check that would prevent a plausible error.
Exercises
- 1Compare word and character TF-IDF.
- 2Calculate cosine similarities for five documents.
- 3Build a train-test text classifier.
- 4Explain how subword tokenization helps rare words.
Source and downloads
Chapter 51 of Fundamentals of Python for Financial Econometrics by Mohammad Safavi, Ph.D.. The lesson is an original Ceteris Lab web adaptation of the supplied publication package.
Live Python
Natural-Language Processing and Transformers for Economics: live Python
Natural-Language Processing and Transformers for Economics: live Python
Stdout
Run Python to see results here.
Status / stderr
Ready to run Python in your browser.
Line-by-line guide
- Line 1Load a Python library needed for data work or regression.
- Line 2Load a Python library needed for data work or regression.
- Line 4Create or update a Python object used in the analysis.
- Line 5Create or update a Python object used in the analysis.
- Line 6Display a result so students can inspect the output.
Python walkthrough
- 1`from sklearn.feature_extraction.text import TfidfVectorizer`: Loads a package or function used by the analysis.
- 2`from sklearn.metrics.pairwise import cosine_similarity`: Loads a package or function used by the analysis.
- 3`docs = ["inflation rose after energy prices increased", "energy costs pushed inflation highe`: Creates or updates a named object used by later steps.
- 4`X = TfidfVectorizer().fit_transform(docs)`: Fits the specified statistical or machine-learning model.
- 5`print(cosine_similarity(X[0], X).round(3).tolist()[0])`: Displays a result so it can be checked and interpreted.
Live notebook
Run this lesson as a notebook
Open an editable notebook cell-by-cell, run Python in the browser, and download the `.ipynb` file for later.
Related dataset
Ceteris Lab teaching sample
Estimated time
25 to 40 min
Packages
pandas, numpy
Expected output
Printed Python results that can be compared with the lesson explanation.
Learning goals
- Load and inspect Ceteris Lab teaching sample.
- Run the Python cells connected to Natural-Language Processing and Transformers for Economics.
- Interpret the output using Tokenize and vectorize text and TF-IDF and word embeddings.
Common errors
- File not found: check that wage_sample.csv is installed or use the course data folder.
- Package import error: use the browser notebook first, then download for local Jupyter if your local packages differ.
- Column name error: compare your variable names with the dataset variables listed for this notebook.
Dataset path helper
import pandas as pd
df = pd.read_csv("/data/wage_sample.csv")
df.head()Interactive activity
Chapter 51 interactive
Model governance check
What is the safest basis for evaluating an ML or AI result?
Immediate feedback
Choose a decision, then test how the claim changes as evidence becomes stronger or weaker.
Try it yourself
Write one plain-English sentence explaining the main idea from this lesson.
Common mistakes
Check these before you move on.
Return to the lesson assumptions, units, diagnostics, and source evidence to replace this shortcut with a defensible interpretation.
Quick quiz
Which statement best answers the Chapter 51 opening question: How can text be converted into numerical representations while preserving enough context to classify or retrieve meaning?
Quick quiz
Which practice should be avoided when applying Natural-Language Processing and Transformers for Economics?
Quick quiz
What is the most defensible way to interpret the Python demonstration?
Quick quiz
Why does Chapter 51 matter in an applied econometrics workflow?
Key takeaway
Text must be tokenized and represented numerically. Sparse TF-IDF is a strong baseline; embeddings add geometric representation. Transformers use attention to construct contextual features.