Lesson 9

pandas Fundamentals

Big question

How can a rectangular dataset preserve labels, dates, and missing values while supporting fast analysis?

Lesson progress

Complete checkpoints as you learn

0% complete0 checkpoint streak

Progress tracking available after sign-in. Sign in to save your work.

Big question

Concept

Activity

Quiz

Learning objectives

Create Series and DataFrames.
Inspect rows, columns, indexes, and dtypes.
Select, filter, sort, and assign variables.
Use method chains without hiding intermediate meaning.
Prerequisites: Chapter 8.
Key terms: Series, DataFrame, index, column, dtype, method chain.

Simple explanation

pandas adds row and column labels to NumPy-style arrays. A Series represents one labelled vector; a DataFrame represents columns that may have different data types. Labels make code readable, but they also introduce alignment rules. Arithmetic between two Series matches index labels rather than blindly matching position. This is powerful when dates align and dangerous when duplicated or inconsistent keys slip through (pandas Development Team 2026).

Key terms

Create Series and DataFrames: A core idea in Chapter 9 that students apply carefully in economic analysis.
Inspect rows, columns, indexes, and dtypes: A core idea in Chapter 9 that students apply carefully in economic analysis.
Select, filter, sort, and assign variables: A core idea in Chapter 9 that students apply carefully in economic analysis.
method chains without hiding intermediate meaning: A core idea in Chapter 9 that students apply carefully in economic analysis.
Prerequisites: Chapter 8: A core idea in Chapter 9 that students apply carefully in economic analysis.
Key terms: Series, DataFrame, index, column, dtype, method chain: A core idea in Chapter 9 that students apply carefully in economic analysis.

Analytical workflow

Question + data + assumptions + transparent Python -> evidence

Interpret the expression in words and units before using it in a claim.

Example

Interpretation. The structure reveals four observations and three variables with different data types.

Prerequisites

Complete the preceding course chapters or review their summaries as needed.

Full theory and examples

9.2

Labels are part of the data model

9.3

Inspection comes before transformation

A disciplined first look includes shape, column names, sample rows, data types, missingness, and summary statistics. head() is useful but insufficient because the first rows may not reveal late missing values or inconsistent categories. The analyst should inspect both structure and content before selecting a model. A DataFrame is not “clean” merely because it displays neatly.

9.4

Selection should reveal intent

Use column names for variables and Boolean masks for conditions. .loc selects by labels; .iloc selects by positions. Method chains can read as a workflow when each step is short: filter, assign, group, summarize. A long chain that mixes validation, transformation, and modeling is difficult to debug. Temporary named objects are helpful when they expose an important analytical stage.

9.5

Python demonstrations

9.5.1

Demonstration 9.1: Build and inspect a DataFrame

Verified output

Interpretation. The structure reveals four observations and three variables with different data types.

9.5.2

Demonstration 9.2: Filter and assign

Verified output

Interpretation. The chain reads as a sequence of analytical decisions and leaves the original DataFrame unchanged.

9.6

Visual evidence

9.7

Reference table

Why This Matters

pandas makes data structure, labels, and transformation rules visible in code.

Common Mistake

Using chained indexing such as df[df.x > 0]["y"] = .... It can create ambiguous copies and warnings.

Ceteris LAB Tip

Use .loc for label-based selection and assignment, then inspect the resulting shape.

R-to-Python / Source Bridge

The source notes used R matrices and data frames for financial series. pandas supplies labelled, date-aware structures while retaining vectorized computation.

Table 9. Chapter reference.

Operation	Preferred form	Meaning
Select columns	df[[“a”, “b”]]	retain named variables
Filter rows	df.loc[df[“x”] > 0]	retain observations satisfying rule
Position	df.iloc[:5]	first five positions
Assign	df.assign(growth=…)	create variable without hidden mutation
Sort	df.sort_values(“date”)	establish order

Visual evidence

Figure 9. A DataFrame organizes observations by rows and variables by labelled columns.

Additional Python demonstrations

Live Python

Source demonstration 2

Stdout

Run Python to see results here.

Status / stderr

Ready to run Python in your browser.

Line-by-line guide

Line 1Create or update a Python object used in the analysis.
Line 2Create or update a Python object used in the analysis.
Line 3Create or update a Python object used in the analysis.
Line 4Create or update a Python object used in the analysis.
Line 5Run this Python instruction as part of the lesson workflow.
Line 6Display a result so students can inspect the output.

Verified source output

(4, 3) {'province': 'object', 'income': 'int64', 'employed': 'bool'}

province income_thousands 3 ON 71.0 2 BC 65.0 0 ON 62.0

(4, 3)
{'province': 'object', 'income': 'int64', 'employed': 'bool'}

province  income_thousands
3       ON              71.0
2       BC              65.0
0       ON              62.0

import pandas as pd
df = pd.DataFrame({
print(df.shape)
print(df.dtypes.astype(str).to_dict())
high_income = (
.assign(income_thousands=lambda x: x["income"] / 1_000)

Interpretation. The structure reveals four observations and three variables with different data types.

Interpretation. The chain reads as a sequence of analytical decisions and leaves the original DataFrame unchanged.

Guided practice

1Re-run Demonstration 9.1 and change one input while keeping the analytical question fixed.
2Explain in two sentences how the output supports, or fails to support, the chapter opening question.
3Add one validation check that would prevent a plausible error.

Exercises

1Create a DataFrame with five observations.
2Filter it using two conditions.
3Add a standardized variable.
4Compare .loc and .iloc on the same data.

Source and downloads

Chapter 9 of Fundamentals of Python for Financial Econometrics by Mohammad Safavi, Ph.D.. The lesson is an original Ceteris Lab web adaptation of the supplied publication package.

Download course PDF Download student package

Live Python

pandas Fundamentals: live Python

Stdout

Run Python to see results here.

Status / stderr

Ready to run Python in your browser.

Line-by-line guide

Line 1Load a Python library needed for data work or regression.
Line 3Create or update a Python object used in the analysis.
Line 4Run this Python instruction as part of the lesson workflow.
Line 5Run this Python instruction as part of the lesson workflow.
Line 6Run this Python instruction as part of the lesson workflow.
Line 7Run this Python instruction as part of the lesson workflow.
Line 8Display a result so students can inspect the output.
Line 9Display a result so students can inspect the output.

Python walkthrough

1`import pandas as pd`: Loads a package or function used by the analysis.
2`df = pd.DataFrame({`: Creates or updates a named object used by later steps.
3`"province": ["ON", "QC", "BC", "ON"],`: Executes the next transparent step in the workflow.
4`"income": [62_000, 58_000, 65_000, 71_000],`: Executes the next transparent step in the workflow.
5`"employed": [True, True, False, True],`: Executes the next transparent step in the workflow.
6`})`: Executes the next transparent step in the workflow.
7`print(df.shape)`: Displays a result so it can be checked and interpreted.
8`print(df.dtypes.astype(str).to_dict())`: Displays a result so it can be checked and interpreted.

Live notebook

Run this lesson as a notebook

Open an editable notebook cell-by-cell, run Python in the browser, and download the `.ipynb` file for later.

Open live notebook Download .ipynb Download .py

Related dataset

Ceteris Lab teaching sample

Estimated time

25 to 40 min

Packages

pandas, numpy

Expected output

Printed Python results that can be compared with the lesson explanation.

Learning goals

Load and inspect Ceteris Lab teaching sample.
Run the Python cells connected to pandas Fundamentals.
Interpret the output using Create Series and DataFrames and Inspect rows, columns, indexes, and dtypes.

Common errors

File not found: check that wage_sample.csv is installed or use the course data folder.
Package import error: use the browser notebook first, then download for local Jupyter if your local packages differ.
Column name error: compare your variable names with the dataset variables listed for this notebook.

Dataset path helper

import pandas as pd

df = pd.read_csv("/data/wage_sample.csv")
df.head()

Interactive activity

Chapter 9 interactive

Data evidence planner

Evidence strength: 55%

Unexamined dataValidated evidence

Which choice makes an exploratory result easier to defend?

Record the assumption and limitation in the analysis notes

Immediate feedback

Choose a decision, then test how the claim changes as evidence becomes stronger or weaker.

Try it yourself

Write one plain-English sentence explaining the main idea from this lesson.

Common mistakes

Check these before you move on.

Return to the lesson assumptions, units, diagnostics, and source evidence to replace this shortcut with a defensible interpretation.

Quick quiz

Which statement best answers the Chapter 9 opening question: How can a rectangular dataset preserve labels, dates, and missing values while supporting fast analysis?

Quick quiz

Which practice should be avoided when applying pandas Fundamentals?

Quick quiz

What is the most defensible way to interpret the Python demonstration?

Quick quiz

Why does Chapter 9 matter in an applied econometrics workflow?

Key takeaway

Series and DataFrames add labels and heterogeneous columns to numerical arrays. Inspection must cover structure, types, and missingness. Explicit selection and assignment reduce ambiguity.

Download data Download results Download interpretation