Lesson 35
Explainability, Uncertainty, and Model Governance
Big question
How do we decide whether a powerful prediction model is trustworthy enough to use?
Lesson progress
Complete checkpoints as you learn
Learning objectives
- Explain and apply coefficients versus feature importance.
- Explain and apply permutation importance.
- Explain and apply partial dependence.
- Explain and apply SHAP as an optional tool.
- Explain and apply calibration.
- Explain and apply bootstrap uncertainty.
- Explain and apply conformal prediction.
- Explain and apply distribution shift.
- Explain and apply model cards and audit trails.
Simple explanation
This chapter begins from the economic question rather than from software syntax. The goal is to understand what information the data can legitimately provide about explainability, uncertainty, and model governance. Python is used as a transparent laboratory: we state the question, define the target, inspect the data, estimate the model, and then challenge the result. A recurring distinction in econometrics is the difference between describing a pattern, predicting an outcome, and estimating a causal effect. These goals can use similar equations but require different assumptions. A model may forecast well while offering little causal interpretation, and a credible causal design may deliberately use a simple estimator because identification comes from the research design rather than algorithmic complexity. Throughout the chapter, pay attention to units, timing, sample construction, and the information set available to the analyst. Economic data are produced by institutions, markets, households, and firms. They are not abstract arrays. A good empirical workflow therefore combines statistical discipline with substantive economic reasoning.
Key terms
- coefficients versus feature importance
- A core idea in Chapter 35 that students apply carefully in economic analysis.
- permutation importance
- A core idea in Chapter 35 that students apply carefully in economic analysis.
- partial dependence
- A core idea in Chapter 35 that students apply carefully in economic analysis.
- SHAP as an optional tool
- A core idea in Chapter 35 that students apply carefully in economic analysis.
- calibration
- A core idea in Chapter 35 that students apply carefully in economic analysis.
- bootstrap uncertainty
- A core idea in Chapter 35 that students apply carefully in economic analysis.
Chapter 35 model
Interpret the expression in words and units before using it in a claim.
Example
Start with a tiny hypothetical sample that can be checked by hand. Write the unit of observation, outcome, explanatory variable or treatment, and the comparison being made. Before estimating anything, ask what would make the comparison informative and what could make it misleading. The difference in group averages is easy to compute, but whether it is a causal effect depends on how exposure was assigned and whether the groups are otherwise comparable. This distinction is the heartbeat of modern applied econometrics.
Prerequisites
- Complete the preceding course chapters or review their summaries as needed.
Full theory and examples
35.2
Economic intuition
This chapter begins from the economic question rather than from software syntax. The goal is to understand what information the data can legitimately provide about explainability, uncertainty, and model governance. Python is used as a transparent laboratory: we state the question, define the target, inspect the data, estimate the model, and then challenge the result.
A recurring distinction in econometrics is the difference between describing a pattern, predicting an outcome, and estimating a causal effect. These goals can use similar equations but require different assumptions. A model may forecast well while offering little causal interpretation, and a credible causal design may deliberately use a simple estimator because identification comes from the research design rather than algorithmic complexity.
Throughout the chapter, pay attention to units, timing, sample construction, and the information set available to the analyst. Economic data are produced by institutions, markets, households, and firms. They are not abstract arrays. A good empirical workflow therefore combines statistical discipline with substantive economic reasoning.
35.3
A small numerical example
Start with a tiny hypothetical sample that can be checked by hand. Write the unit of observation, outcome, explanatory variable or treatment, and the comparison being made. Before estimating anything, ask what would make the comparison informative and what could make it misleading.
The difference in group averages is easy to compute, but whether it is a causal effect depends on how exposure was assigned and whether the groups are otherwise comparable. This distinction is the heartbeat of modern applied econometrics.
| Observation | X / Treatment | Outcome Y | Interpretive note |
|---|---|---|---|
| 1 | 0 | 10 | baseline |
| 2 | 0 | 12 | comparison |
| 3 | 1 | 15 | exposed |
| 4 | 1 | 17 | exposed |
35.4
Formal framework
Write the empirical model in a form that makes the target explicit. A generic conditional-mean representation is
E[Y | X] = m(X), or in a linear approximation, Y_i = β₀ + β₁X_i + u_i.
Here Y is the outcome, X denotes observed information, β parameters summarize the chosen model, and u collects other determinants of the outcome. The crucial question is not only whether β can be estimated, but what β means under the design and assumptions.
35.5
Python demonstration
import numpy as np
rng=np.random.default_rng(35); pred=rng.normal(100,10,300); actual=pred+rng.normal(0,5,300)
err=actual-pred
print('mean error',err.mean()); print('RMSE',np.sqrt(np.mean(err**2))); print('90% empirical error interval',np.quantile(err,[.05,.95]))
Run the code in a clean notebook and inspect both the numerical output and the data-generating assumptions. The associated notebook in the student package reproduces the example with a fixed random seed.
Figure 35.1. How do we decide whether a powerful prediction model is trustworthy enough to use?
35.6
Interpretation
Interpret results in the units of the variables. Report sign, magnitude, uncertainty, and the population or time period to which the estimate applies. If a variable is logged, categorical, interacted, standardized, or transformed, translate the coefficient accordingly rather than reading it as a raw one-unit effect.
Then separate statistical significance from economic significance. A precise estimate may be too small to matter economically; an economically important estimate may be imprecise. Both dimensions belong in the conclusion.
35.7
Assumptions, diagnostics, and threats to validity
Is the unit of observation appropriate for the question?
Are timing and information sets respected?
Could selection, confounding, measurement error, or reverse causality matter?
Does the uncertainty estimator match the sampling or dependence structure?
Would the result survive reasonable alternative specifications?
Is the task predictive, descriptive, or causal?
Visual evidence
Guided practice
- 1Modify the notebook so that one assumption is deliberately violated. Re-estimate the model, compare the result, and explain why the estimate changed. This turns diagnostics from a checklist into an experiment.
Exercises
- 1Concept: In one paragraph, distinguish association, prediction, and causation for the chapter topic.
- 2Python: Reproduce the demonstration with a different documented random seed and verify that the qualitative conclusion is stable.
- 3Applied econometrics: Replace the simulated outcome with a legally shareable economic variable and document its units, source, and sample.
- 4Interpretation: Write a 150-word results paragraph that reports magnitude and uncertainty without exaggerating the evidence.
- 5Research challenge: Propose one alternative design or robustness check that would address the most important threat to validity.
Source and downloads
Chapter 35 of Fundamentals of Python for Financial Econometrics by Mohammad Safavi, Ph.D.. The lesson is an original Ceteris Lab web adaptation of the supplied publication package.
Live Python
Explainability, Uncertainty, and Model Governance: live Python
Explainability, Uncertainty, and Model Governance: live Python
Stdout
Run Python to see results here.
Status / stderr
Ready to run Python in your browser.
Line-by-line guide
- Line 1Display a result so students can inspect the output.
Python walkthrough
- 1Connect the conceptual chapter to an explicit, reproducible workflow.
Live notebook
Run this lesson as a notebook
Open an editable notebook cell-by-cell, run Python in the browser, and download the `.ipynb` file for later.
Related dataset
Ceteris Lab teaching sample
Estimated time
25 to 40 min
Packages
pandas, numpy
Expected output
Printed Python results that can be compared with the lesson explanation.
Learning goals
- Load and inspect Ceteris Lab teaching sample.
- Run the Python cells connected to Explainability, Uncertainty, and Model Governance.
- Interpret the output using coefficients versus feature importance and permutation importance.
Common errors
- File not found: check that wage_sample.csv is installed or use the course data folder.
- Package import error: use the browser notebook first, then download for local Jupyter if your local packages differ.
- Column name error: compare your variable names with the dataset variables listed for this notebook.
Dataset path helper
import pandas as pd
df = pd.read_csv("/data/wage_sample.csv")
df.head()Interactive activity
Chapter 35 interactive
Assumption stress test
What should determine the strength of an econometric claim?
Immediate feedback
Choose a decision, then test how the claim changes as evidence becomes stronger or weaker.
Try it yourself
Write one plain-English sentence explaining the main idea from this lesson.
Common mistakes
Check these before you move on.
A regression coefficient describes a pattern unless the assumptions or research design support a causal interpretation.
Quick quiz
Which statement best answers the Chapter 35 opening question: How do we decide whether a powerful prediction model is trustworthy enough to use?
Quick quiz
Which practice should be avoided when applying Explainability, Uncertainty, and Model Governance?
Quick quiz
What is the most defensible way to interpret the Python demonstration?
Quick quiz
Why does Chapter 35 matter in an applied econometrics workflow?
Key takeaway
Explainability, Uncertainty, and Model Governance is most useful when the economic question, statistical target, assumptions, code, and interpretation point in the same direction. Python makes the workflow reproducible; econometric reasoning determines what the output can mean.