{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Importing, Cleaning, Validating, and Exporting Data\n",
        "\n**Opening question:** How can messy input be transformed without erasing the evidence of what was changed?\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "from io import StringIO\n",
        "\n",
        "raw = StringIO(\"date,value\\n2026-01-01,2.1\\n2026-02-01,NA\\n2026-02-01,2.4\")\n",
        "df = pd.read_csv(raw, parse_dates=[\"date\"], na_values=[\"NA\"])\n",
        "df[\"is_duplicate_date\"] = df.duplicated(\"date\", keep=False)\n",
        "print(df.isna().sum().to_dict())\n",
        "print(df[\"is_duplicate_date\"].sum())\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "**Interpretation check:** Interpretation. The missing-value count and duplicate flag describe separate quality concerns.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "def validate_series(frame):\n",
        "    assert frame[\"date\"].notna().all(), \"dates must be present\"\n",
        "    assert frame[\"date\"].is_monotonic_increasing, \"dates must be sorted\"\n",
        "    assert frame[\"value\"].dropna().between(-100, 100).all(), \"value outside range\"\n",
        "    return True\n",
        "\n",
        "print(validate_series(df.sort_values(\"date\")))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "**Interpretation check:** Interpretation. The range is intentionally broad for illustration. Real thresholds should reflect the variable and units.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "from io import StringIO\n",
        "raw = StringIO(\"date,value\\n2026-01-01,2.1\\n2026-02-01,NA\\n2026-02-01,2.4\")\n",
        "df = pd.read_csv(raw, parse_dates=[\"date\"], na_values=[\"NA\"])\n",
        "df[\"is_duplicate_date\"] = df.duplicated(\"date\", keep=False)\n",
        "print(df.isna().sum().to_dict())\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Verified source output\n",
        "\n",
        "```text\n{'date': 0, 'value': 1, 'is_duplicate_date': 0} 2\n```\n\n```text\nTrue\n```\n\n```text\n{'date': 0, 'value': 1, 'is_duplicate_date': 0}\n2\n```\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3"
    },
    "ceteris_lab": {
      "course_slug": "fundamentals-python-econometrics",
      "source_derived": true,
      "course_title": "Fundamentals of Python for Financial Econometrics",
      "chapter": 10,
      "source_origin": "source-derived"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}