{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "mystnb": {
     "nb_execution_mode": "off"
    }
   },
   "source": [
    "# Total Emissions\n",
    "**Exploring the functionality of NEMED to extract regional total emissions and corresponding emission intensities**<br>\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Extraction\n",
    "#### Import Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": [
     "hide-cell"
    ]
   },
   "outputs": [],
   "source": [
    "import nemed\n",
    "from nemed.downloader import download_aemo_cdeii_summary, download_unit_dispatch\n",
    "from nemed.process import aggregate_data_by\n",
    "\n",
    "# To generate plots shown \n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import plotly.graph_objects as go\n",
    "import plotly.express as px\n",
    "from plotly.subplots import make_subplots\n",
    "\n",
    "# Open plot in browser (optional)\n",
    "import plotly.io as pio\n",
    "pio.renderers.default = \"browser\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "full-width"
    ]
   },
   "source": [
    "#### Processing Emissions Data\n",
    "Total regional emissions and the regional emissions intensities can be extracted using `get_total_emissions`. The following inputs must be specified:\n",
    "- `start_time` define the start of the historical period to collect data for. Must be in the format: \"yyyy/mm/dd HH:MM\"\n",
    "- `end_time` define the end of the historical period to collect data for. Must be in the format: \"yyyy/mm/dd HH:MM\"\n",
    "- `cache` specify the local file directory to temporarily store downloaded files\n",
    "\n",
    "Optionally, you can also pass the arguments:\n",
    "- `filter_regions` specify a list of NEM regions to collect data for. If this is not specified, data is collected for all NEM regions. Must be a list of strings, e.g. ['NSW1','VIC1']\n",
    "- `by` specifys the time resolution to which the data is aggregated. Noting that raw (unaggregated) data is in 5-minute increments which can be obtained by setting this parameter to 'interval' or None. \n",
    "- `generation_sent_out` if set true, auxiliary load factors are considered in the calculation of emissions (as per the CDEII method).\n",
    "- `assume_energy_ramp` if set true, a linear ramp assumption is used to calculate energy generation (MWh). Setting this false will significantly improve computation time but yield less accurate results.\n",
    "- `return_pivot` if set true, results are given as a pivot table on regions instead of the standard dataframe output with REGIONID column.\n",
    "\n",
    "The returned dataframe will contain timeseries data with columns:\n",
    "| Column | Type | Description |\n",
    "| ------ | ---- | ----------- |\n",
    "| TimeBeginning | datetime | reported as the start of the interval or aggregation period (only returned if `by` parameter is used). |\n",
    "| TimeEnding | datetime | reported as the end of the interval, as standard practice using NEM data. |\n",
    "| Region | string | the NEM region corresponding to data. Note 'NEM' field reflects all regions and is returned if `filter_regions` is None. |\n",
    "| Energy | float | the total (sent-out if `generation_sent_out` is True) energy for the corresponding region and time. |\n",
    "| Total_Emissions | float | the total emissions for the corresponding region and time. |\n",
    "| Intensity_Index | float | the intensity index as above, considering the total emissions divided by (sent-out) energy. |\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The simpliest way to collect emissions data is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": [
     "hide-output"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO: Processing total emissions from 2019-01-31 to 2019-02-01\n",
      "INFO: Compiling data for table DISPATCH_UNIT_SCADA\n",
      "INFO: Returning DISPATCH_UNIT_SCADA.\n",
      "INFO: Compiling Energy from Dispatch\n",
      "INFO: Compiling Sent Out Generation\n",
      "INFO: Processing total emissions from 2019-02-01 to 2019-03-01\n",
      "INFO: Compiling data for table DISPATCH_UNIT_SCADA\n",
      "INFO: Returning DISPATCH_UNIT_SCADA.\n",
      "INFO: Compiling Energy from Dispatch\n",
      "INFO: Compiling Sent Out Generation\n",
      "INFO: Loading results file processed_co2_total_2019-01-31_2019-02-01.parquet\n",
      "INFO: Loading results file processed_co2_total_2019-02-01_2019-03-01.parquet\n",
      "INFO: Completed get_total_emissions_by_DI_DUID\n"
     ]
    }
   ],
   "source": [
    "result = nemed.get_total_emissions(start_time=\"2019/02/01 00:00\",\n",
    "                                   end_time=\"2019/03/01 00:00\",\n",
    "                                   cache=\"E:\\TEMPCACHE_nemed_demo\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TimeEnding</th>\n",
       "      <th>Region</th>\n",
       "      <th>Energy</th>\n",
       "      <th>Total_Emissions</th>\n",
       "      <th>Intensity_Index</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2019-02-01 00:05:00</td>\n",
       "      <td>NEM</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2019-02-01 00:05:00</td>\n",
       "      <td>NSW1</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2019-02-01 00:05:00</td>\n",
       "      <td>QLD1</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2019-02-01 00:05:00</td>\n",
       "      <td>SA1</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2019-02-01 00:05:00</td>\n",
       "      <td>TAS1</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48379</th>\n",
       "      <td>2019-03-01 00:00:00</td>\n",
       "      <td>NSW1</td>\n",
       "      <td>575.861778</td>\n",
       "      <td>456.841196</td>\n",
       "      <td>0.793317</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48380</th>\n",
       "      <td>2019-03-01 00:00:00</td>\n",
       "      <td>QLD1</td>\n",
       "      <td>578.012615</td>\n",
       "      <td>438.622799</td>\n",
       "      <td>0.758846</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48381</th>\n",
       "      <td>2019-03-01 00:00:00</td>\n",
       "      <td>SA1</td>\n",
       "      <td>120.111223</td>\n",
       "      <td>44.499818</td>\n",
       "      <td>0.370488</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48382</th>\n",
       "      <td>2019-03-01 00:00:00</td>\n",
       "      <td>TAS1</td>\n",
       "      <td>98.024960</td>\n",
       "      <td>14.633654</td>\n",
       "      <td>0.149285</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48383</th>\n",
       "      <td>2019-03-01 00:00:00</td>\n",
       "      <td>VIC1</td>\n",
       "      <td>465.441392</td>\n",
       "      <td>437.001058</td>\n",
       "      <td>0.938896</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>48384 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "               TimeEnding Region      Energy  Total_Emissions  Intensity_Index\n",
       "0     2019-02-01 00:05:00    NEM    0.000000         0.000000         0.000000\n",
       "1     2019-02-01 00:05:00   NSW1    0.000000         0.000000         0.000000\n",
       "2     2019-02-01 00:05:00   QLD1    0.000000         0.000000         0.000000\n",
       "3     2019-02-01 00:05:00    SA1    0.000000         0.000000         0.000000\n",
       "4     2019-02-01 00:05:00   TAS1    0.000000         0.000000         0.000000\n",
       "...                   ...    ...         ...              ...              ...\n",
       "48379 2019-03-01 00:00:00   NSW1  575.861778       456.841196         0.793317\n",
       "48380 2019-03-01 00:00:00   QLD1  578.012615       438.622799         0.758846\n",
       "48381 2019-03-01 00:00:00    SA1  120.111223        44.499818         0.370488\n",
       "48382 2019-03-01 00:00:00   TAS1   98.024960        14.633654         0.149285\n",
       "48383 2019-03-01 00:00:00   VIC1  465.441392       437.001058         0.938896\n",
       "\n",
       "[48384 rows x 5 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we only wanted emissions for NSW we could selectively define `filter_regions` = ['NSW1'].\n",
    "This will save computation time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": [
     "hide-output"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO: Processing total emissions from 2019-01-31 to 2019-02-01\n",
      "INFO: Compiling data for table DISPATCH_UNIT_SCADA\n",
      "INFO: Returning DISPATCH_UNIT_SCADA.\n",
      "INFO: Compiling Energy from Dispatch\n",
      "INFO: Compiling Sent Out Generation\n",
      "INFO: Processing total emissions from 2019-02-01 to 2019-03-01\n",
      "INFO: Compiling data for table DISPATCH_UNIT_SCADA\n",
      "INFO: Returning DISPATCH_UNIT_SCADA.\n",
      "INFO: Compiling Energy from Dispatch\n",
      "INFO: Compiling Sent Out Generation\n",
      "INFO: Loading results file processed_co2_total_2019-01-31_2019-02-01.parquet\n",
      "INFO: Loading results file processed_co2_total_2019-02-01_2019-03-01.parquet\n",
      "INFO: Completed get_total_emissions_by_DI_DUID\n"
     ]
    }
   ],
   "source": [
    "result = nemed.get_total_emissions(start_time=\"2019/02/01 00:00\",\n",
    "                                   end_time=\"2019/03/01 00:00\",\n",
    "                                   cache=\"E:\\TEMPCACHE_nemed_demo\",\n",
    "                                   filter_regions=['NSW1']\n",
    "                                   )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TimeEnding</th>\n",
       "      <th>Region</th>\n",
       "      <th>Energy</th>\n",
       "      <th>Total_Emissions</th>\n",
       "      <th>Intensity_Index</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2019-02-01 00:05:00</td>\n",
       "      <td>NSW1</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2019-02-01 00:10:00</td>\n",
       "      <td>NSW1</td>\n",
       "      <td>679.117059</td>\n",
       "      <td>540.456506</td>\n",
       "      <td>0.795822</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2019-02-01 00:15:00</td>\n",
       "      <td>NSW1</td>\n",
       "      <td>673.213289</td>\n",
       "      <td>534.759369</td>\n",
       "      <td>0.794339</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2019-02-01 00:20:00</td>\n",
       "      <td>NSW1</td>\n",
       "      <td>670.673463</td>\n",
       "      <td>531.440911</td>\n",
       "      <td>0.792399</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2019-02-01 00:25:00</td>\n",
       "      <td>NSW1</td>\n",
       "      <td>667.649573</td>\n",
       "      <td>528.538772</td>\n",
       "      <td>0.791641</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           TimeEnding Region      Energy  Total_Emissions  Intensity_Index\n",
       "0 2019-02-01 00:05:00   NSW1    0.000000         0.000000         0.000000\n",
       "1 2019-02-01 00:10:00   NSW1  679.117059       540.456506         0.795822\n",
       "2 2019-02-01 00:15:00   NSW1  673.213289       534.759369         0.794339\n",
       "3 2019-02-01 00:20:00   NSW1  670.673463       531.440911         0.792399\n",
       "4 2019-02-01 00:25:00   NSW1  667.649573       528.538772         0.791641"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Say we wanted the results in hourly resolution, we could define the `by` parameter. This can be set as None (for raw 5-minunte intervals) or \"interval\" (for the same thing), or also \"hour\", \"day\", \"month\", \"year\".\n",
    "Using the `by` parameter, the returned dataframe will show both a 'TimeBeginning' and 'TimeEnding' column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": [
     "hide-output"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO: Processing total emissions from 2019-01-31 to 2019-02-01\n",
      "INFO: Compiling data for table DISPATCH_UNIT_SCADA\n",
      "INFO: Returning DISPATCH_UNIT_SCADA.\n",
      "INFO: Compiling Energy from Dispatch\n",
      "INFO: Compiling Sent Out Generation\n",
      "INFO: Processing total emissions from 2019-02-01 to 2019-03-01\n",
      "INFO: Compiling data for table DISPATCH_UNIT_SCADA\n",
      "INFO: Returning DISPATCH_UNIT_SCADA.\n",
      "INFO: Compiling Energy from Dispatch\n",
      "INFO: Compiling Sent Out Generation\n",
      "INFO: Processing total emissions from 2019-03-01 to 2019-04-01\n",
      "INFO: Compiling data for table DISPATCH_UNIT_SCADA\n",
      "INFO: Returning DISPATCH_UNIT_SCADA.\n",
      "INFO: Compiling Energy from Dispatch\n",
      "INFO: Compiling Sent Out Generation\n",
      "INFO: Processing total emissions from 2019-04-01 to 2019-05-01\n",
      "INFO: Compiling data for table DISPATCH_UNIT_SCADA\n",
      "INFO: Returning DISPATCH_UNIT_SCADA.\n",
      "INFO: Compiling Energy from Dispatch\n",
      "INFO: Compiling Sent Out Generation\n",
      "INFO: Loading results file processed_co2_total_2019-01-31_2019-02-01.parquet\n",
      "INFO: Loading results file processed_co2_total_2019-02-01_2019-03-01.parquet\n",
      "INFO: Loading results file processed_co2_total_2019-03-01_2019-04-01.parquet\n",
      "INFO: Loading results file processed_co2_total_2019-04-01_2019-05-01.parquet\n",
      "INFO: Completed get_total_emissions_by_DI_DUID\n"
     ]
    }
   ],
   "source": [
    "result = nemed.get_total_emissions(start_time=\"2019/02/01 00:00\",\n",
    "                                   end_time=\"2019/05/01 00:00\",\n",
    "                                   cache=\"E:\\TEMPCACHE_nemed_demo\",\n",
    "                                   filter_regions=['NSW1','VIC1'],\n",
    "                                   by=\"hour\"\n",
    "                                   )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TimeBeginning</th>\n",
       "      <th>TimeEnding</th>\n",
       "      <th>Region</th>\n",
       "      <th>Energy</th>\n",
       "      <th>Total_Emissions</th>\n",
       "      <th>Intensity_Index</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2019-02-01 00:00:00</td>\n",
       "      <td>2019-02-01 01:00:00</td>\n",
       "      <td>NSW1</td>\n",
       "      <td>7282.753</td>\n",
       "      <td>5748.387</td>\n",
       "      <td>0.789315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2019-02-01 00:00:00</td>\n",
       "      <td>2019-02-01 01:00:00</td>\n",
       "      <td>VIC1</td>\n",
       "      <td>3626.292</td>\n",
       "      <td>3103.639</td>\n",
       "      <td>0.855871</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2019-02-01 01:00:00</td>\n",
       "      <td>2019-02-01 02:00:00</td>\n",
       "      <td>NSW1</td>\n",
       "      <td>7535.616</td>\n",
       "      <td>5865.383</td>\n",
       "      <td>0.778355</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2019-02-01 01:00:00</td>\n",
       "      <td>2019-02-01 02:00:00</td>\n",
       "      <td>VIC1</td>\n",
       "      <td>3607.038</td>\n",
       "      <td>3246.990</td>\n",
       "      <td>0.900182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2019-02-01 02:00:00</td>\n",
       "      <td>2019-02-01 03:00:00</td>\n",
       "      <td>NSW1</td>\n",
       "      <td>7063.113</td>\n",
       "      <td>5494.755</td>\n",
       "      <td>0.777951</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        TimeBeginning          TimeEnding Region    Energy  Total_Emissions  \\\n",
       "0 2019-02-01 00:00:00 2019-02-01 01:00:00   NSW1  7282.753         5748.387   \n",
       "1 2019-02-01 00:00:00 2019-02-01 01:00:00   VIC1  3626.292         3103.639   \n",
       "2 2019-02-01 01:00:00 2019-02-01 02:00:00   NSW1  7535.616         5865.383   \n",
       "3 2019-02-01 01:00:00 2019-02-01 02:00:00   VIC1  3607.038         3246.990   \n",
       "4 2019-02-01 02:00:00 2019-02-01 03:00:00   NSW1  7063.113         5494.755   \n",
       "\n",
       "   Intensity_Index  \n",
       "0         0.789315  \n",
       "1         0.855871  \n",
       "2         0.778355  \n",
       "3         0.900182  \n",
       "4         0.777951  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, we can `get_total_emissions` without any aggregation and later call the function `aggregate_data_by` to collect different time resolved datasets as such:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "tags": [
     "hide-output"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO: Processing total emissions from 2019-01-31 to 2019-02-01\n",
      "INFO: Compiling data for table DISPATCH_UNIT_SCADA\n",
      "INFO: Returning DISPATCH_UNIT_SCADA.\n",
      "INFO: Compiling Energy from Dispatch\n",
      "INFO: Compiling Sent Out Generation\n",
      "INFO: Processing total emissions from 2019-02-01 to 2019-03-01\n",
      "INFO: Compiling data for table DISPATCH_UNIT_SCADA\n",
      "INFO: Returning DISPATCH_UNIT_SCADA.\n",
      "INFO: Compiling Energy from Dispatch\n",
      "INFO: Compiling Sent Out Generation\n",
      "INFO: Processing total emissions from 2019-03-01 to 2019-04-01\n",
      "INFO: Compiling data for table DISPATCH_UNIT_SCADA\n",
      "INFO: Returning DISPATCH_UNIT_SCADA.\n",
      "INFO: Compiling Energy from Dispatch\n",
      "INFO: Compiling Sent Out Generation\n",
      "INFO: Processing total emissions from 2019-04-01 to 2019-05-01\n",
      "INFO: Compiling data for table DISPATCH_UNIT_SCADA\n",
      "INFO: Returning DISPATCH_UNIT_SCADA.\n",
      "INFO: Compiling Energy from Dispatch\n",
      "INFO: Compiling Sent Out Generation\n",
      "INFO: Loading results file processed_co2_total_2019-01-31_2019-02-01.parquet\n",
      "INFO: Loading results file processed_co2_total_2019-02-01_2019-03-01.parquet\n",
      "INFO: Loading results file processed_co2_total_2019-03-01_2019-04-01.parquet\n",
      "INFO: Loading results file processed_co2_total_2019-04-01_2019-05-01.parquet\n",
      "INFO: Completed get_total_emissions_by_DI_DUID\n"
     ]
    }
   ],
   "source": [
    "result = nemed.get_total_emissions(start_time=\"2019/02/01 00:00\",\n",
    "                                   end_time=\"2019/05/01 00:00\",\n",
    "                                   cache=\"E:\\TEMPCACHE_nemed_demo\",\n",
    "                                   filter_regions=['NSW1'],\n",
    "                                   by=None\n",
    "                                   )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TimeEnding</th>\n",
       "      <th>Region</th>\n",
       "      <th>Energy</th>\n",
       "      <th>Total_Emissions</th>\n",
       "      <th>Intensity_Index</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2019-02-01 00:05:00</td>\n",
       "      <td>NSW1</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2019-02-01 00:10:00</td>\n",
       "      <td>NSW1</td>\n",
       "      <td>679.117059</td>\n",
       "      <td>540.456506</td>\n",
       "      <td>0.795822</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2019-02-01 00:15:00</td>\n",
       "      <td>NSW1</td>\n",
       "      <td>673.213289</td>\n",
       "      <td>534.759369</td>\n",
       "      <td>0.794339</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2019-02-01 00:20:00</td>\n",
       "      <td>NSW1</td>\n",
       "      <td>670.673463</td>\n",
       "      <td>531.440911</td>\n",
       "      <td>0.792399</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2019-02-01 00:25:00</td>\n",
       "      <td>NSW1</td>\n",
       "      <td>667.649573</td>\n",
       "      <td>528.538772</td>\n",
       "      <td>0.791641</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           TimeEnding Region      Energy  Total_Emissions  Intensity_Index\n",
       "0 2019-02-01 00:05:00   NSW1    0.000000         0.000000         0.000000\n",
       "1 2019-02-01 00:10:00   NSW1  679.117059       540.456506         0.795822\n",
       "2 2019-02-01 00:15:00   NSW1  673.213289       534.759369         0.794339\n",
       "3 2019-02-01 00:20:00   NSW1  670.673463       531.440911         0.792399\n",
       "4 2019-02-01 00:25:00   NSW1  667.649573       528.538772         0.791641"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TimeBeginning</th>\n",
       "      <th>TimeEnding</th>\n",
       "      <th>Region</th>\n",
       "      <th>Energy</th>\n",
       "      <th>Total_Emissions</th>\n",
       "      <th>Intensity_Index</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2019-02-01 00:00:00</td>\n",
       "      <td>2019-02-01 01:00:00</td>\n",
       "      <td>NSW1</td>\n",
       "      <td>7282.753</td>\n",
       "      <td>5748.387</td>\n",
       "      <td>0.789</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2019-02-01 01:00:00</td>\n",
       "      <td>2019-02-01 02:00:00</td>\n",
       "      <td>NSW1</td>\n",
       "      <td>7535.616</td>\n",
       "      <td>5865.383</td>\n",
       "      <td>0.778</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2019-02-01 02:00:00</td>\n",
       "      <td>2019-02-01 03:00:00</td>\n",
       "      <td>NSW1</td>\n",
       "      <td>7063.113</td>\n",
       "      <td>5494.755</td>\n",
       "      <td>0.778</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2019-02-01 03:00:00</td>\n",
       "      <td>2019-02-01 04:00:00</td>\n",
       "      <td>NSW1</td>\n",
       "      <td>6849.878</td>\n",
       "      <td>5370.722</td>\n",
       "      <td>0.784</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2019-02-01 04:00:00</td>\n",
       "      <td>2019-02-01 05:00:00</td>\n",
       "      <td>NSW1</td>\n",
       "      <td>7506.841</td>\n",
       "      <td>5956.088</td>\n",
       "      <td>0.793</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        TimeBeginning          TimeEnding Region    Energy  Total_Emissions  \\\n",
       "0 2019-02-01 00:00:00 2019-02-01 01:00:00   NSW1  7282.753         5748.387   \n",
       "1 2019-02-01 01:00:00 2019-02-01 02:00:00   NSW1  7535.616         5865.383   \n",
       "2 2019-02-01 02:00:00 2019-02-01 03:00:00   NSW1  7063.113         5494.755   \n",
       "3 2019-02-01 03:00:00 2019-02-01 04:00:00   NSW1  6849.878         5370.722   \n",
       "4 2019-02-01 04:00:00 2019-02-01 05:00:00   NSW1  7506.841         5956.088   \n",
       "\n",
       "   Intensity_Index  \n",
       "0            0.789  \n",
       "1            0.778  \n",
       "2            0.778  \n",
       "3            0.784  \n",
       "4            0.793  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hourly = aggregate_data_by(result, \"hour\")\n",
    "hourly.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TimeBeginning</th>\n",
       "      <th>TimeEnding</th>\n",
       "      <th>Region</th>\n",
       "      <th>Energy</th>\n",
       "      <th>Total_Emissions</th>\n",
       "      <th>Intensity_Index</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2019-02-01</td>\n",
       "      <td>2019-03-01</td>\n",
       "      <td>NSW1</td>\n",
       "      <td>5357381.531</td>\n",
       "      <td>4159597.888</td>\n",
       "      <td>0.776</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2019-03-01</td>\n",
       "      <td>2019-04-01</td>\n",
       "      <td>NSW1</td>\n",
       "      <td>5528872.839</td>\n",
       "      <td>4307736.884</td>\n",
       "      <td>0.779</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2019-04-01</td>\n",
       "      <td>2019-05-01</td>\n",
       "      <td>NSW1</td>\n",
       "      <td>4988808.915</td>\n",
       "      <td>3897627.460</td>\n",
       "      <td>0.781</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  TimeBeginning TimeEnding Region       Energy  Total_Emissions  \\\n",
       "0    2019-02-01 2019-03-01   NSW1  5357381.531      4159597.888   \n",
       "1    2019-03-01 2019-04-01   NSW1  5528872.839      4307736.884   \n",
       "2    2019-04-01 2019-05-01   NSW1  4988808.915      3897627.460   \n",
       "\n",
       "   Intensity_Index  \n",
       "0            0.776  \n",
       "1            0.779  \n",
       "2            0.781  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "month = aggregate_data_by(result, \"month\")\n",
    "month.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can also easily collect and format data into a pivot table by setting `return_pivot` = True "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "tags": [
     "hide-output"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO: Processing total emissions from 2019-01-31 to 2019-02-01\n",
      "INFO: Compiling data for table DISPATCH_UNIT_SCADA\n",
      "INFO: Returning DISPATCH_UNIT_SCADA.\n",
      "INFO: Compiling Energy from Dispatch\n",
      "INFO: Compiling Sent Out Generation\n",
      "INFO: Processing total emissions from 2019-02-01 to 2019-03-01\n",
      "INFO: Compiling data for table DISPATCH_UNIT_SCADA\n",
      "INFO: Returning DISPATCH_UNIT_SCADA.\n",
      "INFO: Compiling Energy from Dispatch\n",
      "INFO: Compiling Sent Out Generation\n",
      "INFO: Processing total emissions from 2019-03-01 to 2019-04-01\n",
      "INFO: Compiling data for table DISPATCH_UNIT_SCADA\n",
      "INFO: Returning DISPATCH_UNIT_SCADA.\n",
      "INFO: Compiling Energy from Dispatch\n",
      "INFO: Compiling Sent Out Generation\n",
      "INFO: Processing total emissions from 2019-04-01 to 2019-05-01\n",
      "INFO: Compiling data for table DISPATCH_UNIT_SCADA\n",
      "INFO: Returning DISPATCH_UNIT_SCADA.\n",
      "INFO: Compiling Energy from Dispatch\n",
      "INFO: Compiling Sent Out Generation\n",
      "INFO: Loading results file processed_co2_total_2019-01-31_2019-02-01.parquet\n",
      "INFO: Loading results file processed_co2_total_2019-02-01_2019-03-01.parquet\n",
      "INFO: Loading results file processed_co2_total_2019-03-01_2019-04-01.parquet\n",
      "INFO: Loading results file processed_co2_total_2019-04-01_2019-05-01.parquet\n",
      "INFO: Completed get_total_emissions_by_DI_DUID\n"
     ]
    }
   ],
   "source": [
    "result = nemed.get_total_emissions(start_time=\"2019/02/01 00:00\",\n",
    "                                   end_time=\"2019/05/01 00:00\",\n",
    "                                   cache=\"E:\\TEMPCACHE_nemed_demo\",\n",
    "                                   filter_regions=['NSW1','QLD1'],\n",
    "                                   by=\"month\",\n",
    "                                   return_pivot=True\n",
    "                                   )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"2\" halign=\"left\">Energy</th>\n",
       "      <th colspan=\"2\" halign=\"left\">Total_Emissions</th>\n",
       "      <th colspan=\"2\" halign=\"left\">Intensity_Index</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Region</th>\n",
       "      <th>NSW1</th>\n",
       "      <th>QLD1</th>\n",
       "      <th>NSW1</th>\n",
       "      <th>QLD1</th>\n",
       "      <th>NSW1</th>\n",
       "      <th>QLD1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TimeEnding</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2019-03-01</th>\n",
       "      <td>5357381.531</td>\n",
       "      <td>4826147.929</td>\n",
       "      <td>4159597.888</td>\n",
       "      <td>3575844.389</td>\n",
       "      <td>0.776424</td>\n",
       "      <td>0.740931</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-01</th>\n",
       "      <td>5528872.839</td>\n",
       "      <td>5427001.962</td>\n",
       "      <td>4307736.884</td>\n",
       "      <td>4029573.218</td>\n",
       "      <td>0.779135</td>\n",
       "      <td>0.742504</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-05-01</th>\n",
       "      <td>4988808.915</td>\n",
       "      <td>4850178.561</td>\n",
       "      <td>3897627.460</td>\n",
       "      <td>3627099.250</td>\n",
       "      <td>0.781274</td>\n",
       "      <td>0.747828</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 Energy              Total_Emissions               \\\n",
       "Region             NSW1         QLD1            NSW1         QLD1   \n",
       "TimeEnding                                                          \n",
       "2019-03-01  5357381.531  4826147.929     4159597.888  3575844.389   \n",
       "2019-04-01  5528872.839  5427001.962     4307736.884  4029573.218   \n",
       "2019-05-01  4988808.915  4850178.561     3897627.460  3627099.250   \n",
       "\n",
       "           Intensity_Index            \n",
       "Region                NSW1      QLD1  \n",
       "TimeEnding                            \n",
       "2019-03-01        0.776424  0.740931  \n",
       "2019-04-01        0.779135  0.742504  \n",
       "2019-05-01        0.781274  0.747828  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sample Plots\n",
    "A very quick glimpse at plotting this data is shown below. Firstly via a box plot. Secondly, a bit of fun distribution! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "tags": [
     "hide-output"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO: Processing total emissions from 2018-12-31 to 2019-01-01\n",
      "INFO: Compiling data for table DISPATCH_UNIT_SCADA\n",
      "INFO: Returning DISPATCH_UNIT_SCADA.\n",
      "INFO: Compiling Sent Out Generation\n",
      "INFO: Processing total emissions from 2019-01-01 to 2019-02-01\n",
      "INFO: Compiling data for table DISPATCH_UNIT_SCADA\n",
      "INFO: Returning DISPATCH_UNIT_SCADA.\n",
      "INFO: Compiling Sent Out Generation\n",
      "INFO: Processing total emissions from 2019-02-01 to 2019-03-01\n",
      "INFO: Compiling data for table DISPATCH_UNIT_SCADA\n",
      "INFO: Returning DISPATCH_UNIT_SCADA.\n",
      "INFO: Compiling Sent Out Generation\n",
      "INFO: Processing total emissions from 2019-03-01 to 2019-04-01\n",
      "INFO: Compiling data for table DISPATCH_UNIT_SCADA\n",
      "INFO: Returning DISPATCH_UNIT_SCADA.\n",
      "INFO: Compiling Sent Out Generation\n",
      "INFO: Processing total emissions from 2019-04-01 to 2019-05-01\n",
      "INFO: Compiling data for table DISPATCH_UNIT_SCADA\n",
      "INFO: Returning DISPATCH_UNIT_SCADA.\n",
      "INFO: Compiling Sent Out Generation\n",
      "INFO: Processing total emissions from 2019-05-01 to 2019-06-01\n",
      "INFO: Compiling data for table DISPATCH_UNIT_SCADA\n",
      "INFO: Returning DISPATCH_UNIT_SCADA.\n",
      "INFO: Compiling Sent Out Generation\n",
      "INFO: Processing total emissions from 2019-06-01 to 2019-07-01\n",
      "INFO: Compiling data for table DISPATCH_UNIT_SCADA\n",
      "INFO: Returning DISPATCH_UNIT_SCADA.\n",
      "INFO: Compiling Sent Out Generation\n",
      "INFO: Processing total emissions from 2019-07-01 to 2019-08-01\n",
      "INFO: Compiling data for table DISPATCH_UNIT_SCADA\n",
      "INFO: Returning DISPATCH_UNIT_SCADA.\n",
      "INFO: Compiling Sent Out Generation\n",
      "INFO: Processing total emissions from 2019-08-01 to 2019-09-01\n",
      "INFO: Compiling data for table DISPATCH_UNIT_SCADA\n",
      "INFO: Returning DISPATCH_UNIT_SCADA.\n",
      "INFO: Compiling Sent Out Generation\n",
      "INFO: Processing total emissions from 2019-09-01 to 2019-10-01\n",
      "INFO: Compiling data for table DISPATCH_UNIT_SCADA\n",
      "INFO: Returning DISPATCH_UNIT_SCADA.\n",
      "INFO: Compiling Sent Out Generation\n",
      "INFO: Processing total emissions from 2019-10-01 to 2019-11-01\n",
      "INFO: Compiling data for table DISPATCH_UNIT_SCADA\n",
      "INFO: Returning DISPATCH_UNIT_SCADA.\n",
      "INFO: Compiling Sent Out Generation\n",
      "INFO: Processing total emissions from 2019-11-01 to 2019-12-01\n",
      "INFO: Compiling data for table DISPATCH_UNIT_SCADA\n",
      "INFO: Returning DISPATCH_UNIT_SCADA.\n",
      "INFO: Compiling Sent Out Generation\n",
      "INFO: Processing total emissions from 2019-12-01 to 2020-01-01\n",
      "INFO: Compiling data for table DISPATCH_UNIT_SCADA\n",
      "INFO: Returning DISPATCH_UNIT_SCADA.\n",
      "INFO: Compiling Sent Out Generation\n",
      "INFO: Loading results file processed_co2_total_2018-12-31_2019-01-01.parquet\n",
      "INFO: Loading results file processed_co2_total_2019-01-01_2019-02-01.parquet\n",
      "INFO: Loading results file processed_co2_total_2019-02-01_2019-03-01.parquet\n",
      "INFO: Loading results file processed_co2_total_2019-03-01_2019-04-01.parquet\n",
      "INFO: Loading results file processed_co2_total_2019-04-01_2019-05-01.parquet\n",
      "INFO: Loading results file processed_co2_total_2019-05-01_2019-06-01.parquet\n",
      "INFO: Loading results file processed_co2_total_2019-06-01_2019-07-01.parquet\n",
      "INFO: Loading results file processed_co2_total_2019-07-01_2019-08-01.parquet\n",
      "INFO: Loading results file processed_co2_total_2019-08-01_2019-09-01.parquet\n",
      "INFO: Loading results file processed_co2_total_2019-09-01_2019-10-01.parquet\n",
      "INFO: Loading results file processed_co2_total_2019-10-01_2019-11-01.parquet\n",
      "INFO: Loading results file processed_co2_total_2019-11-01_2019-12-01.parquet\n",
      "INFO: Loading results file processed_co2_total_2019-12-01_2020-01-01.parquet\n",
      "INFO: Completed get_total_emissions_by_DI_DUID\n"
     ]
    }
   ],
   "source": [
    "co2_data = nemed.get_total_emissions(start_time=\"2019/01/01 00:00\",\n",
    "                                     end_time=\"2020/01/01 00:00\",\n",
    "                                     cache=\"E:\\TEMPCACHE_nemed_demo\",\n",
    "                                     filter_regions=None,\n",
    "                                     by=\"hour\",\n",
    "                                     assume_energy_ramp=False)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Chart Styling:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "tags": [
     "hide-cell"
    ]
   },
   "outputs": [],
   "source": [
    "colors = ['#676765','#af7635','#8F6593','#4F759B', '#638b66','#b65551']\n",
    "def NORD_theme():\n",
    "    plotly_NORD_theme = pio.templates[\"simple_white\"]\n",
    "    plotly_NORD_theme.layout.plot_bgcolor = \"#f4f4f5\" \n",
    "    plotly_NORD_theme.layout.paper_bgcolor = \"#FFFFFF\"\n",
    "    plotly_NORD_theme.layout.xaxis.gridcolor = '#d8dee9'\n",
    "    plotly_NORD_theme.layout.yaxis.gridcolor = '#d8dee9'\n",
    "    return plotly_NORD_theme\n",
    "\n",
    "def set_font_size(layout, font_size=16):\n",
    "    layout['titlefont']['size'] = font_size + 4\n",
    "    layout.legend['font']['size'] = font_size\n",
    "\n",
    "    for ax in [item for item in layout if item.__contains__('xaxis')]:\n",
    "        layout[ax].titlefont.size = font_size\n",
    "        layout[ax].tickfont.size = font_size\n",
    "\n",
    "    for ax in [item for item in layout if item.__contains__('yaxis')]:\n",
    "        layout[ax].titlefont.size = font_size\n",
    "        layout[ax].tickfont.size = font_size"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Chart 1 - Box Plot Distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "tags": [
     "hide-cell",
     "full-width"
    ]
   },
   "outputs": [],
   "source": [
    "plt_df = co2_data.copy()\n",
    "plt_df['hour'] = plt_df['TimeEnding'].dt.hour\n",
    "fig = px.box(plt_df, x=\"hour\", y=\"Intensity_Index\", color=\"Region\", color_discrete_sequence=colors, points=False)\n",
    "fig.update_xaxes(title=\"Hour of Day\", tickvals=[i for i in range(0,25)], mirror=True)\n",
    "fig.update_yaxes(title=\"Emissions Intensity Index (tCO2-e/MWh)\", range=[-0.1,1.2], \\\n",
    "    tickvals=[i*10**-2 for i in range(0,120,10)], mirror=True, showgrid=True)\n",
    "fig.update_layout(title=f\"CY2019 NEM Regional Hourly Emissions Intensity<br><sup>\"+\\\n",
    "                         \"NEMED | Average Emissions Methodology | Box Plot</sup>\",\n",
    "                  legend={'title':'Region','xanchor':'center','x':0.5, 'y':-0.1, 'orientation':'h'},\n",
    "                  template=NORD_theme(), boxmode='group', boxgroupgap=0.1, boxgap=0.5)\n",
    "\n",
    "FONT_SIZE = 16\n",
    "FONT_STYLE = \"Raleway\"\n",
    "fonts = dict(tickfont=dict(size=FONT_SIZE, family=FONT_STYLE),\n",
    "            titlefont=dict(size=FONT_SIZE, family=FONT_STYLE))\n",
    "fig.update_layout(xaxis=fonts, yaxis=fonts,\\\n",
    "                  legend=dict(font=dict(size=FONT_SIZE, family=FONT_STYLE)),\n",
    "                  title_font_family=FONT_STYLE,\n",
    "                  title_font_size=22)\n",
    "fig.update_annotations(font=dict(size=FONT_SIZE, family=FONT_STYLE))\n",
    "fig.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{admonition} Interactive Plot\n",
    ":class: full-width\n",
    "Click the image to open the plot as an interactive plotly\n",
    "```\n",
    "\n",
    "```{image} charts/total_emissions_chart_1.png\n",
    ":class: full-width\n",
    ":target: ../_static/html_charts/total_emissions_chart_1.html\n",
    "```"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Chart 2 - Scatter Density Distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "tags": [
     "hide-cell",
     "full-width"
    ]
   },
   "outputs": [],
   "source": [
    "plt_df = co2_data.copy()\n",
    "plt_df['hour'] = plt_df['TimeEnding'].dt.hour\n",
    "plt_df['day'] = pd.to_datetime(dict(year=plt_df['TimeEnding'].dt.year,\n",
    "                                    month=plt_df['TimeEnding'].dt.month,\n",
    "                                    day=plt_df['TimeEnding'].dt.day))\n",
    "plt_df['day'] = plt_df['day'].astype(str)\n",
    "\n",
    "data = plt_df\n",
    "\n",
    "# Time axis bins/increments\n",
    "data['xbins'] = data['hour']\n",
    "\n",
    "# Vertical axis bins/increments\n",
    "bands = pd.cut(data['Intensity_Index'], 40)\n",
    "data['ybands'] = bands\n",
    "\n",
    "# For time axis, count vertical bands\n",
    "bar_df = data.pivot_table(index=['Region','xbins'], columns='ybands', values='Intensity_Index', aggfunc=len, \\\n",
    "                          dropna=False)\n",
    "bar_df.fillna(0, inplace=True)\n",
    "\n",
    "# # Summary for bar table\n",
    "bar_summary = pd.DataFrame(data.groupby(['Region', 'xbins', 'ybands']).size())\n",
    "bar_summary.columns = ['count']\n",
    "bar_summary.reset_index(inplace=True)\n",
    "\n",
    "bar_summary['bin_end'] = bar_summary['xbins'].astype(str)\n",
    "bar_summary['bands_end'] = [bar_summary['ybands'][i].right for i in range(len(bar_summary))]\n",
    "bar_summary['bands_end'] = bar_summary['bands_end'].astype(float)\n",
    "\n",
    "# # Plotly\n",
    "fig = px.scatter(bar_summary, x=\"bin_end\", y=\"bands_end\", color=\"Region\", size=\"count\", size_max=40, \\\n",
    "                 color_discrete_sequence=colors, opacity=.75)\n",
    "fig.update_xaxes(title=\"Hour of Day\", tickvals=[i for i in range(0,25)], mirror=True)\n",
    "fig.update_yaxes(title=\"Emissions Intensity Index (tCO2-e/MWh)\", range=[-0.1,1.2], \\\n",
    "                 tickvals=[i*10**-2 for i in range(0,120,10)], mirror=True, showgrid=True)\n",
    "fig.update_layout(title=f\"CY2019 NEM Regional Hourly Emissions Intensity<br><sup>\"+\\\n",
    "                         \"NEMED | Average Emissions Methodology | Scatter Density Plot</sup>\",\n",
    "                  legend={'title':'Region','xanchor':'center','x':0.5, 'y':-0.1, 'orientation':'h'},\n",
    "                  template=NORD_theme())\n",
    "\n",
    "FONT_SIZE = 16\n",
    "FONT_STYLE = \"Raleway\"\n",
    "fonts = dict(tickfont=dict(size=FONT_SIZE, family=FONT_STYLE),\n",
    "            titlefont=dict(size=FONT_SIZE, family=FONT_STYLE))\n",
    "fig.update_layout(xaxis=fonts, yaxis=fonts,\\\n",
    "                  legend=dict(font=dict(size=FONT_SIZE, family=FONT_STYLE)),\n",
    "                  title_font_family=FONT_STYLE,\n",
    "                  title_font_size=22)\n",
    "fig.update_annotations(font=dict(size=FONT_SIZE, family=FONT_STYLE))\n",
    "fig.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{image} charts/total_emissions_chart_2.png\n",
    ":class: full-width\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.12 ('nempy')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12 (main, Apr  4 2022, 05:22:27) [MSC v.1916 64 bit (AMD64)]"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "4aab49ac747d4948ee2428bd46f4ac833ef94a37ecb38233c747c75e4d05fe4b"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}