{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Introduction to Digital Earth Australia \n", "\n", "* **[Sign up to the DEA Sandbox](https://app.sandbox.dea.ga.gov.au/)** to run this notebook interactively from a browser\n", "* **Compatibility**: Notebook currently compatible with both the `NCI` and `DEA Sandbox` environments\n", "* **Prerequisites**: Users of this notebook should have a basic understanding of:\n", " * How to run a [Jupyter notebook](01_Jupyter_notebooks.ipynb)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Background\n", "[Digital Earth Australia](https://www.ga.gov.au/dea) (DEA) is a digital platform that catalogues large amounts of Earth observation data covering continental Australia.\n", "It is underpinned by the [Open Data Cube](https://www.opendatacube.org/) (ODC), an open source software package that has an ever growing number of users, contributors and implementations.\n", "\n", "The ODC and DEA platforms are designed to:\n", "\n", "* Catalogue large amounts of Earth observation data\n", "* Provide a Python based API for high performance querying and data access\n", "* Give users easy ability to perform exploratory data analysis\n", "* Allow scalable continent-scale processing of the stored data\n", "* Track the provenance of data to allow for quality control and updates\n", "\n", "The DEA program catalogues data from a range of satellite sensors and has adopted processes and terminology that users should be aware of to enable efficient querying and use of the datasets stored within.\n", "This notebook introduces these important concepts and forms the basis of understanding for the remainder of the notebooks in this beginner's guide.\n", "Resources to further explore these concepts are recommended at the end of the notebook." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Description\n", "This introduction to DEA will briefly introduce the ODC and review the types of data catalogued in the DEA platform.\n", "It will also cover commonly-used terminology for measurements within product datasets.\n", "Topics covered include:\n", "\n", "* A brief introduction to the ODC\n", "* A review of the satellite sensors that provide data to DEA\n", "* An introduction to analysis ready data and the processes to make it \n", "* DEA's data naming conventions\n", "* Coordinate reference scheme\n", "* Derived products\n", " \n", "***" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Open Data Cube\n", "\n", "![Open Data Cube logo](../Supplementary_data/02_DEA/odc.png)\n", "\n", "The [Open Data Cube](https://www.opendatacube.org/) (ODC) is an open-source software package for organising and analysing large quantities of Earth observation data.\n", "At its core, the Open Data Cube consists of a database where data is stored, along with commands to load, view and analyse that data.\n", "This functionality is delivered by the [datacube-core](https://github.com/opendatacube/datacube-core) open-source Python library.\n", "The library is designed to enable and support:\n", "\n", "* Large-scale workflows on high performance computing infrastructures\n", "* Exploratory data analysis\n", "* Cloud-based services\n", "* Standalone applications\n", "\n", "There are a number of existing implementations of the ODC, including DEA and [Digital Earth Africa](https://www.digitalearthafrica.org/).\n", "More information can be found in the [Open Data Cube Manual](https://datacube-core.readthedocs.io/en/latest/index.html).\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Satellite datasets in DEA\n", "Digital Earth Australia catalogues data from a range of satellite sensors. \n", "The earliest datasets of optical satellite imagery in DEA date from 1986.\n", "DEA includes data from:\n", "\n", "* [Landsat 5 TM](https://www.usgs.gov/land-resources/nli/landsat/landsat-5?qt-science_support_page_related_con=0#qt-science_support_page_related_con) (LS5 TM), operational between March 1984 and January 2013\n", "* [Landsat 7 ETM+](https://www.usgs.gov/land-resources/nli/landsat/landsat-7?qt-science_support_page_related_con=0#qt-science_support_page_related_con) (LS7 ETM+), operational since April 1999\n", "* [Landsat 8 OLI](https://www.usgs.gov/land-resources/nli/landsat/landsat-8?qt-science_support_page_related_con=0#qt-science_support_page_related_con) (LS8 OLI), operational since February 2013\n", "* [Sentinel 2A MSI](https://sentinel.esa.int/web/sentinel/missions/sentinel-2) (S2A MSI), operational since June 2015\n", "* [Sentinel 2B MSI](https://sentinel.esa.int/web/sentinel/missions/sentinel-2) (S2B MSI, operational since March 2017\n", "\n", "Landsat missions are jointly operated by the United States Geological Survey (USGS) and National Aeronautics and Space Administration (NASA).\n", "Sentinel missions are operated by the European Space Agency (ESA).\n", "One major difference between the two programs is the spatial resolution: each Landsat pixel represents 30 x 30 m on the ground while each Sentinel-2 pixel represents 10 x 10 m to 60 x 60 m depending on the spectral band." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Spectral bands\n", "All of the datasets listed above are captured by multispectral satellites.\n", "This means that the satellites measure primarily light that is reflected from the Earth's surface in discrete sections of the electromagnetic spectrum, known as *spectral bands*. \n", "Figure 1 shows the spectral bands for recent Landsat and Sentinel-2 sensors, allowing a direct comparison of how each sensor samples the overall electromagnetic spectrum.\n", "Landsat 5 TM is not displayed in this image; for reference, it measured light in seven bands that covered the same regions as bands 1 to 7 on Landsat 7 ETM+.\n", "\n", "![Image](../Supplementary_data/02_DEA/Landsat9_Auto2E.jpeg)\n", "\n", "> **Figure 1:** The bands that are detected by each of the satellites are shown in the numbered boxes and the width of each box represents the spectral range that band detects.\n", "The bands are overlaid on the percentage transmission of each wavelength returned to the atmosphere from the Earth relative to the amount of incoming solar radiation. \n", "The y-axis has no bearing on the comparison of the satellite sensors [[source]](https://directory.eoportal.org/web/eoportal/satellite-missions/l/landsat-9).\n", "\n", "Figure 1 highlights that the numbering of the bands relative to the detected wavelengths is inconsistent between sensors.\n", "As an example, in the green region of the electromagnetic spectrum (around 560 nm), Landsat 5 TM and Landsat 7 ETM+ detect a wide green region called band 2, where as Landsat 8 OLI detects a slightly narrower region and calls it band 3.\n", "Finally, Sentinel-2 MSI (A and B) detects a narrow green region but also calls it band 3.\n", "Consequently, when working with different sensors, it is important to understand the differences in their bands, and any impact this could have on an analysis.\n", "To promote awareness of these differences, DEA band naming is based on both the spectral band name and sample region.\n", "The naming convention will be covered in more detail in the [DEA band naming conventions section](#DEA-band-naming-conventions)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Analysis Ready Data\n", "\n", "Digital Earth Australia produces Analysis Ready Data (ARD) for each of the sensors listed above.\n", "The [ARD standard](http://ceos.org/ard/) for satellite data requires that data have undergone a number of processing steps, along with the creation of additional attributes for the data.\n", "DEA's ARD datasets include the following characteristics:\n", "\n", "* **Geometric correction:** This includes establishing ground position, accounting for terrain (orthorectification) and ground control points, and assessing absolute position accuracy. \n", "Geometric calibration means that imagery is positioned accurately on the Earth's surface and stacked consistently so that sequential observations can be used to track meaningful change over time.\n", "Adjustments for ground variability typically use a Digital Elevation Model (DEM).\n", "* **Surface reflectance correction:** This includes adjustments for sensor/instrument gains, biases and offsets, include adjustments for terrain illumination and sensor viewing angle with respect to the pixel position on the surface.\n", "Once satellite data is processed to surface reflectance, pixel values from the same sensor can be compared consistently both spatially and over time.\n", "* **Observation attributes:** Per-pixel metadata such as quality flags and content attribution that enable users to make informed decisions about the suitability of the products for their use. For example, clouds, cloud shadows, missing data, saturation and water are common pixel level attributes.\n", "* **Metadata:** Dataset metadata including the satellite, instrument, acquisition date and time, spatial boundaries, pixel locations, mode, processing details, spectral or frequency response and grid projection." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Surface reflectance\n", "\n", "Optical sensors, such as those on the Landsat and Sentinel-2 satellites, measure light that has come from the sun and been reflected by the Earth's surface.\n", "The sensor measures the intensity of light in each of its spectral bands (known as \"radiance\").\n", "The intensity of this light is affected by many factors including the angle of the sun relative to the ground, the angle of the sensor relative to the ground, and how the light interacts with the Earth's atmosphere on its way to the sensor. \n", "Because radiance can be affected by so many factors, it is typically more valuable to determine how much light was originally reflected at the ground level.\n", "This is known as bottom-of-atmosphere **surface reflectance**.\n", "Surface reflectance can be calculated by using robust physical models to correct the observed radiance values based on atmospheric conditions, the angle of the sun, sensor geometry and local topography or terrain.\n", "\n", "There are many approaches to satellite surface reflectance correction and DEA opts to use two: NBAR and NBART.\n", "**Users will choose which of these measurements to load when querying the DEA datacube and so it is important to understand their major similarities and differences.**\n", "\n", "#### NBAR\n", "NBAR stands for *Nadir-corrected BRDF Adjusted Reflectance*, where BRDF stands for *Bidirectional reflectance distribution function*.\n", "The approach involves atmospheric correction to compute bottom-of-atmosphere radiance, and bi-directional reflectance modelling to remove the effects of topography and angular variation in reflectance.\n", "NBAR can be useful for analyses in extremely flat areas not affected by terrain shadow, and for producing attractive data visualisations that are not affected by NBART's nodata gaps (see below).\n", "\n", "#### NBART\n", "NBART has the same features of NBAR but includes an additional *terrain illumination* reflectance correction and as such considered to be actual surface reflectance as it takes into account the surface topography.\n", "Terrain affects optical satellite images in a number of ways; for example, slopes facing the sun receive more sunlight and appear brighter compared to those facing away from the sun.\n", "To obtain comparable surface reflectance from satellite images covering hilly areas, it is therefore necessary to process the images to reduce or remove the topographic effect.\n", "This correction is performed with a Digital Surface Model (DSM) that has been resampled to the same resolution as the satellite data being corrected.\n", "NBART is typically the default choice for most analyses as removing terrain illumination and shadows allows changes in the landscape to be compared more consistently across time. \n", "However, it can be prone to distortions in extremely flat areas if noisy elevation values exist in the DSM.\n", "\n", "![Comparison between NBAR and NBART](../Supplementary_data/02_DEA/nbar_nbart_animation.gif)\n", "\n", "\n", "**Figure 2:** The animation above demonstrates how the NBART correction results in a significantly more two-dimensional looking image that is less affected by terrain illumination and shadow.\n", "Black pixels in the NBART image represent areas of deep terrain shadow that can't be corrected as they're determined not to be viewable by either the sun or the satellite. \n", "These are represented by -999 `nodata` values in the data.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Observation Attributes\n", "\n", "The *Observation Attributes (OA)* are a suite of measurements included in DEA's analysis ready datasets.\n", "They are an assessment of each image pixel to determine if it is an unobscured, unsaturated observation of the Earth's surface, along with whether the pixel is represented in each spectral band. \n", "The OA product allows users to exclude pixels that do not meet the quality criteria for their analysis.\n", "The capacity to automatically exclude such pixels is essential for analysing any change over time, since poor-quality pixels can significantly alter the percieved change over time.\n", "The most common use of OA is for cloud masking, where users can choose to remove images that have too much cloud, or ignore the clouds within each satellite image.\n", "A demonstration of how to use cloud masking can be found in the [masking data](../How_to_guides/Masking_data.ipynb) notebook.\n", "\n", "The OA suite of measurements include the following observation pixel-based attributes:\n", "\n", "* Null pixels\n", "* Clear pixels\n", "* Cloud pixels\n", "* Cloud shadow pixels\n", "* Snow pixels\n", "* Water pixels\n", "* Terrain shaded pixels\n", "* Spectrally contiguous pixels (i.e. whether a pixel contains data in every spectral band)\n", "\n", "Also included is a range of pixel-based attributes related to the satellite, solar and sensing geometries:\n", "\n", "* Solar zenith\n", "* Solar azimuth\n", "* Satellite view\n", "* Incident angle\n", "* Exiting angle\n", "* Azimuthal incident\n", "* Azimuthal exiting\n", "* Relative azimuth\n", "* Timedelta\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Data format\n", "\n", "### DEA band naming conventions\n", "\n", "To account for the various available satellite datasets, DEA uses a band naming convention to help distinguish datasets that come from the different sensors. \n", "The band names are comprised of the applied surface reflectance correction (NBAR or NBART) and the spectral region detected by the satellite. \n", "This removes all reference to the sensor band numbering scheme (e.g. [Figure 1](#Spectral-Bands)) and assumes that users understand that the spectral region described by the DEA band name is only approximately the same between sensors, not identical.\n", "\n", "**Table 1** summarises the DEA band naming terminology for the spectral regions common to both Landsat and Sentinel, coupled with the corresponding NBAR and NBAR band names for the available sensors:\n", "\n", "|Spectral region|DEA measurement name (NBAR)|DEA measurement name (NBAR)|Landsat 5
TM|Landsat 7
ETM+|Landsat 8
OLI|Sentinel-2A,B
MSI|\n", "|----|----|----|----|----|----|----|\n", "|Coastal aerosol|nbar_coastal_aerosol|nbart_coastal_aerosol|||1|1|\n", "|Blue|nbar_blue|nbart_blue|1|1|2|2|\n", "|Green|nbar_green|nbart_green|2|2|3|3|\n", "|Red|nbar_red|nbart_red|3|3|4|4|\n", "|NIR (Near infra-red)|nbar_nir (Landsat)
nbar_nir_1 (Sentinel-2)|nbart_nir (Landsat)
nbart_nir_1 (Sentinel-2)|4|4|5|8|\n", "|SWIR 1 (Short wave infra-red 1)|nbar_swir_1 (Landsat)
nbar_swir_2 (Sentinel-2) |nbart_swir_1 (Landsat)
nbart_swir_2 (Sentinel-2)|5|5|6|11|\n", "|SWIR 2 (Short wave infra-red 2)|nbar_swir_2 (Landsat)
nbar_swir_3 (Sentinel-2) |nbart_swir_2 (Landsat)
nbart_swir_3 (Sentinel-2)|7|7|7|12|\n", "\n", "> **Note:** Be aware that NIR and SWIR band names differ between Landsat and Sentinel-2 due to the different number of these bands available in Sentinel-2. The `nbar_nir` Landsat band corresponds to the spectral region covered by Sentinel-2's `nbar_nir_1` band, the `nbar_swir_1` Landsat band corresponds to Sentinel-2's `nbar_swir_2` band, and the `nbar_swir_2` Landsat band corresponds to Sentinel-2's `nbar_swir_3` band.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### DEA satellite data projection and holdings\n", "Keeping with the practices of the Landsat and Sentinel satellite programs, all DEA satellite datasets are projected using the **Universal Transverse Mercator (UTM)** coordinate reference system.\n", "The World Geodectic System 84 (WGS84) ellipsoid is used to model the UTM projection. All data queries default to the WGS84 datum's coordinate reference system unless specified otherwise.\n", "\n", "By default, the spatial extent of the DEA data holdings is approximately the Australian coastal shelf. \n", "The actual extent varies based on the sensor and product. \n", "The current extents of each DEA product can be viewed using the interactive [DEA Datacube Explorer](https://explorer.dea.ga.gov.au)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Derived products\n", "\n", "![DEA products](../Supplementary_data/02_DEA/dea_products.jpg)\n", "\n", "In addition to ARD satellite data, DEA generates a range of products that are derived from Landsat or Sentinel-2 surface reflectance data.\n", "These products have been developed to characterise and monitor different aspects of Australia's natural and built environment, such as mapping the distribution of water and vegetation across the landscape through time.\n", "Derived DEA products include:\n", "\n", "* **Water Observations from Space (WOfS):** WOfS is the world's first continent-scale map of surface water and provides images and data showing where water has been seen in Australia from 1987 to the present. This map can be used to better understand where water usually occurs across the continent and to plan water management strategies. \n", "\n", "* **Fractional Cover (FC):** Fractional Cover (FC) is a measurement that splits the landscape into three parts, or fractions; green (leaves, grass, and growing crops), brown (branches, dry grass or hay, and dead leaf litter), and bare ground (soil or rock). DEA uses Fractional Cover to characterise every 25 m square of Australia for any point in time from 1987 to today. This measurement can inform a broad range of natural resource management issues. \n", "\n", "* **High and Low Tide Composites (HLTC):** The High and Low Tide Composites (HLTC) are imagery mosaics developed to visualise Australia's coasts, estuaries and reefs at low and high tide, whilst removing the influence of noise features such as clouds, breaking water and sun-glint. These products are highly interpretable, and provide a valuable snapshot of the coastline at different biophysical states.\n", "\n", "* **Intertidal Extents Model (ITEM):** The Intertidal Extents Model (ITEM) product utilises 30 years of Earth observation data from the Landsat archive to map the extents and topography of Australia's intertidal mudflats, beaches and reefs; the area exposed between high and low tide.\n", "\n", "* **National Intertidal Digital Elevation Model (NIDEM):** The National Intertidal Digital Elevation Model (NIDEM) is a national dataset that maps the three-dimensional structure of Australia’s intertidal zone. NIDEM provides a first-of-its kind source of intertidal elevation data for Australia’s entire coastline. \n", "\n", "Each of the products above have dataset-specific naming conventions, measurements, resolutions, data types and coordinate reference systems.\n", "For more information about DEA's derived products, refer to the [DEA website](http://www.ga.gov.au/dea/products), the [Content Management Interface](https://knowledge.dea.ga.gov.au/) (CMI) containing detailed product metadata, or the \"DEA_products\" notebooks in this repository." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Recommended next steps\n", "For more detailed information on the concepts introduced in this notebook, please see the [DEA User Guide](https://knowledge.dea.ga.gov.au/) and [Open Data Cube Manual](https://datacube-core.readthedocs.io/en/latest/).\n", "For more information on the development of the DEA platform, please see [Dhu et al. 2017](https://doi.org/10.1080/20964471.2017.1402490).\n", "\n", "To continue with the beginner's guide, the following notebooks are designed to be worked through in the following order:\n", "\n", "1. [Jupyter Notebooks](01_Jupyter_notebooks.ipynb)\n", "2. **Digital Earth Australia (this notebook)**\n", "3. [Products and Measurements](03_Products_and_measurements.ipynb)\n", "4. [Loading data](04_Loading_data.ipynb)\n", "5. [Plotting](05_Plotting.ipynb)\n", "6. [Performing a basic analysis](06_Basic_analysis.ipynb)\n", "7. [Introduction to Numpy](07_Intro_to_numpy.ipynb)\n", "8. [Introduction to Xarray](08_Intro_to_xarray.ipynb)\n", "9. [Parallel processing with Dask](09_Parallel_processing_with_Dask.ipynb)\n", "\n", "Once you have worked through the beginner's guide, you can join advanced users by exploring:\n", "\n", "* The demonstration of how to use cloud masking in the [masking data](../How_to_guides/Masking_data.ipynb) notebook.\n", "* The \"DEA_products\" directory in the repository, where you can explore DEA products in depth.\n", "* The \"How_to_guides\" directory, which contains a recipe book of common techniques and methods for analysing DEA data.\n", "* The \"Real_world_examples\" directory, which provides more complex workflows and analysis case studies." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "***\n", "\n", "## Additional information\n", "\n", "**License:** The code in this notebook is licensed under the [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0). \n", "Digital Earth Australia data is licensed under the [Creative Commons by Attribution 4.0](https://creativecommons.org/licenses/by/4.0/) license.\n", "\n", "**Contact:** If you need assistance, please post a question on the [Open Data Cube Slack channel](http://slack.opendatacube.org/) or on the [GIS Stack Exchange](https://gis.stackexchange.com/questions/ask?tags=open-data-cube) using the `open-data-cube` tag (you can view previously asked questions [here](https://gis.stackexchange.com/questions/tagged/open-data-cube)).\n", "If you would like to report an issue with this notebook, you can file one on [GitHub](https://github.com/GeoscienceAustralia/dea-notebooks).\n", "\n", "**Last modified:** December 2023" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Tags\n", "" ] }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ "**Tags**: :index:`sandbox compatible`, :index:`NCI compatible`, :index:`sensors`, :index:`band names`, :index:`NBAR`, :index:`NBAR`, :index:`observation attributes`, :index:`naming conventions`" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.13" }, "widgets": { "application/vnd.jupyter.widget-state+json": { "state": {}, "version_major": 2, "version_minor": 0 } } }, "nbformat": 4, "nbformat_minor": 4 }