Introduction to products and measurements¶
Compatability: Notebook currently compatible with both the
NCIandDEA SandboxenvironmentsProducts used: ga_ls5t_ard_3
Prerequisites: Users of this notebook should have a basic understanding of:
How to run a Jupyter notebook
The basic structure of the DEA satellite datasets
Background¶
A “datacube” is a digital information architecture that specialises in hosting and cataloguing spatial information. Digital Earth Australia (DEA) is based on the Open Data Cube infrastructure, and specialises in storing remotely sensed data, particularly from Earth Observation satellites such as Landsat and Sentinel-2.
The DEA datacube contains both raw satellite data and derivative data “products”. These data products are often composed of a range of “measurements” such as the suite of remote sensing band values or statistical product summaries. Before running a query to load data from the datacube, it is useful to know what it contains. This notebook demonstrates several straightforward ways to inspect the product and measurement contents of a datacube.
Description¶
This notebook demonstrates how to connect to a datacube and interrogate the available products and measurements stored within. Topics covered include:
How to connect to a datacube
How to list all the products
How to list all the product measurements
How to interactively visualise data in the datacube
Getting started¶
To run this introduction to products and measurements, run all the cells in the notebook starting with the “Load packages” cell. For help with running notebook cells, refer back to the Jupyter Notebooks notebook.
Load packages¶
The datacube package is required to access and work with available data. The pandas package is required to format tables. The DcViewer utility provides an interface for interactively exploring the products available in the datacube.
[1]:
import datacube
import pandas as pd
from odc.ui import DcViewer
# Set some configurations for displaying tables nicely
pd.set_option('display.max_colwidth', 200)
pd.set_option('display.max_rows', None)
Connect to the datacube¶
After importing the datacube package, users need to specify a name for their session, known as the app name.
This name is generated by the user and is used to track down issues with database queries. It does not have any effect on the analysis. Use a short name that is consistent with the purpose of your notebook such as the way 03_Products_and_measurements has been used as the app name in this notebook.
The resulting dc object provides access to all the data contained within the Digital Earth Australia datacube.
[2]:
# Temporary solution to account for Collection 3 data being in a different
# database on the NCI
try:
dc = datacube.Datacube(app="03_Products_and_measurements", env="c3-samples")
except:
dc = datacube.Datacube(app="03_Products_and_measurements")
List products¶
Once a datacube instance has been created, users can explore the products and measurements stored within.
The following cell lists all products that are currently available in the DEA datacube by using the dc.list_products() function.
Products listed under name in the following table represent the product options available when querying the datacube. The table below provides some useful information about each product, including a brief product description, the instrument and platform the data originated from (e.g. Landsat 8 OLI), and the product’s default crs (coordinate reference system) and resolution if applicable.
For a comprehensive product description and access to complete product metadata, users are directed to the Geoscience Australia Content Management Interface
[3]:
products = dc.list_products()
display_columns = ["name",
"description",
"platform",
"instrument",
"crs",
"resolution"]
products[display_columns].sort_index()
[3]:
| name | description | platform | instrument | crs | resolution | |
|---|---|---|---|---|---|---|
| id | ||||||
| 1 | ls8_nbart_geomedian_annual | Surface Reflectance Geometric Median 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) | LANDSAT_8 | OLI | EPSG:3577 | (-25, 25) |
| 2 | ls7_nbart_geomedian_annual | Surface Reflectance Geometric Median 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) | LANDSAT_7 | ETM | EPSG:3577 | (-25, 25) |
| 3 | ls5_nbart_geomedian_annual | Surface Reflectance Geometric Median 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) | LANDSAT_5 | TM | EPSG:3577 | (-25, 25) |
| 4 | ls5_fc_albers | Landsat 5 Fractional Cover 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) | LANDSAT_5 | TM | EPSG:3577 | (-25, 25) |
| 5 | ls8_fc_albers | Landsat 8 Fractional Cover 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) | LANDSAT_8 | OLI_TIRS | EPSG:3577 | (-25, 25) |
| 6 | high_tide_comp_20p | High tide 20 percentage composites 25m v. 2.0.0 | None | None | EPSG:3577 | (-25, 25) |
| 7 | low_tide_comp_20p | Low tide 20 percentage composites 25m v. 2.0.0 | None | None | EPSG:3577 | (-25, 25) |
| 8 | ls8_barest_earth_albers | Landsat-8 Barest Earth pixel composite albers 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) | LANDSAT_8 | OLI | EPSG:3577 | (-25, 25) |
| 9 | landsat_barest_earth | Landsat-5/Landsat-7/Landsat-8 combined Barest Earth pixel composite albers 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) | LANDSAT_5,LANDSAT_7,LANDSAT_8 | TM,ETM+,OLI | EPSG:3577 | (-25, 25) |
| 10 | fc_percentile_albers_annual | Landsat Fractional Cover percentile 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) | LANDSAT_5,LANDSAT_7,LANDSAT_8 | TM,ETM+,OLI | EPSG:3577 | (-25, 25) |
| 11 | fc_percentile_albers_seasonal | Landsat Fractional Cover percentile 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) | LANDSAT_5,LANDSAT_7,LANDSAT_8 | TM,ETM+,OLI | EPSG:3577 | (-25, 25) |
| 12 | mangrove_cover | Mangrove Cover, Australian Albers Equal Area projection (EPSG:3577) | LANDSAT_5,LANDSAT_7,LANDSAT_8 | TM,ETM+,OLI | EPSG:3577 | (-25, 25) |
| 13 | nidem | National Intertidal Digital Elevation Model 25m 1.0.0 | None | None | EPSG:3577 | (-25, 25) |
| 14 | s2a_l1c_aws_pds | Sentinel-2A MSI L1C - AWS PDS | SENTINEL_2A | MSI | NaN | NaN |
| 15 | s2b_l1c_aws_pds | Sentinel-2B MSI L1C - AWS PDS | SENTINEL_2B | MSI | NaN | NaN |
| 16 | s2a_nrt_granule | Sentinel-2A MSI NRT - NBAR NBART and Pixel Quality | SENTINEL_2A | MSI | NaN | NaN |
| 17 | s2b_nrt_granule | Sentinel-2B MSI NRT - NBAR NBART and Pixel Quality | SENTINEL_2B | MSI | NaN | NaN |
| 20 | sentinel2_wofs_nrt | Sentinel 2 NRT Water Observations from Space | None | MSI | NaN | NaN |
| 21 | wofs_albers | Historic Flood Mapping Water Observations from Space | None | None | EPSG:3577 | (-25, 25) |
| 22 | wofs_filtered_summary | Water Observations from Space Statistics confidence filtered | LANDSAT_5,LANDSAT_7,LANDSAT_8 | TM,ETM+,OLI | EPSG:3577 | (-25, 25) |
| 23 | wofs_summary | Water Observations from Space Statistics | LANDSAT_5,LANDSAT_7,LANDSAT_8 | TM,ETM+,OLI | EPSG:3577 | (-25, 25) |
| 24 | wofs_annual_summary | Water Observations from Space Annual Statistics | LANDSAT_5,LANDSAT_7,LANDSAT_8 | TM,ETM,OLI | EPSG:3577 | (-25, 25) |
| 25 | wofs_apr_oct_summary | Water Observations from Space April to October Statistics | LANDSAT_5,LANDSAT_7,LANDSAT_8 | TM,ETM,OLI | EPSG:3577 | (-25, 25) |
| 26 | wofs_nov_mar_summary | Water Observations from Space November to March Statistics | LANDSAT_5,LANDSAT_7,LANDSAT_8 | TM,ETM,OLI | EPSG:3577 | (-25, 25) |
| 27 | item_v2 | Relative Extents Model | None | None | EPSG:3577 | (-25, 25) |
| 28 | item_v2_conf | Average ndwi Standard Deviation, the Confidence Layer | None | None | EPSG:3577 | (-25, 25) |
| 29 | weathering_intensity | Weathering Intensity Model | unknown | STRM | EPSG:4326 | (-0.000833333333347, 0.000833333333347) |
| 30 | multi_scale_topographic_position | Multi-scale Topographic Position Image | None | None | EPSG:4326 | (-0.000833333333347, 0.000833333333347) |
| 36 | s2a_level1c_granule | Sentinel-2A Level1C - Ortho Rectified Top of Atmosphere Reflectance | Sentinel-2A | MSI | NaN | NaN |
| 37 | s2b_level1c_granule | Sentinel-2B Level1C - Ortho Rectified Top of Atmosphere Reflectance | Sentinel-2B | MSI | NaN | NaN |
| 38 | s2a_ard_granule | Sentinel-2A MSI ARD - NBAR NBART and Pixel Quality | SENTINEL_2A | MSI | NaN | NaN |
| 39 | s2b_ard_granule | Sentinel-2B MSI ARD - NBAR NBART and Pixel Quality | SENTINEL_2B | MSI | NaN | NaN |
| 40 | s1_gamma0_geotif_scene | Sentinel-1A/B SAR Gamma0 scenes, processed to the CEOS ARD standard - Orbit updates, GRD border noise, thermal noise, radiometric calibration and terrain correction, orthorectification. | SENTINEL_1 | SAR | NaN | NaN |
| 41 | ls7_fc_albers | Landsat 7 Fractional Cover 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) | LANDSAT_7 | ETM | EPSG:3577 | (-25, 25) |
| 46 | ga_s2a_ard_nbar_granule | Sentinel-2A MSI Definitive ARD - NBAR and Pixel Quality | SENTINEL_2A | MSI | NaN | NaN |
| 47 | ga_s2b_ard_nbar_granule | Sentinel-2B MSI Definitive ARD - NBAR and Pixel Quality | SENTINEL_2B | MSI | NaN | NaN |
| 48 | ls5_level1_usgs | Landsat 5 USGS Level 1 Collection-1 OLI-TIRS | LANDSAT_5 | TM | NaN | NaN |
| 49 | ls7_level1_usgs | Landsat 7 USGS Level 1 Collection-1 OLI-TIRS | LANDSAT_7 | ETM | NaN | NaN |
| 50 | ls8_level1_usgs | Landsat 8 USGS Level 1 Collection-1 OLI-TIRS | None | None | NaN | NaN |
| 51 | ls5_ard | Landsat 5 ARD datasets for the GA/USGS interoperability study | LANDSAT_5 | TM | NaN | NaN |
| 52 | ls7_ard | Landsat 7 ARD datasets for the GA/USGS interoperability study | LANDSAT_7 | ETM+ | NaN | NaN |
| 53 | ls8_ard | Landsat 8 ARD datasets | LANDSAT_8 | OLI | NaN | NaN |
| 54 | ls5_usgs_l2c1 | Landsat 5 Thematic Mapper (TM) USGS Analysis Ready Data 30m scene | LANDSAT_5 | TM | NaN | NaN |
| 55 | ls7_usgs_l2c1 | Landsat 7 Enhanced Thematic Mapper Plus (ETM+) USGS ARD 30 metre tile | LANDSAT_7 | ETM | NaN | NaN |
| 56 | ls8_usgs_l2c1 | Landsat 8 Operational Land Imager (OLI) and Thermal Infrared Sensor (TIRS) USGS Analysis Ready Data 30m scene | LANDSAT_8 | OLI_TIRS | NaN | NaN |
| 57 | ls8_nbart_tmad_annual | Surface Reflectance Triple Median Absolute Deviation 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) | LANDSAT_8 | OLI | EPSG:3577 | (-25, 25) |
| 58 | ls7_nbart_tmad_annual | Surface Reflectance Triple Median Absolute Deviation 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) | LANDSAT_7 | ETM+ | EPSG:3577 | (-25, 25) |
| 59 | ls5_nbart_tmad_annual | Surface Reflectance Triple Median Absolute Deviation 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) | LANDSAT_5 | TM | EPSG:3577 | (-25, 25) |
| 60 | ls5_pq_albers | Landsat 5 Pixel Quality 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) | LANDSAT_5 | TM | EPSG:3577 | (-25, 25) |
| 61 | ls7_pq_albers | Landsat 7 Pixel Quality 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) | LANDSAT_7 | ETM | EPSG:3577 | (-25, 25) |
| 62 | ls8_pq_albers | Landsat 8 Pixel Quality 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) | LANDSAT_8 | OLI_TIRS | EPSG:3577 | (-25, 25) |
| 66 | usgs_ls5t_level1_1 | Landsat 5 TM Level 1, USGS Collection 1 | None | None | NaN | NaN |
| 67 | usgs_ls7e_level1_1 | Landsat 7 ETM+ Level 1, USGS Collection 1 | None | None | NaN | NaN |
| 68 | usgs_ls8c_level1_1 | Landsat 8 OLI-TIRS Level 1, USGS Collection 1 | None | None | NaN | NaN |
| 70 | ga_ls5t_ard_3 | Landsat 5 TM ARD, GA Collection 3 | None | None | NaN | NaN |
| 71 | ga_ls7e_ard_3 | Landsat 7 ETM+ ARD, GA Collection 3 | None | None | NaN | NaN |
| 72 | ga_ls8c_ard_3 | Landsat 8 OLI-TIRS ARD, GA Collection 3 | None | None | NaN | NaN |
| 73 | ga_s2am_fractional_cover_2 | Sentinel-2A Fractional Cover in original UTM projections. | None | None | NaN | NaN |
| 74 | ga_s2bm_fractional_cover_2 | Sentinel-2B Fractional Cover in original UTM projections. | None | None | NaN | NaN |
| 75 | camden_insar_alos_displacement | Camden InSAR ALOS Displacement | None | None | NaN | NaN |
| 78 | s2_tsmask | Time series cloud and cloud shadow detection for Sentinel-2A and Sentinel-2B surface reflectance data.\n\nTSmask classifies a pixel as one of the following four categories: no observation, clear, ... | SENTINEL_2A,SENTINEL_2B | MSI | NaN | NaN |
List measurements¶
Most products are associated with a range of available measurements. These can be individual satellite bands (e.g. Landsat’s near-infrared band) or statistical product summaries.
The dc.list_measurements() function can be used to interrogate the measurements associated with a given product (specified by the name column from the table above). For example, ga_ls5t_ard_3 refers to the Geoscience Australia Landsat 5 Analysis-ready data Collection 3 product.
The table below includes a range of technical information about each band in the ga_ls5t_ard_3 dataset, including any aliases which can be used to load the data, the data type or dtype, any flags_definition that are associated with the measurement (this information is used for tasks like cloud masking), and the measurement’s nodata value.
Change the product name below and re-run the following cell to explore available measurements associated with other products.
[4]:
product = "ga_ls5t_ard_3"
measurements = dc.list_measurements()
measurements.loc[product]
[4]:
| aliases | dtype | flags_definition | name | nodata | spectral_definition | units | |
|---|---|---|---|---|---|---|---|
| measurement | |||||||
| nbar_blue | [nbar_band01] | int16 | NaN | nbar_blue | -999 | NaN | 1 |
| nbar_green | [nbar_band02] | int16 | NaN | nbar_green | -999 | NaN | 1 |
| nbar_red | [nbar_band03] | int16 | NaN | nbar_red | -999 | NaN | 1 |
| nbar_nir | [nbar_band04] | int16 | NaN | nbar_nir | -999 | NaN | 1 |
| nbar_swir_1 | [nbar_band05] | int16 | NaN | nbar_swir_1 | -999 | NaN | 1 |
| nbar_swir_2 | [nbar_band07] | int16 | NaN | nbar_swir_2 | -999 | NaN | 1 |
| nbart_blue | [nbart_band01, blue] | int16 | NaN | nbart_blue | -999 | NaN | 1 |
| nbart_green | [nbart_band02, green] | int16 | NaN | nbart_green | -999 | NaN | 1 |
| nbart_red | [nbart_band03, red] | int16 | NaN | nbart_red | -999 | NaN | 1 |
| nbart_nir | [nbart_band04, nir] | int16 | NaN | nbart_nir | -999 | NaN | 1 |
| nbart_swir_1 | [nbart_band05, swir_1, swir1] | int16 | NaN | nbart_swir_1 | -999 | NaN | 1 |
| nbart_swir_2 | [nbart_band07, swir_2, swir2] | int16 | NaN | nbart_swir_2 | -999 | NaN | 1 |
| oa_fmask | [fmask] | uint8 | {'fmask': {'bits': [0, 1, 2, 3, 4, 5, 6, 7], 'values': {'0': 'nodata', '1': 'valid', '2': 'cloud', '3': 'shadow', '4': 'snow', '5': 'water'}, 'description': 'Fmask'}} | oa_fmask | 0 | NaN | 1 |
| oa_nbar_contiguity | [nbar_contiguity] | uint8 | {'contiguous': {'bits': [0], 'values': {'0': False, '1': True}}} | oa_nbar_contiguity | 255 | NaN | 1 |
| oa_nbart_contiguity | [nbart_contiguity] | uint8 | {'contiguous': {'bits': [0], 'values': {'0': False, '1': True}}} | oa_nbart_contiguity | 255 | NaN | 1 |
| oa_azimuthal_exiting | [azimuthal_exiting] | float32 | NaN | oa_azimuthal_exiting | NaN | NaN | 1 |
| oa_azimuthal_incident | [azimuthal_incident] | float32 | NaN | oa_azimuthal_incident | NaN | NaN | 1 |
| oa_combined_terrain_shadow | [combined_terrain_shadow] | uint8 | NaN | oa_combined_terrain_shadow | 255 | NaN | 1 |
| oa_exiting_angle | [exiting_angle] | float32 | NaN | oa_exiting_angle | NaN | NaN | 1 |
| oa_incident_angle | [incident_angle] | float32 | NaN | oa_incident_angle | NaN | NaN | 1 |
| oa_relative_azimuth | [relative_azimuth] | float32 | NaN | oa_relative_azimuth | NaN | NaN | 1 |
| oa_relative_slope | [relative_slope] | float32 | NaN | oa_relative_slope | NaN | NaN | 1 |
| oa_satellite_azimuth | [satellite_azimuth] | float32 | NaN | oa_satellite_azimuth | NaN | NaN | 1 |
| oa_satellite_view | [satellite_view] | float32 | NaN | oa_satellite_view | NaN | NaN | 1 |
| oa_solar_azimuth | [solar_azimuth] | float32 | NaN | oa_solar_azimuth | NaN | NaN | 1 |
| oa_solar_zenith | [solar_zenith] | float32 | NaN | oa_solar_zenith | NaN | NaN | 1 |
| oa_time_delta | [time_delta] | float32 | NaN | oa_time_delta | NaN | NaN | 1 |
Visualising available data¶
The interactive DcViewer utility provides a more visual way of exploring the data that is available within the Digital Earth Australia datacube.
After running the cell below, select a product from the drop-down menu on the top-right of the map to show the areas where data are available in blue. Use the back and forward buttons above the map to toggle through time.
The utility is only able to visualise a limited number of datasets at one time. If the available data footprints do not appear, either press the “show” button on the top right, or zoom in on the map.
[5]:
DcViewer(dc=dc,
time='2015',
width='800px',
center=(-27.48, 153.10),
zoom=7)
DEA Explorer sites¶
Another way to view the available data within a datacube is to visit the DEA Datacube Explorer sites. These webpages visualise the data that is available for every product in DEA.
There is a different Datacube Explorer page for both the NCI and DEA Sandbox environments to account for the different datasets that are available in each of these datacubes:
Recommended next steps¶
To continue working through the notebooks in this beginner’s guide, the following notebooks are designed to be worked through in the following order:
Products and Measurements (this notebook)
Once you have worked through the beginner’s guide, you can join advanced users by exploring:
The “DEA datasets” directory in the repository, where you can explore DEA products in depth.
The “Frequently used code” directory, which contains a recipe book of common techniques and methods for analysing DEA data.
The “Real-world examples” directory, which provides more complex workflows and analysis case studies.
Additional information¶
License: The code in this notebook is licensed under the Apache License, Version 2.0. Digital Earth Australia data is licensed under the Creative Commons by Attribution 4.0 license.
Contact: If you need assistance, please post a question on the Open Data Cube Slack channel or on the GIS Stack Exchange using the open-data-cube tag (you can view previously asked questions here). If you would like to report an issue with this notebook, you can file one on
Github.
Last modified: December 2019
Compatible datacube version:
[6]:
print(datacube.__version__)
1.7+164.gbdf45994.dirty
Tags¶
Browse all available tags on the DEA User Guide’s Tags Index
Tags: sandbox compatible, NCI compatible, dc.list_products, dc.list_measurements, products, measurements, landsat 5, datacube explorer, DcViewer