Introduction to products and measurements

  • Compatability: Notebook currently compatible with both the NCI and DEA Sandbox environments

  • Products used: ga_ls5t_ard_3

  • Prerequisites: Users of this notebook should have a basic understanding of:

Background

A “datacube” is a digital information architecture that specialises in hosting and cataloguing spatial information. Digital Earth Australia (DEA) is based on the Open Data Cube infrastructure, and specialises in storing remotely sensed data, particularly from Earth Observation satellites such as Landsat and Sentinel-2.

The DEA datacube contains both raw satellite data and derivative data “products”. These data products are often composed of a range of “measurements” such as the suite of remote sensing band values or statistical product summaries. Before running a query to load data from the datacube, it is useful to know what it contains. This notebook demonstrates several straightforward ways to inspect the product and measurement contents of a datacube.

Description

This notebook demonstrates how to connect to a datacube and interrogate the available products and measurements stored within. Topics covered include:

  • How to connect to a datacube

  • How to list all the products

  • How to list all the product measurements

  • How to interactively visualise data in the datacube


Getting started

To run this introduction to products and measurements, run all the cells in the notebook starting with the “Load packages” cell. For help with running notebook cells, refer back to the Jupyter Notebooks notebook.

Load packages

The datacube package is required to access and work with available data. The pandas package is required to format tables. The DcViewer utility provides an interface for interactively exploring the products available in the datacube.

[2]:
import datacube
import pandas as pd
from odc.ui import DcViewer

# Set some configurations for displaying tables nicely
pd.set_option('display.max_colwidth', 200)
pd.set_option('display.max_rows', None)

Connect to the datacube

After importing the datacube package, users need to specify a name for their session, known as the app name.

This name is generated by the user and is used to track down issues with database queries. It does not have any effect on the analysis. Use a short name that is consistent with the purpose of your notebook such as the way 03_Products_and_measurements has been used as the app name in this notebook.

The resulting dc object provides access to all the data contained within the Digital Earth Australia datacube.

[3]:
# Temporary solution to account for Collection 3 data being in a different
# database on the NCI
try:
    dc = datacube.Datacube(app="03_Products_and_measurements", env="c3-samples")
except:
    dc = datacube.Datacube(app="03_Products_and_measurements")

List products

Once a datacube instance has been created, users can explore the products and measurements stored within.

The following cell lists all products that are currently available in the DEA datacube by using the dc.list_products() function.

Products listed under name in the following table represent the product options available when querying the datacube. The table below provides some useful information about each product, including a brief product description, the instrument and platform the data originated from (e.g. Landsat 8 OLI), and the product’s default crs (coordinate reference system) and resolution if applicable.

For a comprehensive product description and access to complete product metadata, users are directed to the Geoscience Australia Content Management Interface

[4]:
products = dc.list_products()

display_columns = ["name",
                   "description",
                   "platform",
                   "instrument",
                   "crs",
                   "resolution"]

products[display_columns].sort_index()
[4]:
name description platform instrument crs resolution
id
1 ls8_nbart_geomedian_annual Surface Reflectance Geometric Median 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_8 OLI EPSG:3577 (-25, 25)
2 ls7_nbart_geomedian_annual Surface Reflectance Geometric Median 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_7 ETM EPSG:3577 (-25, 25)
3 ls5_nbart_geomedian_annual Surface Reflectance Geometric Median 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_5 TM EPSG:3577 (-25, 25)
4 ls5_fc_albers Landsat 5 Fractional Cover 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_5 TM EPSG:3577 (-25, 25)
5 ls8_fc_albers Landsat 8 Fractional Cover 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_8 OLI_TIRS EPSG:3577 (-25, 25)
6 high_tide_comp_20p High tide 20 percentage composites 25m v. 2.0.0 None None EPSG:3577 (-25, 25)
7 low_tide_comp_20p Low tide 20 percentage composites 25m v. 2.0.0 None None EPSG:3577 (-25, 25)
8 ls8_barest_earth_albers Landsat-8 Barest Earth pixel composite albers 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_8 OLI EPSG:3577 (-25, 25)
9 landsat_barest_earth Landsat-5/Landsat-7/Landsat-8 combined Barest Earth pixel composite albers 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_5,LANDSAT_7,LANDSAT_8 TM,ETM+,OLI EPSG:3577 (-25, 25)
10 fc_percentile_albers_annual Landsat Fractional Cover percentile 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_5,LANDSAT_7,LANDSAT_8 TM,ETM+,OLI EPSG:3577 (-25, 25)
11 fc_percentile_albers_seasonal Landsat Fractional Cover percentile 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_5,LANDSAT_7,LANDSAT_8 TM,ETM+,OLI EPSG:3577 (-25, 25)
12 mangrove_cover Mangrove Cover, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_5,LANDSAT_7,LANDSAT_8 TM,ETM+,OLI EPSG:3577 (-25, 25)
13 nidem National Intertidal Digital Elevation Model 25m 1.0.0 None None EPSG:3577 (-25, 25)
14 s2a_l1c_aws_pds Sentinel-2A MSI L1C - AWS PDS SENTINEL_2A MSI NaN NaN
15 s2b_l1c_aws_pds Sentinel-2B MSI L1C - AWS PDS SENTINEL_2B MSI NaN NaN
16 s2a_nrt_granule Sentinel-2A MSI NRT - NBAR NBART and Pixel Quality SENTINEL_2A MSI NaN NaN
17 s2b_nrt_granule Sentinel-2B MSI NRT - NBAR NBART and Pixel Quality SENTINEL_2B MSI NaN NaN
20 sentinel2_wofs_nrt Sentinel 2 NRT Water Observations from Space None MSI NaN NaN
21 wofs_albers Historic Flood Mapping Water Observations from Space None None EPSG:3577 (-25, 25)
22 wofs_filtered_summary Water Observations from Space Statistics confidence filtered LANDSAT_5,LANDSAT_7,LANDSAT_8 TM,ETM+,OLI EPSG:3577 (-25, 25)
23 wofs_summary Water Observations from Space Statistics LANDSAT_5,LANDSAT_7,LANDSAT_8 TM,ETM+,OLI EPSG:3577 (-25, 25)
24 wofs_annual_summary Water Observations from Space Annual Statistics LANDSAT_5,LANDSAT_7,LANDSAT_8 TM,ETM,OLI EPSG:3577 (-25, 25)
25 wofs_apr_oct_summary Water Observations from Space April to October Statistics LANDSAT_5,LANDSAT_7,LANDSAT_8 TM,ETM,OLI EPSG:3577 (-25, 25)
26 wofs_nov_mar_summary Water Observations from Space November to March Statistics LANDSAT_5,LANDSAT_7,LANDSAT_8 TM,ETM,OLI EPSG:3577 (-25, 25)
27 item_v2 Relative Extents Model None None EPSG:3577 (-25, 25)
28 item_v2_conf Average ndwi Standard Deviation, the Confidence Layer None None EPSG:3577 (-25, 25)
29 weathering_intensity Weathering Intensity Model unknown STRM EPSG:4326 (-0.000833333333347, 0.000833333333347)
30 multi_scale_topographic_position Multi-scale Topographic Position Image None None EPSG:4326 (-0.000833333333347, 0.000833333333347)
36 s2a_level1c_granule Sentinel-2A Level1C - Ortho Rectified Top of Atmosphere Reflectance Sentinel-2A MSI NaN NaN
37 s2b_level1c_granule Sentinel-2B Level1C - Ortho Rectified Top of Atmosphere Reflectance Sentinel-2B MSI NaN NaN
38 s2a_ard_granule Sentinel-2A MSI ARD - NBAR NBART and Pixel Quality SENTINEL_2A MSI NaN NaN
39 s2b_ard_granule Sentinel-2B MSI ARD - NBAR NBART and Pixel Quality SENTINEL_2B MSI NaN NaN
40 s1_gamma0_geotif_scene Sentinel-1A/B SAR Gamma0 scenes, processed to the CEOS ARD standard - Orbit updates, GRD border noise, thermal noise, radiometric calibration and terrain correction, orthorectification. SENTINEL_1 SAR NaN NaN
41 ls7_fc_albers Landsat 7 Fractional Cover 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_7 ETM EPSG:3577 (-25, 25)
46 ga_s2a_ard_nbar_granule Sentinel-2A MSI Definitive ARD - NBAR and Pixel Quality SENTINEL_2A MSI NaN NaN
47 ga_s2b_ard_nbar_granule Sentinel-2B MSI Definitive ARD - NBAR and Pixel Quality SENTINEL_2B MSI NaN NaN
48 ls5_level1_usgs Landsat 5 USGS Level 1 Collection-1 OLI-TIRS LANDSAT_5 TM NaN NaN
49 ls7_level1_usgs Landsat 7 USGS Level 1 Collection-1 OLI-TIRS LANDSAT_7 ETM NaN NaN
50 ls8_level1_usgs Landsat 8 USGS Level 1 Collection-1 OLI-TIRS None None NaN NaN
51 ls5_ard Landsat 5 ARD datasets for the GA/USGS interoperability study LANDSAT_5 TM NaN NaN
52 ls7_ard Landsat 7 ARD datasets for the GA/USGS interoperability study LANDSAT_7 ETM+ NaN NaN
53 ls8_ard Landsat 8 ARD datasets LANDSAT_8 OLI NaN NaN
54 ls5_usgs_l2c1 Landsat 5 Thematic Mapper (TM) USGS Analysis Ready Data 30m scene LANDSAT_5 TM NaN NaN
55 ls7_usgs_l2c1 Landsat 7 Enhanced Thematic Mapper Plus (ETM+) USGS ARD 30 metre tile LANDSAT_7 ETM NaN NaN
56 ls8_usgs_l2c1 Landsat 8 Operational Land Imager (OLI) and Thermal Infrared Sensor (TIRS) USGS Analysis Ready Data 30m scene LANDSAT_8 OLI_TIRS NaN NaN
57 ls8_nbart_tmad_annual Surface Reflectance Triple Median Absolute Deviation 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_8 OLI EPSG:3577 (-25, 25)
58 ls7_nbart_tmad_annual Surface Reflectance Triple Median Absolute Deviation 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_7 ETM+ EPSG:3577 (-25, 25)
59 ls5_nbart_tmad_annual Surface Reflectance Triple Median Absolute Deviation 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_5 TM EPSG:3577 (-25, 25)
60 ls5_pq_albers Landsat 5 Pixel Quality 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_5 TM EPSG:3577 (-25, 25)
61 ls7_pq_albers Landsat 7 Pixel Quality 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_7 ETM EPSG:3577 (-25, 25)
62 ls8_pq_albers Landsat 8 Pixel Quality 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_8 OLI_TIRS EPSG:3577 (-25, 25)
66 usgs_ls5t_level1_1 Landsat 5 TM Level 1, USGS Collection 1 None None NaN NaN
67 usgs_ls7e_level1_1 Landsat 7 ETM+ Level 1, USGS Collection 1 None None NaN NaN
68 usgs_ls8c_level1_1 Landsat 8 OLI-TIRS Level 1, USGS Collection 1 None None NaN NaN
70 ga_ls5t_ard_3 Landsat 5 TM ARD, GA Collection 3 None None NaN NaN
71 ga_ls7e_ard_3 Landsat 7 ETM+ ARD, GA Collection 3 None None NaN NaN
72 ga_ls8c_ard_3 Landsat 8 OLI-TIRS ARD, GA Collection 3 None None NaN NaN
73 ga_s2am_fractional_cover_2 Sentinel-2A Fractional Cover in original UTM projections. None None NaN NaN
74 ga_s2bm_fractional_cover_2 Sentinel-2B Fractional Cover in original UTM projections. None None NaN NaN
75 camden_insar_alos_displacement Camden InSAR ALOS Displacement None None NaN NaN
78 s2_tsmask Time series cloud and cloud shadow detection for Sentinel-2A and Sentinel-2B surface reflectance data.\n\nTSmask classifies a pixel as one of the following four categories: no observation, clear, ... SENTINEL_2A,SENTINEL_2B MSI NaN NaN
79 s2_gm_seasonal Sentinel-2A/B MSI ARD - NBAR Geometric Median None None NaN NaN
81 sentinel2_wofs_def Sentinel 2 Definitive Water Observations from Space None MSI NaN NaN

List measurements

Most products are associated with a range of available measurements. These can be individual satellite bands (e.g. Landsat’s near-infrared band) or statistical product summaries.

The dc.list_measurements() function can be used to interrogate the measurements associated with a given product (specified by the name column from the table above). For example, ga_ls5t_ard_3 refers to the Geoscience Australia Landsat 5 Analysis-ready data Collection 3 product.

The table below includes a range of technical information about each band in the ga_ls5t_ard_3 dataset, including any aliases which can be used to load the data, the data type or dtype, any flags_definition that are associated with the measurement (this information is used for tasks like cloud masking), and the measurement’s nodata value.

Change the product name below and re-run the following cell to explore available measurements associated with other products.

[5]:
product = "ga_ls5t_ard_3"

measurements = dc.list_measurements()
measurements.loc[product]
[5]:
name dtype units nodata aliases flags_definition spectral_definition
measurement
nbar_blue nbar_blue int16 1 -999 [nbar_band01] NaN NaN
nbar_green nbar_green int16 1 -999 [nbar_band02] NaN NaN
nbar_red nbar_red int16 1 -999 [nbar_band03] NaN NaN
nbar_nir nbar_nir int16 1 -999 [nbar_band04] NaN NaN
nbar_swir_1 nbar_swir_1 int16 1 -999 [nbar_band05] NaN NaN
nbar_swir_2 nbar_swir_2 int16 1 -999 [nbar_band07] NaN NaN
nbart_blue nbart_blue int16 1 -999 [nbart_band01, blue] NaN NaN
nbart_green nbart_green int16 1 -999 [nbart_band02, green] NaN NaN
nbart_red nbart_red int16 1 -999 [nbart_band03, red] NaN NaN
nbart_nir nbart_nir int16 1 -999 [nbart_band04, nir] NaN NaN
nbart_swir_1 nbart_swir_1 int16 1 -999 [nbart_band05, swir_1, swir1] NaN NaN
nbart_swir_2 nbart_swir_2 int16 1 -999 [nbart_band07, swir_2, swir2] NaN NaN
oa_fmask oa_fmask uint8 1 0 [fmask] {'fmask': {'bits': [0, 1, 2, 3, 4, 5, 6, 7], 'values': {'0': 'nodata', '1': 'valid', '2': 'cloud', '3': 'shadow', '4': 'snow', '5': 'water'}, 'description': 'Fmask'}} NaN
oa_nbar_contiguity oa_nbar_contiguity uint8 1 255 [nbar_contiguity] {'contiguous': {'bits': [0], 'values': {'0': False, '1': True}}} NaN
oa_nbart_contiguity oa_nbart_contiguity uint8 1 255 [nbart_contiguity] {'contiguous': {'bits': [0], 'values': {'0': False, '1': True}}} NaN
oa_azimuthal_exiting oa_azimuthal_exiting float32 1 NaN [azimuthal_exiting] NaN NaN
oa_azimuthal_incident oa_azimuthal_incident float32 1 NaN [azimuthal_incident] NaN NaN
oa_combined_terrain_shadow oa_combined_terrain_shadow uint8 1 255 [combined_terrain_shadow] NaN NaN
oa_exiting_angle oa_exiting_angle float32 1 NaN [exiting_angle] NaN NaN
oa_incident_angle oa_incident_angle float32 1 NaN [incident_angle] NaN NaN
oa_relative_azimuth oa_relative_azimuth float32 1 NaN [relative_azimuth] NaN NaN
oa_relative_slope oa_relative_slope float32 1 NaN [relative_slope] NaN NaN
oa_satellite_azimuth oa_satellite_azimuth float32 1 NaN [satellite_azimuth] NaN NaN
oa_satellite_view oa_satellite_view float32 1 NaN [satellite_view] NaN NaN
oa_solar_azimuth oa_solar_azimuth float32 1 NaN [solar_azimuth] NaN NaN
oa_solar_zenith oa_solar_zenith float32 1 NaN [solar_zenith] NaN NaN
oa_time_delta oa_time_delta float32 1 NaN [time_delta] NaN NaN

Visualising available data

The interactive DcViewer utility provides a more visual way of exploring the data that is available within the Digital Earth Australia datacube.

After running the cell below, select a product from the drop-down menu on the top-right of the map to show the areas where data are available in blue. Use the back and forward buttons above the map to toggle through time.

The utility is only able to visualise a limited number of datasets at one time. If the available data footprints do not appear, either press the “show” button on the top right, or zoom in on the map.

[6]:
DcViewer(dc=dc,
         time='2015',
         width='800px',
         center=(-27.48, 153.10),
         zoom=7)

DEA Explorer sites

Another way to view the available data within a datacube is to visit the DEA Datacube Explorer sites. These webpages visualise the data that is available for every product in DEA.

There is a different Datacube Explorer page for both the NCI and DEA Sandbox environments to account for the different datasets that are available in each of these datacubes:

DEA Explorer


Additional information

License: The code in this notebook is licensed under the Apache License, Version 2.0. Digital Earth Australia data is licensed under the Creative Commons by Attribution 4.0 license.

Contact: If you need assistance, please post a question on the Open Data Cube Slack channel or on the GIS Stack Exchange using the open-data-cube tag (you can view previously asked questions here). If you would like to report an issue with this notebook, you can file one on Github.

Last modified: December 2019

Compatible datacube version:

[7]:
print(datacube.__version__)
1.7+262.g1cf3cea8

Tags

Browse all available tags on the DEA User Guide’s Tags Index

Tags: sandbox compatible, NCI compatible, dc.list_products, dc.list_measurements, products, measurements, landsat 5, datacube explorer, DcViewer