Introduction to products and measurements

  • Compatability: Notebook currently compatible with both the NCI and DEA Sandbox environments

  • Products used: ga_ls5t_ard_3

  • Prerequisites: Users of this notebook should have a basic understanding of:

Background

A “datacube” is a digital information architecture that specialises in hosting and cataloguing spatial information. Digital Earth Australia (DEA) is based on the Open Data Cube infrastructure, and specialises in storing remotely sensed data, particularly from Earth Observation satellites such as Landsat and Sentinel-2.

The DEA datacube contains both raw satellite data and derivative data “products”. These data products are often composed of a range of “measurements” such as the suite of remote sensing band values or statistical product summaries. Before running a query to load data from the datacube, it is useful to know what it contains. This notebook demonstrates several straightforward ways to inspect the product and measurement contents of a datacube.

Description

This notebook demonstrates how to connect to a datacube and interrogate the available products and measurements stored within. Topics covered include:

  • How to connect to a datacube

  • How to list all the products

  • How to list all the product measurements

  • How to interactively visualise data in the datacube


Getting started

To run this introduction to products and measurements, run all the cells in the notebook starting with the “Load packages” cell. For help with running notebook cells, refer back to the Jupyter Notebooks notebook.

Load packages

The datacube package is required to access and work with available data. The pandas package is required to format tables. The DcViewer utility provides an interface for interactively exploring the products available in the datacube.

[1]:
import datacube
import pandas as pd
from odc.ui import DcViewer

# Set some configurations for displaying tables nicely
pd.set_option('display.max_colwidth', 200)
pd.set_option('display.max_rows', None)

Connect to the datacube

After importing the datacube package, users need to specify a name for their session, known as the app name.

This name is generated by the user and is used to track down issues with database queries. It does not have any effect on the analysis. Use a short name that is consistent with the purpose of your notebook such as the way 03_Products_and_measurements has been used as the app name in this notebook.

The resulting dc object provides access to all the data contained within the Digital Earth Australia datacube.

[2]:
dc = datacube.Datacube(app="03_Products_and_measurements")

List products

Once a datacube instance has been created, users can explore the products and measurements stored within.

The following cell lists all products that are currently available in the DEA datacube by using the dc.list_products() function.

Products listed under name in the following table represent the product options available when querying the datacube. The table below provides some useful information about each product, including a brief product description, the instrument and platform the data originated from (e.g. Landsat 8 OLI), and the product’s default crs (coordinate reference system) and resolution if applicable.

For a comprehensive product description and access to complete product metadata, users are directed to the Geoscience Australia Content Management Interface

[3]:
products = dc.list_products()

display_columns = ["name",
                   "description",
                   "platform",
                   "instrument",
                   "crs",
                   "resolution"]

products[display_columns].sort_index()
[3]:
name description platform instrument crs resolution
id
1 ls5_satellite_telemetry_data Landsat 5 Satellite Telemetry Data LANDSAT_5 TM NaN NaN
2 ls5_level1_scene Landsat 5 Level 1 At-sensor Radiance 25 metre LANDSAT_5 TM NaN NaN
3 ls5_nbar_scene Landsat 5 NBAR 25 metre LANDSAT_5 TM NaN NaN
4 ls5_nbart_scene Landsat 5 NBART 25 metre LANDSAT_5 TM NaN NaN
5 ls5_pq_scene Landsat 5 PQ 25 metre LANDSAT_5 TM NaN NaN
6 ls5_nbar_albers Landsat 5 Surface Reflectance NBAR 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_5 TM EPSG:3577 (-25, 25)
8 ls7_satellite_telemetry_data Landsat 7 Satellite Telemetry Data LANDSAT_7 ETM NaN NaN
9 ls7_level1_scene Landsat 7 Level 1 At-sensor Radiance 25 metre LANDSAT_7 ETM NaN NaN
10 ls7_nbar_scene Landsat 7 NBAR 25 metre LANDSAT_7 ETM NaN NaN
11 ls7_nbart_scene Landsat 7 NBART 25 metre LANDSAT_7 ETM NaN NaN
12 ls7_pq_scene Landsat 7 PQ 25 metre LANDSAT_7 ETM NaN NaN
13 ls8_satellite_telemetry_data Landsat 8 Satellite Telemetry Data LANDSAT_8 None NaN NaN
14 ls8_level1_scene Landsat 8 Level 1 OLI-TIRS At-sensor Radiance 25 metre LANDSAT_8 OLI_TIRS NaN NaN
15 ls8_nbar_scene Landsat 8 NBAR 25 metre LANDSAT_8 OLI_TIRS NaN NaN
16 ls8_nbart_scene Landsat 8 NBART 25 metre LANDSAT_8 OLI_TIRS NaN NaN
17 ls8_pq_scene Landsat 8 PQ 25 metre LANDSAT_8 OLI_TIRS NaN NaN
19 ls8_nbar_albers Landsat 8 Surface Reflectance NBAR 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_8 OLI_TIRS EPSG:3577 (-25, 25)
20 ls8_pq_albers Landsat 8 Pixel Quality 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_8 OLI_TIRS EPSG:3577 (-25, 25)
21 ls7_nbar_albers Landsat 7 Surface Reflectance NBAR 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_7 ETM EPSG:3577 (-25, 25)
22 ls7_pq_albers Landsat 7 Pixel Quality 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_7 ETM EPSG:3577 (-25, 25)
23 ls5_pq_albers Landsat 5 Pixel Quality 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_5 TM EPSG:3577 (-25, 25)
26 ls5_nbart_albers Landsat 5 Surface Reflectance NBART 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_5 TM EPSG:3577 (-25, 25)
28 ls8_nbart_albers Landsat 8 Surface Relfectance NBART 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_8 OLI_TIRS EPSG:3577 (-25, 25)
29 ls7_nbart_albers Landsat 7 Surface Reflectance NBART 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_7 ETM EPSG:3577 (-25, 25)
32 dsm1sv10 DSM 1sec Version 1.0 SRTM SIR EPSG:4326 (-0.00027777777778, 0.00027777777778)
36 bom_rainfall_grids Interpolated Rain Gauge Precipitation 1-Day Australia 5km Gridded BoM rain gauge EPSG:4326 (-0.05, 0.05)
40 ls8_level1_oli_scene Landsat 8 OLI Level 1 At-sensor Radiance 25 metre LANDSAT_8 OLI NaN NaN
53 gamma_ray The 2015 radiometric or gamma-ray grid of Australia aircraft gamma_ray spectrometer GEOGCS["GEOCENTRIC DATUM of AUSTRALIA",DATUM["GDA94",SPHEROID["GRS80",6378137,298.257222101]],PRIMEM["Greenwich",0],UNIT["degree",0.0174532925199433]] (-0.001, 0.001)
56 srtm_dem1sv1_0 DEM 1sec Version 1.0 Space Shuttle Endeavour SIR EPSG:4326 (-0.00027777777778, 0.00027777777778)
57 ls8_nbar_oli_scene Landsat 8 OLI NBAR 25 metre LANDSAT_8 OLI NaN NaN
58 ls8_nbart_oli_scene Landsat 8 OLI NBART 25 metre LANDSAT_8 OLI NaN NaN
59 ls8_pq_oli_scene Landsat 8 OLI PQ 25 metre LANDSAT_8 OLI NaN NaN
60 ls8_nbar_oli_albers Landsat 8 Surface Reflectance NBAR 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_8 OLI EPSG:3577 (-25, 25)
61 ls8_nbart_oli_albers Landsat 8 Surface Relfectance NBART 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_8 OLI EPSG:3577 (-25, 25)
63 high_tide_comp_20p High Tide 20 percentage composites for entire coastline None None EPSG:3577 (-25, 25)
64 high_tide_comp_count High Tide 20 percentage pixel count None None EPSG:3577 (-25, 25)
65 low_tide_comp_20p Low Tide 20 percentage composites for entire coastline None None EPSG:3577 (-25, 25)
66 low_tide_comp_count Low Tide 20 percentage pixel count None None EPSG:3577 (-25, 25)
67 item_v2 Intertidal Extents Model None None EPSG:3577 (-25, 25)
68 item_v2_conf Average ndwi Standard Deviation None None EPSG:3577 (-25, 25)
69 ls5_fc_albers Landsat 5 Fractional Cover 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_5 TM EPSG:3577 (-25, 25)
70 ls7_fc_albers Landsat 7 Fractional Cover 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_7 ETM EPSG:3577 (-25, 25)
71 ls8_fc_albers Landsat 8 Fractional Cover 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_8 OLI_TIRS EPSG:3577 (-25, 25)
73 ls8_pq_legacy_scene Landsat 8 PQ Legacy 25 metre LANDSAT_8 OLI_TIRS NaN NaN
74 ls8_pq_legacy_oli_scene Landsat 8 OLI PQ Legacy 25 metre LANDSAT_8 OLI NaN NaN
75 ls5_pq_legacy_scene Landsat 5 PQ Legacy 25 metre LANDSAT_5 TM NaN NaN
76 ls7_pq_legacy_scene Landsat 7 PQ Legacy 25 metre LANDSAT_7 ETM NaN NaN
77 wofs_albers Historic Flood Mapping Water Observations from Space None None EPSG:3577 (-25, 25)
78 pq_count_annual_summary Pixel Quality Statistics 25m None None EPSG:3577 (-25, 25)
79 pq_count_summary Pixel Quality Statistics 25m None None EPSG:3577 (-25, 25)
80 ls8_pq_oli_albers Landsat 8 Pixel Quality 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_8 OLI EPSG:3577 (-25, 25)
81 ls8_nbart_geomedian_annual Surface Reflectance Geometric Median 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_8 OLI EPSG:3577 (-25, 25)
82 ls7_nbart_geomedian_annual Surface Reflectance Geometric Median 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_7 ETM EPSG:3577 (-25, 25)
83 ls5_nbart_geomedian_annual Surface Reflectance Geometric Median 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_5 TM EPSG:3577 (-25, 25)
84 wofs_summary Water Observations from Space Statistics LANDSAT_5,LANDSAT_7,LANDSAT_8 TM,ETM+,OLI EPSG:3577 (-25, 25)
85 wofs_filtered_summary Water Observations from Space Statistics confidence filtered LANDSAT_5,LANDSAT_7,LANDSAT_8 TM,ETM+,OLI EPSG:3577 (-25, 25)
86 mrvbf MrVBF ancillary data for wofs confidence None None EPSG:3577 (-25, 25)
87 modis Modis ancillary data for wofs confidence None None EPSG:3577 (-25, 25)
88 dsm dsm ancillary data for wofs confidence, 25 metre, 100km tiles, Australian Albers Equal Area projection None None EPSG:3577 (-25, 25)
89 urban_areas urbanAreas ancillary data for wofs confidence None None EPSG:3577 (-25, 25)
90 s2a_ard_granule Sentinel-2A MSI ARD - NBAR NBART and Pixel Quality SENTINEL_2A MSI NaN NaN
91 s2b_ard_granule Sentinel-2B MSI ARD - NBAR NBART and Pixel Quality SENTINEL_2B MSI NaN NaN
92 s2a_level1c_granule Sentinel-2A Level1C - Ortho Rectified Top of Atmosphere Reflectance Sentinel-2A MSI NaN NaN
93 s2b_level1c_granule Sentinel-2B Level1C - Ortho Rectified Top of Atmosphere Reflectance Sentinel-2B MSI NaN NaN
94 ls5_usgs_l2c1 Landsat 5 Thematic Mapper (TM) USGS Analysis Ready Data 30m scene LANDSAT_5 TM NaN NaN
95 ls7_usgs_l2c1 Landsat 7 Enhanced Thematic Mapper Plus (ETM+) USGS ARD 30 metre tile LANDSAT_7 ETM NaN NaN
96 ls8_usgs_l2c1 Landsat 8 Operational Land Imager (OLI) and Thermal Infrared Sensor (TIRS) USGS Analysis Ready Data 30m scene LANDSAT_8 OLI_TIRS NaN NaN
97 wofs_annual_summary Water Observations from Space Annual Statistics LANDSAT_5,LANDSAT_7,LANDSAT_8 TM,ETM,OLI EPSG:3577 (-25, 25)
98 ls8_barest_earth_mosaic Landsat-8 Barest Earth pixel composite mosaic 25 metre, 25km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_8 OLI EPSG:3577 (-25, 25)
99 fc_percentile_albers_annual Landsat Fractional Cover percentile 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_5,LANDSAT_7,LANDSAT_8 TM,ETM+,OLI EPSG:3577 (-25, 25)
100 wofs_apr_oct_summary Water Observations from Space April to October Statistics LANDSAT_5,LANDSAT_7,LANDSAT_8 TM,ETM,OLI EPSG:3577 (-25, 25)
101 wofs_nov_mar_summary Water Observations from Space November to March Statistics LANDSAT_5,LANDSAT_7,LANDSAT_8 TM,ETM,OLI EPSG:3577 (-25, 25)
102 ls8_nbart_tmad_annual Surface Reflectance Triple Median Absolute Deviation 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_8 OLI EPSG:3577 (-25, 25)
103 ls7_nbart_tmad_annual Surface Reflectance Triple Median Absolute Deviation 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_7 ETM+ EPSG:3577 (-25, 25)
104 ls5_nbart_tmad_annual Surface Reflectance Triple Median Absolute Deviation 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_5 TM EPSG:3577 (-25, 25)
105 mangrove_extent_cover_albers Mangroves conver extent and classes 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_5,LANDSAT_7,LANDSAT_8 TM,ETM+,OLI EPSG:3577 (-25, 25)
106 s2_tsmask Time series cloud and cloud shadow detection for Sentinel-2A and Sentinel-2B surface reflectance data.\n\nTSmask classifies a pixel as one of the following four categories: no observation, clear, ... SENTINEL_2A,SENTINEL_2B MSI NaN NaN
107 landsat_barest_earth Landsat-5/Landsat-7/Landsat-8 combined Barest Earth pixel composite albers 25 metre, 100km tile, Australian Albers Equal Area projection (EPSG:3577) LANDSAT_5,LANDSAT_7,LANDSAT_8 TM,ETM+,OLI EPSG:3577 (-25, 25)
108 usgs_ls5t_level1_1 Landsat 5 TM Level 1, USGS Collection 1 None None NaN NaN
109 usgs_ls7e_level1_1 Landsat 7 ETM+ Level 1, USGS Collection 1 None None NaN NaN
110 usgs_ls8c_level1_1 Landsat 8 OLI-TIRS Level 1, USGS Collection 1 None None NaN NaN
111 ga_ls5t_ard_3 Landsat 5 TM ARD, GA Collection 3 None None NaN NaN
112 ga_ls7e_ard_3 Landsat 7 ETM+ ARD, GA Collection 3 None None NaN NaN
113 ga_ls8c_ard_3 Landsat 8 OLI-TIRS ARD, GA Collection 3 None None NaN NaN
114 ga_ls5t_level1_3 Landsat 5 TM Level 1, GA Collection 3 None None NaN NaN
115 ga_ls7e_level1_3 Landsat 7 ETM+ Level 1, GA Collection 3 None None NaN NaN
116 ga_ls8c_level1_3 Landsat 8 OLI-TIRS Level 1, GA Collection 3 None None NaN NaN

List measurements

Most products are associated with a range of available measurements. These can be individual satellite bands (e.g. Landsat’s near-infrared band) or statistical product summaries.

The dc.list_measurements() function can be used to interrogate the measurements associated with a given product (specified by the name column from the table above). For example, ga_ls5t_ard_3 refers to the Geoscience Australia Landsat 5 Analysis-ready data Collection 3 product.

The table below includes a range of technical information about each band in the ga_ls5t_ard_3 dataset, including any aliases which can be used to load the data, the data type or dtype, any flags_definition that are associated with the measurement (this information is used for tasks like cloud masking), and the measurement’s nodata value.

Change the product name below and re-run the following cell to explore available measurements associated with other products.

[4]:
product = "ga_ls5t_ard_3"

measurements = dc.list_measurements()
measurements.loc[product]
[4]:
name dtype units nodata aliases flags_definition spectral_definition number
measurement
nbar_blue nbar_blue int16 1 -999 [nbar_band01] NaN NaN NaN
nbar_green nbar_green int16 1 -999 [nbar_band02] NaN NaN NaN
nbar_red nbar_red int16 1 -999 [nbar_band03] NaN NaN NaN
nbar_nir nbar_nir int16 1 -999 [nbar_band04] NaN NaN NaN
nbar_swir_1 nbar_swir_1 int16 1 -999 [nbar_band05] NaN NaN NaN
nbar_swir_2 nbar_swir_2 int16 1 -999 [nbar_band07] NaN NaN NaN
nbart_blue nbart_blue int16 1 -999 [nbart_band01, blue] NaN NaN NaN
nbart_green nbart_green int16 1 -999 [nbart_band02, green] NaN NaN NaN
nbart_red nbart_red int16 1 -999 [nbart_band03, red] NaN NaN NaN
nbart_nir nbart_nir int16 1 -999 [nbart_band04, nir] NaN NaN NaN
nbart_swir_1 nbart_swir_1 int16 1 -999 [nbart_band05, swir_1, swir1] NaN NaN NaN
nbart_swir_2 nbart_swir_2 int16 1 -999 [nbart_band07, swir_2, swir2] NaN NaN NaN
oa_fmask oa_fmask uint8 1 0 [fmask] {'fmask': {'bits': [0, 1, 2, 3, 4, 5, 6, 7], 'values': {'0': 'nodata', '1': 'valid', '2': 'cloud', '3': 'shadow', '4': 'snow', '5': 'water'}, 'description': 'Fmask'}} NaN NaN
oa_nbar_contiguity oa_nbar_contiguity uint8 1 255 [nbar_contiguity] {'contiguous': {'bits': [0], 'values': {'0': False, '1': True}}} NaN NaN
oa_nbart_contiguity oa_nbart_contiguity uint8 1 255 [nbart_contiguity] {'contiguous': {'bits': [0], 'values': {'0': False, '1': True}}} NaN NaN
oa_azimuthal_exiting oa_azimuthal_exiting float32 1 NaN [azimuthal_exiting] NaN NaN NaN
oa_azimuthal_incident oa_azimuthal_incident float32 1 NaN [azimuthal_incident] NaN NaN NaN
oa_combined_terrain_shadow oa_combined_terrain_shadow uint8 1 255 [combined_terrain_shadow] NaN NaN NaN
oa_exiting_angle oa_exiting_angle float32 1 NaN [exiting_angle] NaN NaN NaN
oa_incident_angle oa_incident_angle float32 1 NaN [incident_angle] NaN NaN NaN
oa_relative_azimuth oa_relative_azimuth float32 1 NaN [relative_azimuth] NaN NaN NaN
oa_relative_slope oa_relative_slope float32 1 NaN [relative_slope] NaN NaN NaN
oa_satellite_azimuth oa_satellite_azimuth float32 1 NaN [satellite_azimuth] NaN NaN NaN
oa_satellite_view oa_satellite_view float32 1 NaN [satellite_view] NaN NaN NaN
oa_solar_azimuth oa_solar_azimuth float32 1 NaN [solar_azimuth] NaN NaN NaN
oa_solar_zenith oa_solar_zenith float32 1 NaN [solar_zenith] NaN NaN NaN
oa_time_delta oa_time_delta float32 1 NaN [time_delta] NaN NaN NaN

Visualising available data

The interactive DcViewer utility provides a more visual way of exploring the data that is available within the Digital Earth Australia datacube.

After running the cell below, select a product from the drop-down menu on the top-right of the map to show the areas where data are available in blue. Use the back and forward buttons above the map to toggle through time.

The utility is only able to visualise a limited number of datasets at one time. If the available data footprints do not appear, either press the “show” button on the top right, or zoom in on the map.

[5]:
DcViewer(dc=dc,
         time='2015',
         width='800px',
         center=(-27.48, 153.10),
         zoom=7)

DEA Explorer sites

Another way to view the available data within a datacube is to visit the DEA Datacube Explorer sites. These webpages visualise the data that is available for every product in DEA.

There is a different Datacube Explorer page for both the NCI and DEA Sandbox environments to account for the different datasets that are available in each of these datacubes:

DEA Explorer


Additional information

License: The code in this notebook is licensed under the Apache License, Version 2.0. Digital Earth Australia data is licensed under the Creative Commons by Attribution 4.0 license.

Contact: If you need assistance, please post a question on the Open Data Cube Slack channel or on the GIS Stack Exchange using the open-data-cube tag (you can view previously asked questions here). If you would like to report an issue with this notebook, you can file one on Github.

Last modified: June 2020

Compatible datacube version:

[6]:
print(datacube.__version__)
1.8.0

Tags

Browse all available tags on the DEA User Guide’s Tags Index

Tags: sandbox compatible, NCI compatible, dc.list_products, dc.list_measurements, products, measurements, landsat 5, datacube explorer, DcViewer