The CF Metadata Conventions
The CF Metadata Conventions (“CF” henceforth) are being developed by academics and practitioners in the climate and forecasting field, with the objective to facilitate interoperability between data producers and data consumers (whether human or computer systems).
Reading the CF documentation can be a daunting task because it is written as a standards document, and not as a guideline or tutorial for people who want to understand the concept and the structure but who are not looking for all of the low-level detail.
This vignette presents a view of CF from the perspective of R. The main elements of CF are introduced in terms that should be easily understood by anyone who has worked with matrices and arrays in R.
By definition, a CF-compliant data set is stored in a netCDF file. The CF elements and their relationship to the building blocks of the netCDF file are given in the figure below:
If you’ve worked with netCDF data in R before, probably using package
ncdf4 (but what follows is equally true when you use
package RNetCDF), you should be familiar with the yellow
boxes in the above figure, especially “NC::Variable” (the specific
notation used in the figure is not important here), corresponding to the
ncvar4 class in ncdf4. CF, however, recognizes
eight different objects that are each based on an “NC::Variable”. This
ranges from axes, or generic coordinate variables in the white
box in the figure (to get the dimension values of your data array when
not loaded upon opening the file you have to call
ncdf4::ncvar_get(), i.e. you read a variable to get the
values of a dimension), to grid mapping variables (that define
the coordinate reference system of your data) to the actual data array
in a data variable. This is the source of all those surprising
“variables” that you see when you do names(nc$var):
library(ncdf4)
fn <- system.file("extdata", "tasmax_NAM-44_day_20410701-vncdfCF.nc", package = "ncdfCF")
nc <- nc_open(fn)
# The "variables"
names(nc$var)
#> [1] "time_bnds" "lon" "lat"
#> [4] "Lambert_Conformal" "height" "tasmax"
# The dimensions
names(nc$dim)
#> [1] "time" "bnds" "x" "y"So which of these variables is the one that holds your data? And how
do you tell it apart from the other “variables”? And why are there
x and y dimensions and lon and
lat “variables”?
Enter the attributes
You may not be as familiar with the attributes in the netCDF file,
the “NC::Attribute” object in the above figure, as ncdf4
does not make these as easily accessible as the other information in the
file. That is a shame because attributes are the thing that makes netCDF
stand apart from pretty much any other gridded data format. CF is
actually all about attributes, with some of them organized in
“NC::Variable” objects that do not hold any other data! (Such as the
grid mapping variables mentioned above.) Put the other way
around, you are not really using a CF-compliant netCDF data set unless
you read the proper attributes.
Looking at the same netCDF file as above, but now using the
ncdfCF package:
library(ncdfCF)
fn <- system.file("extdata", "tasmax_NAM-44_day_20410701-vncdfCF.nc", package = "ncdfCF")
ds <- open_ncdf(fn)
# The data variable
names(ds)
#> [1] "tasmax"
# The details of the data variable
tmax <- ds[["tasmax"]]
tmax$print(width = 25)
#> <Variable> tasmax
#> Long name: Daily Maximum Near-Surface Air Temperature
#>
#> Coordinate reference system:
#> name grid_mapping
#> Lambert_Conformal lambert_conformal_conic
#>
#> Axes:
#> id axis name long_name length unlim values
#> 2 X x x-coordinate in Cartes... 148 [0 ... 7350000]
#> 3 Y y y-coordinate in Cartes... 140 [0 ... 6950000]
#> 0 T time 1 U [2041-07-01 12:00:00]
#> Z height 1 [2]
#> unit
#> m
#> m
#> days since 1949-12-1 0...
#> m
#>
#> Auxiliary longitude-latitude grid:
#> orientation axis name extent unit
#> longitude X lon [-174.382 ... -19.618] degrees_east
#> latitude Y lat [10.154 ... 76.459] degrees_north
#>
#> Attributes:
#> id name type length value
#> 0 standard_name NC_CHAR 15 air_temperature
#> 1 long_name NC_CHAR 42 Daily Maximum Near-Sur...
#> 2 units NC_CHAR 1 K
#> 3 grid_mapping NC_CHAR 17 Lambert_Conformal
#> 4 coordinates NC_CHAR 14 height lat lon
#> 5 _FillValue NC_FLOAT 1 1.00000002004088e+20
#> 6 missing_value NC_FLOAT 1 1.00000002004088e+20
#> 7 original_name NC_CHAR 11 TEMP at 2 M
#> 8 cell_methods NC_CHAR 13 time: maximum
#> 9 FieldType NC_INT 1 104
#> 10 MemoryOrder NC_CHAR 3 XY
# The "time" axis
ds[["time"]]
#> <Time axis> [0] time
#> Length : 1 (unlimited)
#> Axis : T
#> Calendar : 365_day
#> Range : 2041-07-01 12:00:00
#> Bounds : 2041-07-01 ... 2041-07-02
#>
#> Attributes:
#> id name type length value
#> 0 standard_name NC_CHAR 4 time
#> 1 long_name NC_CHAR 4 time
#> 2 bounds NC_CHAR 9 time_bnds
#> 3 units NC_CHAR 29 days since 1949-12-1 00:00:00
#> 4 calendar NC_CHAR 7 365_day
#> 5 axis NC_CHAR 1 T
# Parameters in the grid mapping variable
ds[["Lambert_Conformal"]]
#> <Grid mapping> Lambert_Conformal
#> Grid mapping name: lambert_conformal_conic
#>
#> Attributes:
#> id name type length value
#> 0 grid_mapping_name NC_CHAR 23 lambert_conformal_conic
#> 1 longitude_of_central_meridian NC_DOUBLE 1 -97
#> 2 latitude_of_projection_origin NC_DOUBLE 1 46.0000038146973
#> 3 standard_parallel NC_DOUBLE 2 35, 60
#> 4 false_easting NC_DOUBLE 1 3675000
#> 5 false_northing NC_DOUBLE 1 3475000Ok, that’s a lot more information. Let’s focus on a few important things:
- There is only one data variable, called “tasmax”. If we
examine it’s details we can see that it’s coordinate space uses four
axes
x,y,timeandheight. That is not all clear from thencdf4output which lists the first three dimensions and excludesheight, but includesbndsinstead. - The data set has a grid mapping variable called
Lambert_Conformalthat, by definition, applies to theXandYaxes of the data variabletasmax. As suggested by CF, the data set also includes two ancillary coordinate variableslatandlon, each a matrix with the same dimension as the length of theYandXaxes of the data variable they are associated with, where each cell gives the latitude and longitude, respectively, of the corresponding cell in the data variable (whose axes have coordinates in the Lambert Conformal coordinate reference system). You can identify these two aspects by looking at the attributes oftasmax: attributegrid_mappingpoints to the grid mapping variableLambert_Conformal, attributecoordinateslists both the scalar coordinate variableheightand the auxiliary coordinate variableslatandlon. - The
timeaxis has an attributeboundswhich points totime_bnds, a boundary variable that indicates for each coordinate along the axis its lower and higher values. For thetimeaxis there is just one coordinate at2041-07-01 12:00:00with range values2041-07-01 ... 2041-07-02(midnight to midnight). (It is thetime_bndsvariable that uses thebndsdimension thatncdf4reports.) Thecalendarused by this axis is365_day, meaning that no year uses the leap day of 29 February. This is a so-called “model calendar” and the standard date-time functions will not work properly with this data; use the built-in time methods instead.
It should be obvious that correctly interpreting a netCDF file requires a package that not only provides easy access to the attributes of the file, but that also applies the CF conventions to put all the pieces together in such a way that the data set is presented to the user of the data - that would be you - as the data producers intended.
The issues arising from the particularities of CF on top of the
netCDF format are presented and discussed in the remainder of this
document, seen from the perspective of the R user, with examples using
the ncdfCF package. At the end of this vignette is a
feature matrix that indicates the support for each CF element in package
ncdfCF.
Please note that I have nothing against
ncdf4orRNetCDFpackages. In fact, packagencdfCFis built on top ofRNetCDF. My point is merely to demonstrate how CF-compliant netCDF files, of which there are very many out there on the internet, require you to look beyond the basic netCDF building blocks and use the attributes. If your netCDF files do not use the CF conventions, then by all means usencdf4if you prefer. I would suggest, though, to read the next section to understand better how data structures in netCDF differ from R standard practices.
This structure enables a full
support of the features provided by the