module NumRu::GPhys::NetCDF_Convention_Users_Guide

(To be written.)

module NumRu::GPhys::NetCDF_IO

a NetCDF read/write helper by automatically interpreting conventions

Module functions

is_a_NetCDF?(filename)

test whether the file is a NetCDF file.

ARGUMENTS

• filename (String): filename to test.

RETURN VALUE

• true/false

open(files, varname)

a GPhys constructor from a NetCDF file (or multiple NetCDF files).

ARGUMENTS

• files (String, NetCDF, NArray, or Regexp): file specifier. A single file is specified by a String (containing the path), of a NetCDF. Multiple files can be specified by a NArray of String or NetCDF or by a Regexp to match paths. In that case, data and axes are represented by VArrayComposite.
• varname (String): name of the variable.

RETURN VALUE

• a GPhys

EXAMPLES

• From a single file:

  file = NetCDF.open('hogehoge.nc')
  gphys = GPhys::NetCDF_IO(file, 'temp')
  
  file = NetCDF.open('hogehoge.nc', 'a')     # writable
  gphys = GPhys::NetCDF_IO(file, 'temp')

• From a single file:

  gphys = GPhys::NetCDF_IO('hogehoge.nc', 'temp')
  
  gphys = GPhys::NetCDF_IO('/data/netcdf/hogehoge.nc', 'temp')

  If you use a String to specify a file path, the file is opened as read-only.

• To use data separated into multiple files. Suppose that you have hoge_yr2000.nc, hoge_yr2001.nc, and hoge_yr2002.nc in the current directory. You can open it by using a regular expression as follows:

  gphys = GPhys::NetCDF_IO(/hoge_yr(\d\d\d\d).nc/, 'temp')

  Here, the parentheses to enclose \d\d\d\d is NEEDED.

  The same thing can be done as follows by using Array or NArray:

  files = ['hoge_yr2000.nc', 'hoge_yr2001.nc', 'hoge_yr2002.nc']
  gphys = GPhys::NetCDF_IO(files, 'temp')
  
  files = NArray['hoge_yr2000.nc', 'hoge_yr2001.nc', 'hoge_yr2002.nc']
  gphys = GPhys::NetCDF_IO(files, 'temp')

• Same as above but to use the full path:

  gphys = GPhys::NetCDF_IO(/\/data\/nc\/hoge_yr(\d\d\d\d).nc/, 'temp')

  Here, the directory separator '/' is escaped as '\/'.

• To use data separated into multiple files. Suppose that you have hoge_x0y0.nc, hoge_x1y0.nc, hoge_x0y1.nc, hoge_x1y1.nc, where the data is separated 2 dimensionally into 2*2 = 4 files.

  gphys = GPhys::NetCDF_IO(/hoge_x(\d)y(\d).nc/, 'temp')

  Note that 2 pairs of parentheses are needed here. Alternatively, you can also do it like this:

  files = NArray[ ['hoge_x0y0.nc', 'hoge_x1y0.nc'],
                  ['hoge_x0y1.nc', 'hoge_x1y1.nc'] ]
  gphys = GPhys::NetCDF_IO(files, 'temp')

write(file, gphys, name=nil)

Write a GPhys into a NetCDF file. The whole data under the GPhys (such as coordinate vars) are written self-descriptively.

ARGUMENTS

• file (NetCDF): the NetCDF file to write in. Must be writable of course.
• gphys (GPhys): the GPhys to write.
• name (String): (optional) name in the new file -- if you want to save with a different variable name than that of gphys.

RETURN VALUE

• nil

write_grid(file, grid_or_gphys)

Same as write but for writing only the contents of the grid. (Used in write.)

ARGUMENTS

• file (NetCDF): the NetCDF file to write in. Must be writable of course.
• grid_or_gphys (Grid or GPhys):

RETURN VALUE

• a Grid, in which all VArrays in the original grid are replaced with the new ones in the file.

each_along_dims_write(gphyses, files, *loopdims){...} # a block is expected

Iterator to process GPhys objects too big to read on memory at once. Makes a loop (loops) by dividing the GPhys object(s) (gphyses) with the dimension(s) specified by loopdims, and the results (which is the return value of the block) are written in files.

ARGUMENTS

• gphyses (GPhys or Array of GPhys): GPhys object(s) to be processed. All of them must have dimensions spcified with loopdims, and their lengths must not vary among files. Other dimensions are aribtary, so, for example, gphyses could be [a(lon,lat,time), b(lat,time)] as long as loopdims==["time"].
• files (NetCDF or Array of NetCDF): the file in which the results are written. The number of the file must be smaller than or equalt to the number of resultant GPhys objects (following the multiple assignment rule of Ruby).
• loopdims (Array of String or Integer) : name (when String) or count starting from zero (when Integer)
• expected block : Number of arguments == number of GPhys objects in gphyses. Expected return value is an Array of GPhys objects to be written files.

RETURN VALUE

• GPhys objects in which the results are written

ERRORS

The following raise exceptions (in adition to errors in arguments).

• Dimensions specified by loopdims are not shared among GPhys objects in gphyses.
• Return value of the block is not an Array of GPhys.
• Dimension(s) used for looping (loopdims) is(are) eliminated from the retunred GPhys objects.

USAGE

• EXAMPLE 1

  Suppose that you want to do the following:

  in = GPhys::NetCDF_IO.open(infile, varname)
  ofile = NetCDF.create(ofilename)
  out = in.mean(0)
  GPhys::NetCDF_IO.write( ofile, out )
  ofile.close

  The data object (in) is read on memory and an averagin is made. If the size of the data is too big to read on memory at once, you can divid this process by using this iterator. The following gives the same result as above, but the processing is made for each subset divided at the last dimension (represented by -1, as in the negative indexing of Array).

  in = GPhys::NetCDF_IO.open(infile, varname)
  ofile = NetCDF.create(ofilename)
  out = GPhys::NetCDF_IO.each_along_dims_write(in, ofile, -1){|in_sub|
    [ in_sub.mean(0) ]
  }
  ofile.close

  In this case, each_along_dims_write makes a loop by substituting in[false,0..0], in[false,1..1], in[false,2..2],.. into the argument of the block (in_sub). Thus, the return value of the block (here, [ in_sub.mean(0) ]) consists of in[false,0..0].mean(0), in[false,1..1].mean(0),.. . This iterator creates a GPhys object in out that represents the whole part of the results (here, in.mean(0)), and write the resultant subsets in it one by one. Therefore, the output file is filled correctly when exiting the iterator.

  Note that the subset (in_sub) retains the last dimension but the length is 1 becasue of the slicing by Range (0..0, 1..1,..). Therefore, the subset has the same rank as the original. The output GPhys objects, as given by the return value of the block, must have the dimension retained, since the dimension (whose length is one) is replaced by the original one when written in the file. Therefore, THE FOLLOWING CAUSE AN ERROR (an exception is raised):

  out = GPhys::NetCDF_IO.each_along_dims_write(in, ofile, 0){|in_sub|
    [ in_sub.mean(0) ]
  }

  Here, looping is made by the first dimension (0), but it is eliminated from the result by averaging with the same dimension. (Also, note that this averaging is non-sense, since the length of the first dimension of the subset is 1).

• EXAMPLE 2

  You can specify mutiple dimensions for looping to further decrease the size of data to read on memory:

  GPhys::NetCDF_IO.each_along_dims_write(in, ofile, -2, -1){|in_sub|
    ...
  }

  Also, you can specify the loop dimension(s) by name(s):

  GPhys::NetCDF_IO.each_along_dims_write(in, ofile, "y"){|in_sub|
    ...
  }
  
  GPhys::NetCDF_IO.each_along_dims_write(in, ofile, "y", "z"){|in_sub|
    ...
  }

• EXAMPLE 3

  You can give multiple objects in the iterotor if they have the same shape (in future, this restriction may been loosened), as follows:

  in1 = GPhys::NetCDF_IO.open(infile1, varname1)
  in2 = GPhys::NetCDF_IO.open(infile2, varname2)
  in3 = GPhys::NetCDF_IO.open(infile3, varname3)
  ofile = NetCDF.create(ofilename)
  outA, outB = \
    GPhys::NetCDF_IO.each_along_dims_write([in1,in2,in3], ofile, -1){
      |isub1,isub2,isub3|
      osubA = (isub1*isub2).mean(0)
      osubB = (isub2*isub3).mean(1)
      [ osubA, osubB ]
    }
  ofile.close

  In this case, two output objects (outA and outB) are made from the three input objects (in1,in2,in3) and written in a single file (ofile). If you want to separate into two files, you can do it like this:

  in1 = GPhys::NetCDF_IO.open(infile1, varname1)
  in2 = GPhys::NetCDF_IO.open(infile2, varname2)
  in3 = GPhys::NetCDF_IO.open(infile3, varname3)
  ofile1 = NetCDF.create(ofilename1)
  ofile2 = NetCDF.create(ofilename2)
  outA, outB = \
    GPhys::NetCDF_IO.each_along_dims_write([in1,in2,in3], [ofile1,ofile2], -1){
      |isub1,isub2,isub3|
      osubA = (isub1*isub2).mean(0)
      osubB = (isub2*isub3).mean(1)
      [ osubA, osubB ]
    }
  ofile.close

set_convention(convention)

Set a NetCDF convention to be interpreted.

ARGUMENTS

• convention (Module): the convention

RETURN VALUE

• convention (Module)

convention

Returns the current NetCDF convention to be interpreted.

RETURN VALUE

• convention (Module)

var_names(file)

ARGUMENTS

• file (NetCDF or String): if string, it must be the name (path) of a NetCDF file.

RETURN VALUE

• names of variables (Array): this return the names of variables which the file has.

var_names_except_coordinate(file)

ARGUMENTS

• file (NetCDF or String): if string, it must be the name (path) of a NetCDF file.

RETURN VALUE

• names of variables (Array): this return the names of variables which the file has, except variables for coordinate.