Frequently Asked Questions About netCDF This document contains answers to some of the most frequently asked questions on the netcdfgroup mailing list and in the email sent to support@unidata.ucar.edu. ------------------------------------------------------------- General * What Is netCDF? * How do I get the netCDF software package? * What's new in the netCDF 3.5 release? * What is the best way to represent [some particular data] using netCDF? * What convention should be used for the names of netCDF files? * Is there a mailing list for netCDF discussions and questions? * What is the best way to handle time using netCDF? * Who else uses netCDF? * What are some references to netCDF? * Is there a document describing the actual physical format for a Unidata netCDF file? Platforms and Portability * What does netCDF run on? * Is netCDF available for Windows 95/98 or Windows NT? * Is netCDF available for the Apple Macintosh? * Is netCDF available for MSDOS/Borland platforms? netCDF and Other Software * What other software is available for accessing, displaying, and manipulating netCDF data? * What other data access interfaces and formats are available for scientific data? * What is the connection between netCDF and CDF? * What is the connection between netCDF and HDF? * Has anyone implemented client-server access for netCDF data? Problems and Bugs * Can I recover data from a netCDF file that was not closed properly? * Is there a list of reported problems and workarounds? * How do I make a bug report? * How do I search through past problem reports? Language Interfaces * Which programming languages have netCDF interfaces? * How does the C++ interface differ from the C interface? * How does the Fortran interface differ from the C interface? * How does the perl interface differ from the C interface? Plans * Are there plans to add facilities for data compression to netCDF? * What other future work on netCDF is planned? ------------------------------------------------------------- General What Is netCDF? NetCDF (network Common Data Form) is an interface for array-oriented data access and a freely-distributed collection of software libraries for C, Fortran, C++, Java, and perl that provide implementations of the interface. The netCDF software was developed by Glenn Davis, Russ Rew, Steve Emmerson, John Caron, and Harvey Davies at the Unidata Program Center in Boulder, Colorado, and augmented by contributions from other netCDF users. The netCDF libraries define a machine-independent format for representing scientific data. Together, the interface, libraries, and format support the creation, access, and sharing of scientific data. NetCDF data is: * Self-Describing. A netCDF file includes information about the data it contains. * Architecture-independent. A netCDF file is represented in a form that can be accessed by computers with different ways of storing integers, characters, and floating-point numbers. * Direct-access. A small subset of a large dataset may be accessed efficiently, without first reading through all the preceding data. * Appendable. Data can be appended to a netCDF dataset along one dimension without copying the dataset or redefining its structure. The structure of a netCDF dataset can be changed, though this sometimes causes the dataset to be copied. * Sharable. One writer and multiple readers may simultaneously access the same netCDF file. ------ How do I get the netCDF software package? Source distributions are available via anonymous FTP from the directory ftp://ftp.unidata.ucar.edu/pub/netcdf/ or from other mirror sites. Files in that directory include: netcdf.tar.Z or netcdf.ZIP compressed tar file or ZIP archive of source code for the latest general release, version 3.5, March 2001 guidec.ps.gz NetCDF User's Guide for C (gzipped PostScript) guidec.pdf.gz NetCDF User's Guide for C (gzipped PDF) guidec.html.tar.gz NetCDF User's Guide for C (gzipped tar HTML files) f90/Documentation/guide.book.ps NetCDF User's Guide for Fortran 90 (PostScript) f90/Documentation/guide.book.pdf NetCDF User's Guide for Fortran 90 (PDF) guidef.ps.gz NetCDF User's Guide for Fortran 77 (gzipped PostScript) guidef.pdf.gz NetCDF User's Guide for Fortran 77 (gzipped PDF) guidef.html.tar.gz NetCDF User's Guide for Fortran 77 (gzipped tar HTML files) guide-2.4.3.ps.gz NetCDF (Version 2) User's Guide (gzipped PostScript) Binary distributions for some platforms are available from the directory ftp://ftp.unidata.ucar.edu/pub/binary. Source for the perl interface is available as a separate package, netCDFPerl, via anonymous FTP from the ftp://ftp.unidata.ucar.edu/pub/netcdf-perl/ directory. ------ What's new in the netCDF 3.5 release? Version 3.5 includes bug fixes and portability enhancements to version 3.4, but also includes a new Fortran90 interface. The netCDF file format was unchanged, so files written with version 3.5 can be read with previous versions and vice versa (except that very large files will only be readable on platforms that support very large files). For more details, see the version 3.5 release notes. Version 3.4 included large file support, some performance enhancements, some portability improvements, and fixes for known bugs in netCDF 3.3.1. The netCDF file format was unchanged, so files written with version 3.4 can be read with previous versions and vice versa (except that very large files will only be readable on platforms that support very large files). For more details, see the version 3.4 announcement. Version 3.3 was a major update to netCDF that included a complete rewrite of the netCDF library, significant performance improvements, and updated interfaces. For more details, see the version 3.3 release notes. ------ What is the best way to represent [some particular data] using netCDF? There are many ways to represent the same information in any general-purpose data model. Choices left up to the user in the case of netCDF include which information to represent as variables or as variable attributes; what names to choose for variables, dimensions, and attributes; what order to use for the dimensions of multidimensional variables; what variables to include in the same netCDF file; and how to use variable attributes to capture the structure and meaning of data. We provide some guidelines in the NetCDF User's Guide (e.g., the section on ``Differences between Attributes and Variables''), but we've found that a little experience helps. Occasionally we have decided it was useful to change the structure of netCDF files after experience with how the data is used. ------ What convention should be used for the names of netCDF files? NetCDF files should have the file name extension ".nc". The recommended extension for netCDF files was changed from ".cdf" to ".nc" in 1994 in order to avoid a clash with the NASA CDF file extension, and now it also avoids confusion with "Channel Definition Format" files. ------ Is there a mailing list for netCDF discussions and questions? Yes. For information about the mailing list and how to subscribe or unsubscribe, send a message to majordomo@unidata.ucar.edu with no subject and with the following line in the body of the message: info netcdfgroup ------ What is the best way to handle time using netCDF? Discussions of conventions for representing time and handling time-dependent data is a fairly frequent topic of discussion on the netcdfgroup mailing list. When the subject comes up, interesting discussions often result, so we've archived past discussions on this subject at http://www.unidata.ucar.edu/packages/netcdf/time/. A summary of Unidata's recommendations is available from http://www.unidata.ucar.edu/packages/netcdf/time/recs.html. Briefly, we recommend use of the units conventions supported by the udunits library for time and other units attributes. Other groups have established more specific conventions that include the representation of time in netCDF files. For more information on such conventions, see the NetCDF Conventions Page at http://www.unidata.ucar.edu/packages/netcdf/conventions.html. ------ Who else uses netCDF? The netCDF mailing list has over 500 addresses (some of which are aliases to more addresses) in thirty countries. Several groups have adopted netCDF as a standard way to represent some forms of scientific data. A description of some of the projects and groups that have used netCDF is available from http://www.unidata.ucar.edu/packages/netcdf/usage.html. ------ What are some references to netCDF? A primary reference is the User's Guide: Rew, R. K., G. P. Davis, S. Emmerson, and H. Davies, NetCDF User's Guide for C, An Interface for Data Access, Version 3, April 1997. (Available from Unidata or in PostScript form by anonymous FTP from ftp://ftp.unidata.ucar.edu/pub/netcdf/guidec.ps.Z). There is also a hypertext version of the User's Guide available at http://www.unidata.ucar.edu/packages/netcdf/guidec. A similar User's Guide for Fortran is also available. Other references include: Brown, S. A, M. Folk, G. Goucher, and R. Rew, "Software for Portable Scientific Data Management," Computers in Physics, American Institute of Physics, Vol. 7, No. 3, May/June 1993, pp. 304-308. Fulker, D. W., "Unidata Strawman for Storing Earth-Referencing Data," Seventh International Conference on Interactive Information and Processing Systems for Meteorology, Oceanography, and Hydrology, New Orleans, La., American Meteorology Society, January 1991. Jenter, H. L. and R. P. Signell, 1992. "NetCDF: A Freely-Available Software-Solution to Data-Access Problems for Numerical Modelers". Proceedings of the American Society of Civil Engineers Conference on Estuarine and Coastal Modeling. Tampa, Florida. (Also available as a PostScript file from http://www.unidata.ucar.edu/packages/netcdf/netcdf.asce.ps). Kuehn, J.A., "Faster Libraries for Creating Network-Portable Self-Describing Datasets", Proceedings of the 37th Cray User Group Meeting, (Barcelona, Spain, March 1996), Cray User Group, Inc. Rew, R. K. and G. P. Davis, "NetCDF: An Interface for Scientific Data Access," IEEE Computer Graphics and Applications, Vol. 10, No. 4, pp. 76-82, July 1990. Rew, R. K. and G. P. Davis, "The Unidata netCDF: Software for Scientific Data Access," Sixth International Conference on Interactive Information and Processing Systems for Meteorology, Oceanography, and Hydrology, Anaheim, California, American Meteorology Society, pp. 33-40, February 1990. Rew, R. K. and G. P. Davis, " Unidata's netCDF Interface for Data Access: Status and Plans," Thirteenth International Conference on Interactive Information and Processing Systems for Meteorology, Oceanography, and Hydrology, Anaheim, California, American Meteorology Society, February 1997. ------ Is there a document describing the actual physical format for a Unidata netCDF file? Yes, the NetCDF User's Guide contains an appendix that specifies the netCDF file format. Also, the chapter on "NetCDF File Structure and Performance" explains the physical structure of netCDF data at a high enough level to make clear the performance implications of different data organizations. If users only go through the documented interfaces to access netCDF data, any future changes to the format will be transparent. ------------------------------------------------------------- Platforms and Portability What does netCDF run on? The 3.5 release of netCDF is currently supported on the following platforms: * AIX-4.3 * HPUX-11.00 * IRIX-6.5, IRIX64-6.5 * Linux 2.2 * MacOS X (public beta) * OSF1-4.0 * SunOS-4.1.4, SunOS-5.7, SunOS-5.8 (Sparc and i386) The INSTALL file in the distribution contains more details about what compilers and compiler flags were used for building netCDF on various platforms. Additionally, here is a Web page of reports of successful builds in other environments that we did not or could not try. ------ Is netCDF available for Windows 95/98 or Windows NT? We have pre-built netCDF 3.5.0 libraries for Win 95/98/NT, available here. To do your own source build, get this zip file (these make files are for Microsoft Visual C 6 and Digital Fortran 6, you will have to modify for a different environment). Read WIN32_README.TXT for more info. Also, a pre-compiled netCDF 3.3 for GNU Win32 that includes a port of udunits is available from ftp://niteroi.gsfc.nasa.gov/pub/win32/unidata/netcdf. A README describing this version is also available. ------ Is netCDF available for the Apple Macintosh? Unidata doesn't test or maintain a version of netCDF for MacOS 9 or below, but others have made their ports available: * Chuck Denham (USGS) has ported and made available a version of the netCDF libraries for Macs. You can fetch cdflib.o and xdrlib.o for PowerMac via anonymous FTP from the directory ftp://crusty.er.usgs.gov/incoming/NetCDF-Mac/. The format is MacBinary II. * Sergey Malyshev (UIUC) has developed a Macintosh version of the netCDF library, for both PowerPC and 68K, and two MPW tools working with NetCDF files (they are a part of source distribution). This includes the Fortran interface built for Absoft FORTRAN. If you are a MATLAB user, try using the mexcdf function developed at USGS, which you can pick up from http://crusty.er.usgs.gov/mexcdf.html. A compiled netCDF library for PowerMacs is available from http://crusty.er.usgs.gov/~cdenham/MexCDF/nc4ml5.html as well. ------ Is netCDF available for MSDOS/Borland platforms? Unidata doesn't test or maintain a version of netCDF for Borland compilers, but see the files in the directory ftp://ftp.unidata.ucar.edu/pub/netcdf/contrib/borland/ for compiled netcdf 2.3.2 libraries for MSDOS/Borland platforms, including XDR and netcdf libraries for all four memory models, and declarations for using netCDF with Borland Delphi version 5.0. ------------------------------------------------------------- NetCDF and Other Software What other software is available for accessing, displaying, and manipulating netCDF data? Utilities available in the current netCDF distribution from Unidata are ncdump, for converting netCDF files to an ASCII human-readable form, and ncgen for converting from the ASCII human-readable form back to a binary netCDF file or a C or FORTRAN program for generating the netCDF file. A list of other software that can be used to manipulate or display netCDF data is available from http://www.unidata.ucar.edu/packages/netcdf/software.html. Another guide to netCDF utilities is available from http://www.gfdl.gov/~jps/GFDL_VG_NetCDF_Utils.html. ------ What other data access interfaces and formats are available for scientific data? The Scientific Data Format Information FAQ provides a good description of other access interfaces and formats for scientific data, including CDF and HDF. A brief comparison of CDF, netCDF, and HDF is available in the CDF FAQ. ------ What is the connection between netCDF and CDF? CDF was developed at the NASA Space Science Data Center at Goddard, and is freely available. It was originally a VMS FORTRAN interface for scientific data access. Unidata reimplemented the library from scratch to use XDR for a machine-independent representation, designed the CDL (network Common Data form Language) text representation for netCDF data, wrote a User's Guide and made other additions including aggregate data access, single-file implementation, named dimensions, and variable-specific attributes. NetCDF and CDF have evolved independently. CDF now supports many of the same features as netCDF (aggregate data access, XDR representation, single-file representation, variable-specific attributes), but some differences remain (netCDF doesn't support native-mode representation, CDF doesn't support named dimensions). There is no compatibility between data in CDF and netCDF form, and as yet no translation software exists to convert data in one form to data in the other form. For a more detailed description of differences between CDF and netCDF, see the CDF FAQ. ------ What is the connection between netCDF and HDF? The National Center for Supercomputing Applications (NCSA) developed the HDF software and makes it freely available. HDF is an extensible data format for self-describing files that was developed independently of netCDF. Applications and utilities based on HDF are available that support raster-image manipulation and display and browsing through multidimensional scientific data. The HDF software includes a package of routines for accessing each HDF data type, as well as a lower-level interface for building packages to support new types. HDF supports both C and Fortran interfaces, and it has been successfully ported to a wide variety of machine architectures and operating systems. HDF emphasizes a single common format for data, on which many interfaces can be built. NCSA has implemented software that provides a netCDF interface to HDF. With this software, it is possible to use the netCDF calling interface to place data into an HDF file. The netCDF calling interface has not changed and netCDF files stored in XDR format are readable, so existing programs and data will still be usable (although programs will need to be relinked to the new library). There is currently no support for the mixing of HDF and netCDF structures. For example, a raster image can exist in the same file as a netCDF object, but you have to use the Raster Image interface to read the image and the netCDF interface to read the netCDF object. The other HDF interfaces are currently being modified to allow multi-file access, closer integration with the netCDF interface will probably be delayed until the end of that project. Eventually, it may be possible to integrate netCDF objects with the rest of the HDF tool suite. Such an integration will then allow tools written for netCDF and tools written for HDF to both interact intelligently with the new data files. ------ Has anyone implemented client-server access for netCDF data? Yes, as part of the Distributed Oceanographic Data System (DODS), developers at the University of Rhode Island and Massachusetts Institute of Technology have implemented a client-server based distributed system for access to oceanographic data over the Internet that supports use of the netCDF interface for clients. The source code is available in source and binary forms from ftp://ftp.unidata.ucar.edu/pub/dods/ for the server, sample clients, and libraries. You do not need to get the source code to try to re-link your netCDF applications. The server supports use of URL notation for accessing netCDF data from remote sites running the DODS server. ------------------------------------------------------------- Problems and Bugs Can I recover data from a netCDF file that was not closed properly? I have some netcdf files which have data in them and were apparently not properly closed. When I examine them using ncdump they report zero data points, although the size is a few meg. Is there a way of recovering them? Yes, see the archived message at http://www.unidata.ucar.edu/glimpse/netcdfgroup-list/1250 for a way to do this. ------ Is there a list of reported problems and workarounds? Yes, the document Known problems with the netCDF Distribution describes reported problems and workarounds in the latest version and some earlier releases. ------ How do I make a bug report? If you find a bug, send a description to support@unidata.ucar.edu. This is also the address to use for questions or discussions about netCDF that are not appropriate for the entire netcdfgroup mailing list. ------ How do I search through past problem reports? A search form is available at the bottom of the netCDF homepage, providing a full-text search of the support questions and answers about netCDF provided by Unidata support staff. ------------------------------------------------------------- Language Interfaces Which programming languages have netCDF interfaces? The netCDF distribution comes with interfaces for Fortran77, C, and C++. Other languages for which interfaces are available separately include: * perl * MATLAB-5 * Python * Java How does the C++ interface differ from the C interface? It provides all the functionality of the C interface (except for the generalized hyperslab access of ncvarputg() and ncvargetg()) and is somewhat simpler to use than the C interface. With the C++ interface, no IDs are needed for netCDF components, there is no need to specify types when creating attributes, and less indirection is required for dealing with dimensions. However, the C++ interface is less mature and less-widely used than the C interface, and the documentation for the C++ interface is less extensive, assuming a familiarity with the netCDF data model and the C interface. ------ How does the Fortran interface differ from the C interface? It provides all the functionality of the C interface. The Fortran interface uses Fortran conventions for array indices, subscript order, and strings. There is no difference in the on-disk format for data written from the different language interfaces. Data written by a C language program may be read from a Fortran program and vice-versa. ------ How does the Perl interface differ from the C interface? It provides all the functionality of the C interface. The Perl interface uses Perl conventions for arrays and strings. There is no difference in the on-disk format for data written from the different language interfaces. Data written by a C language program may be read from a Perl program and vice-versa. ------------------------------------------------------------- Plans Are there plans to add facilities for data compression to netCDF? We have no plans to add data compression to netCDF, although we do plan to add a form of transparent data packing on write and unpacking on read in netCDF 4. Arbitrary array cross-section access and direct access to individual array values conflict with most simple compression schemes. With netCDF, the elements of an array variable can be filled in any order or as cross-sections in any direction. NetCDF permits writing elements in one order and reading them later in different orders. Some compression methods require that all the data to be compressed are known before starting the compression. Techniques like run-length encoding or anything that depends on exploiting similarities in nearby values can't be used if nearby values aren't all known at the time some of the data are to be written. An alternative that can be implemented above the netCDF library is to adopt a convention for compressed data that uses a "compression" attribute to encode the method of compression, e.g. x:compression = "rle" ; for run-length encoding of the data in a variable x. Then when you write the data, compress them into a bland array of bytes and write all the bytes. Note that it would be difficult to define the size of such a variable in advance, since its compressed size depends on its values. You would also have to give up on cross-section access for such variables, but instead read the compressed array in all at once and uncompress it before using it. Another alternative, written and made available by Bill Noon, is a set of modifications to the netCDF 3.3.1 library source that allows transparent access to both compressed and uncompressed netCDF files. A more complete explanation, the source changes, instructions on how to patch the netcdf-3.3.1 source, and some perfomance numbers are available from http://snow.cit.cornell.edu/noon/z_netcdf.html ------ What other future work on netCDF is planned? Plans for the next major release, netCDF-4, include the addition of new types for packed data and the elimination of the 2 Gbyte limit on file size. A paper is available describing current netCDF status and plans. Other desirable netCDF enhancements that may be implemented, depending on available resources include: * Link variables pointing to virtual data or data in other files, using Davies' FAN notation * Multiple unlimited dimensions per dataset * Access by key for generalized coordinate variables * Support for nested arrays (as in J), ragged arrays, and structures. * Support for new types: 64-bit integers, unsigned types * NetCDF client-server interfaces * Support for wide characters * Thread-safety (modulo underlying system calls) This page is maintained by Russ Rew. Questions or comments can be sent to . This page was updated on Apr 09, 2001.