-
Notifications
You must be signed in to change notification settings - Fork 1
/
index.html
270 lines (255 loc) · 12.2 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "https://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="https://www.w3.org/1999/xhtml">
<head>
<title> PnetCDF </title>
<!-- Global site tag (gtag.js) - Google Analytics -->
<script async src="https://www.googletagmanager.com/gtag/js?id=G-CJ8LEHVTXW"></script>
<script>
window.dataLayer = window.dataLayer || [];
function gtag(){dataLayer.push(arguments);}
gtag('js', new Date());
gtag('config', 'G-CJ8LEHVTXW');
</script>
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-45938012-1', 'auto');
ga('create', 'UA-45937491-1', 'auto', {'name': 'pnetcdfTracker'});
ga('create', 'UA-46002884-1', 'auto', {'name': 'parallelnetcdfTracker'});
ga('send', 'pageview');
ga('pnetcdfTracker.send', 'pageview');
ga('parallelnetcdfTracker.send', 'pageview');
</script>
</head>
<body>
<h1>PnetCDF: A Parallel I/O Library for NetCDF File Access</h1>
<hr>
<h2 id="News">News</h2>
<ul>
<li><strong>March 29, 2024</strong>: Release of 1.13.0 is available.</li>
<li>Starting from version <a href="https://github.com/Unidata/netcdf-c/blob/v4.4.0/RELEASE_NOTES.md">
4.4.0</a>, the NetCDF library developed at Unidata supports the CDF-5 file
format for both sequential and parallel file access.</li>
<li><a href="wiki/NewsArchive.html">News Archive</a></li>
</ul>
<h2>Overview of PnetCDF</h2>
<p>
PnetCDF is a high-performance parallel I/O library for accessing
<a href="https://www.unidata.ucar.edu/software/netcdf/">Unidata's NetCDF</a>,
files in classic formats, specifically the formats of CDF-1, 2, and 5.
CDF-1 is the default NetCDF classic format. CDF-2 is an extended format
created through using flag NC_64BIT_OFFSET to support 64-bit file offsets. The
CDF-5 file format, an extension of CDF-2 and created through using flag
NC_64BIT_DATA, supports unsigned data types and uses 64-bit integers to allow
users to define large dimensions, attributes, and variables (> 2B array
elements).
</p>
<p>
In addition to the conventional netCDF read and write APIs, PnetCDF
provides a new set of nonblocking APIs. Nonblocking APIs allow users to post
multiple read and write requests first, and let PnetCDF to aggregate them
into a large request, hence to achieve a better performance.
See <a href="wiki/CombiningOperations.html">nonblocking I/O</a> for further
description and example programs.
</p>
<ul>
<li><a href="wiki/Download.html">Software Downloads</a>: latest and
previous software releases, as well as information for accessing the
<a href=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/Parallel-NetCDF/PnetCDF>GitHub repository</a> of
source codes under development.</li>
<li>Documents:
<ul>
<li><a href="wiki/Documentation.html">User Guide</a>, published papers,
presentations, articles, and other resources.</li>
<li><a href="wiki/QuickTutorial.html">Quick tutorial</a></li>
<li><a href="https://cucis.ece.northwestern.edu/projects/PnetCDF/#InteroperabilityWithNetCDF4">
NetCDF-4 example programs</a> that show how to access files through PnetCDF and HDF5 libraries underneath.</li>
<li>PnetCDF <a href="https://cucis.ece.northwestern.edu/projects/PnetCDF/doc/pnetcdf-c/index.html">C Programming Interface Reference</a></li>
<li><a href="https://cucis.ece.northwestern.edu/projects/PnetCDF/faq.html">Q&A</a>
contains some guide lines for I/O performance tuning.</li>
</ul></li>
<li><a href="wiki/Benchmarking.html">Benchmarking</a>: programs and suggestions
for evaluating PnetCDF performance.</li>
</ul>
<h2>A Brief Background About NetCDF</h2>
<p>
NetCDF gives scientific programmers a self-describing and portable means for
storing data. However, prior to version 4, netCDF does so in a serial manner.
</p>
<p>
NetCDF started to support parallel I/O from version 4, whose parallel I/O
feature was at first built on top of parallel HDF5. Thus, the file format
required by NetCDF-4 parallel I/O operations was restricted to HDF5 format.
Starting from the
<a href="https://www.unidata.ucar.edu/mailing_lists/archives/netcdfgroup/2010/msg00061.html">release of 4.1</a>,
NetCDF has also included a dispatcher that enables parallel I/O operations on
files in classic formats (CDF-1 and 2) through PnetCDF. Official support for
the CDF-5 format started in the
<a href="https://github.com/Unidata/netcdf-c/releases/tag/v4.4.0-rc5">release of NetCDF 4.4.0</a>.
</p>
<p>
Note NetCDF now can be built with PnetCDF as its sole parallel I/O mechanism by
using command-line option <strong>"--disable-netcdf-4
--enable-pnetcdf"</strong>. Certainly, NetCDF can also be built with both
PnetCDF and Parallel HDF5 enabled. In this case, a NetCDF program can choose
either PnetCDF or Parallel HDF5 to carry out the parallel I/O by adding
<strong>NC_MPIIO</strong> or <strong>NC_NETCDF4</strong> respectively to the
file open/create flag argument when calling API nc_create_par or nc_open_par.
When using PnetCDF underneath, the files must be in the classic formats
(CDF-1/2/5). Similarly for HDF5, the files must be in the HDF5 format (aka
NetCDF-4 format). A few
<a href="https://cucis.ece.northwestern.edu/projects/PnetCDF/#InteroperabilityWithNetCDF4"> NetCDF-4 example programs</a>
are available that shows parallel I/O operations through PnetCDF and HDF5.
</p>
<h2>A Brief History of PnetCDF</h2>
<p>
The PnetCDF project started in 2001, independently from the Unidata's NetCDF
project. Applications can use PnetCDF library completely without NetCDF
library. The initial goal of PnetCDF is to develop a parallel I/O library for
applications to access CDF-1 and 2 formats on parallel computers. Its focus is
to achieve high I/O performance. The design adopts a new set of APIs (with
prefix name of "ncmpi_") due to its implementation being tightly coupled with
MPI. To encourage adoption from NetCDF users, the syntax of PnetCDF APIs stays
mostly the same as the NetCDF's. Through fully use of existing optimizations
available in MPI-IO implementation, PnetCDF has been demonstrated to be able to
deliver high-performance parallel I/O.
</p>
<h2>A Note About Large File Support</h2>
<p>
The classic
<a href="https://www.unidata.ucar.edu/software/netcdf/old_docs/really_old/guide_15.html">CDF</a>
file format (referred as CDF-1) has been solely in use by NetCDF library
through version 3.5.1. The classic format has been updated by NASA ESDS
community standard and added a support for 64-bit offset file format
(also referred as CDF-2). See
<a href="https://earthdata.nasa.gov/standards/netcdf-classic">NetCDF Classic and
64-bit Offset File Formats</a>.
</p>
<p>
Starting from 3.6.0, the serial NetCDF library added support for the
<a href="https://www.unidata.ucar.edu/software/netcdf/documentation/historic/netcdf/Classic-Format-Spec.html">CDF-2</a>
format. With this format, even 32 bit platforms can create NetCDF files
greater than 2 GiB in size. CDF-2 also allows more special characters in the
name strings of defined dimension, variables, and attributes. The support was
based largely on work from Greg Sjaardema.</p>
<p>
Starting from the release of 0.9.2, PnetCDF supports CDF-2 format. See
<a href="https://github.com/Parallel-NetCDF/PnetCDF/blob/master/doc/README.large_files">README.large_files</a>
for more information.
</p>
<p>
Starting from the release of <a href="Release_notes/1.3.html">1.3.0</a>,
PnetCDF supports
<a href="https://cucis.ece.northwestern.edu/projects/PnetCDF/CDF-5.html">CDF-5</a>
format, an extension of CDF-2 that adds unsigned and 64-bit integer data types
and allows variables to be defined with more than 2<sup>32</sup> array
elements.</p>
<h2>File and Variable Limits</h2>
<p>
Both PnetCDF and NetCDF share limitations on file and variable sizes. More
information can be found on the <a href="wiki/FileLimits.html">FileLimits</a>
page.
</p>
<h2>Required Software</h2>
<p>
PnetCDF requires an MPI implementation with MPI-IO support. Most MPI libraries
have this nowadays. A parallel file system would also go a long way towards
achieving highest performance.
</p>
<h2>Related Projects</h2>
<p>
PnetCDF makes use of several other technologies.
</p>
<ul>
<li><a href="https://www.mcs.anl.gov/romio">ROMIO</a>, an implementation of
MPI-IO, provides optimized collective and noncontiguous operations. It also
provides an abstract interface for a large number of parallel file systems.
</li>
<li>One of those file systems ROMIO supports is
<a href="https://www.pvfs.org">PVFS</a>, a high performance parallel
file system for Linux clusters.
</li>
</ul>
<p>
Today, there are several options for high level I/O libraries. Here are some
discussions on the role of PnetCDF in this ecosystem: </p>
<ul>
<li><a >pnetcdf_vs_hdf5?</a></li>
<li><a >pnetcdf_vs_netcdf4?</a></li>
</ul>
<p>
</p>
<h2>Mailing List</h2>
<p>
We discuss the design and use of the PnetCDF library on the
<tt>[email protected]</tt> mailing list. Anyone interested in
developing or using PnetCDF is encouraged to join. Visit
<a href="https://lists.mcs.anl.gov/mailman/listinfo/parallel-netcdf">the list
information page</a> for details. This mailing list is also for
announcements, bug reports, and questions about PnetCDF software.
</p>
<p>
The URL for the list archive is <a
href="https://lists.mcs.anl.gov/pipermail/parallel-netcdf/">
https://lists.mcs.anl.gov/pipermail/parallel-netcdf/</a>. You can browse even
older mailing list messages at the older
<a href="https://www.mcs.anl.gov/web-mail-archive/lists/parallel-netcdf/threads.html">mailing list archives</a>.
</p>
<h2>Project Members</h2>
<p>
PnetCDF is jointly developed by <strong>Northwestern University</strong> and
<strong>Argonne National Laboratory</strong>.</p>
<ul>
<li>Wei-keng Liao, Kai-yuan Hou, and Alok Choudhary (Northwestern University)</li>
<li>Rob Latham, Rob Ross, and Rajeev Thakur (Argonne National Lab)</li>
<li>Kai-yuan Hou(Ph.D. student at Northwestern, graduated in 2022)</li>
<li>Seung Woo Son (formerly a postdoc at ANL, and then a postdoc at Northwestern, now a Professor at UMass Lowell)</li>
<li>Kui Gao (formerly a postdoc at Northwestern, now Dassault Systèmes Simulia Corp.)</li>
<li>Jianwei Li (Ph.D. student at Northwestern, graduated in 2006, now at Bloomberg)</li>
<li>Bill Gropp (formerly ANL, now UIUC)</li>
</ul>
<h2 id="Citations">Citations</h2>
<p>
The official name of this project is <b>PnetCDF</b> (capital "P", lowercase
"net", followed by capital "CDF"). When referring to the PnetCDF project,
please use the following URL:
</p>
<ul>
<li>https://parallel-netcdf.github.io</li>
</ul>
<p>
If you are looking for a reference to use in a published paper, please cite
our SC2003 paper below. </p>
<ul>
<li>Jianwei Li, Wei-keng Liao, Alok Choudhary, Robert Ross, Rajeev Thakur, William Gropp, Rob Latham, Andrew Siegel, Brad Gallagher, and Michael Zingale. <a href="https://www.computer.org/csdl/proceedings/sc/2003/2113/00/21130039-abs.html">Parallel netCDF: A Scientific High-Performance I/O Interface</a>. In the Proceedings of ACM/IEEE conference on Supercomputing, pp. 39, November, 2003.</li>
</ul>
<p>
The two previous project web pages can still be found.
<ul>
<li><a href=https://trac.mcs.anl.gov/projects/parallel-netcdf>https://trac.mcs.anl.gov/projects/parallel-netcdf</a></li>
<li><a href="https://cucis.ece.northwestern.edu/projects/PnetCDF">https://cucis.ece.northwestern.edu/projects/PnetCDF</a> (a page maintained by Northwestern University)</li>
</ul>
<h2>Acknowledgements</h2>
<p>
Original PnetCDF development was sponsored by the Scientific Data Management
Center (<a href="https://sdm.lbl.gov/sdmcenter/">SDM</a>) under the DOE
program of Scientific Discovery through Advanced Computing (<a
href="https://www.scidac.gov/">SciDAC</a>). It was also supported in part by
</p>
<ul>
<li>National Science Foundation under the SDCI HPC program award numbers
OCI-0724599 and HECURA program award numbers CCF-0938000. </li>
<li>Scientific Data, Analysis, and Visualization (<a
href="https://sdm.lbl.gov/sdav/">SDAV</a>) Institute under the DOE SciDAC
program. </li>
<li>The Exascale Computing Project (<a href="https://exascaleproject.org/">ECP</a>)
under the DOE Office of Science.</li>
</ul>
<p>
Ongoing maintenance is funded by the OASIS Project under the DOE Office of Science.
</p>
<hr>
</body>
</html>