Other Data Format proposals

teuben@astro.umd.edu
Fri, 23 Dec 1994 15:30:12 -0500

Hi all,

Quite a few things happened over the past week or so, which made me
post a few thoughts on the subject of the N-body Data Interchange
Format. I've been deliberately silent on my own mail-exploder so far,
but there's been some exciting developments:

I recently learned that Greg Bryan and Frank Summers are also
actively thinking about N-body Data Interchange Formats. This is a
good thing and I was pleased to see their discussion, albeit
a bit late.

If you go to our 'Data Interchange Format' discussion page on the WEB
(http://www.astro.umd.edu/nemo/nbody/) you will see an additional entry
to his current thinking on the subject. I encourage you to
read it, and send any comments back to nbody@astro.umd.edu,
or to Greg himself; the GC3 home page on the WEB
(http://zeus.ncsa.uiuc.edu:8080/GC3_Home_Page.html), has pointers
to their data archive where I found this material.

Although HDF is a very powerful format, my current thinking is still
concentrated towards the FITS BINTABLE (binary tables) format. It has
it's limitations, being a bit old-fashioned in a rather rigid structure.
However, FITS does have some strong advantages, e.g. the dataformat is
described in a journal and a reader can be coded in any language,
widely known and supported in astronomical software packages (I can read
and manipulate my sample Nbody FITS data into e.g. IRAF and MIDAS), NASA
encourages (sometimes forces) to store data in FITS format etc.

Currently, HDF data is only accessible if you get NCSA's library, which
currently supports fortran and C bindings. What the situation will be 10
or 20 years from now, we can only guess. In astronomy we have
traditionally been very good (other sciences, like meteorology, medical
imaging etc. have a large amount of different 'INTERCHANGE' formats).
I would also like to see the interchange format become the de-facto
archive format.

But let me also say there that I see no fundamental differences in the
capabilities of e.g. FITS vs. HDF. (although in the ADASS-III
there is a discussion by Jennings I believe, which I have to
track down - he investigated exactly this problem): as far as I
see the major 'INTERCHANGE' problems are in the area of the
labeling conventions.

Let me again stress one very obvious thing: the interchange format is
meant to reliably interchange data between e.g. the GC3 group, NEMO and
TIPSY software, and what/which-ever package/researcher around the world.
The dataformat you use in your own code does not have to have any
resemblence to this interchange format, but willl frequently have a
rather simple translation.

I also discussed various aspects of the format issues with the
HPCC group at UW in Seattle (George Lake, Neal Katz, Tom Quinn)
but wasn't aware of any of the details of the HDF proposal.
We concluded FITS was oldfashioned (but I still wonder that
given it's robust history we should not stick to something we know
works and is currently widely known and supported) and we could
come up with something TIPSY-like. However, quite some work
is needed to make something robust, extendible, portable etc.
The FITS and HDF people have invested a lot of effort in the
past years on this subject, which we should not easily discard.

Comments back to nbody@astro.umd.edu!

Peter