PACS

From CARMA
(Difference between revisions)
Jump to: navigation, search
(New/modified software)
(PACS Season 2 Testing)
 
(42 intermediate revisions by 6 users not shown)
Line 1: Line 1:
[http://www.mmarray.org/project/PairedAntennas/ PACS] notes for [[Miriad]], in particular for those maintaining their own MIRIAD version:
+
=PACS Season 2 Testing=
 +
 
 +
'''PACS Data Flow''' * ''John Carpenter'' *  <br><br>
 +
 
 +
***See the Paired Antenna Section of the CARMA Technical pages for task lists and detailed notes.<br>
 +
http://www.mmarray.org/project/PairedAntennas
 +
 
 +
 
 +
General Notes<br><br>
 +
 
 +
How to generate a CARMA/PACS observing script:<br>
 +
  - PACS observing scripts are generated using the normal web site at http://cedarflat.mmarray.org/observing/scripts/
 +
  - For scripts using pacs, the following changes should be made:
 +
    a) Enter a PACS source for the science target in the "sources" dictionary
 +
    b) Change record time, if needed, in the "limits" dictionary. Typical values will likely be 4-8 seconds.<br>
 +
  - PACS calibrators can be found at http://cedarflat.mmarray.org/cgi-bin/calibrators/calibrators.cgi<br>
 +
 
 +
=Results=
 +
 
 +
*[[Science Results]]
 +
*[[Test Results]]
 +
 
 +
=Data Reduction/User Notes=
 +
'''NOTE:''' To add a page of your own, edit this page and add a line<br>
 +
<nowiki>[[Your page name here]]</nowiki><br>
 +
or click the internal link button: <u>'''Ab'''</u>
 +
This will bring the link up in <span style="color: #ba0000">red</span>. If you click on the link, a new page will be created and you can start editing it.
 +
 
 +
 
 +
[[Baseline Issues]]
 +
 
 +
=MIRIAD Routines=
 +
 
 +
This section contains [http://www.mmarray.org/project/PairedAntennas/ PACS] notes for [[Miriad]], in particular for those maintaining their own MIRIAD version:
  
 
Peter will normally update the OVRO and CARMA machines, as well as the 32 bit version ('astroload miriad') and 64 bit version
 
Peter will normally update the OVRO and CARMA machines, as well as the 32 bit version ('astroload miriad') and 64 bit version
Line 12: Line 45:
 
* '''uvcal''': has new options=atmcal to apply the atmphase to a dataset. Make sure you have a good freqatm value in your dataset (for now)
 
* '''uvcal''': has new options=atmcal to apply the atmphase to a dataset. Make sure you have a good freqatm value in your dataset (for now)
 
* '''uvwide''': SZA data do not have narrow flags, this program was updated for this. [25-nov or later]
 
* '''uvwide''': SZA data do not have narrow flags, this program was updated for this. [25-nov or later]
* '''panda''': baseline based (not in miriad yet, but now imminent (Laura))
+
* '''bldelay''': baseline based
* '''bldelay''': baseline based (not in miriad yet, but now imminent (James))
+
* '''panda''': baseline based  
 
* '''maxdim''': new program that just displays your miriad version and the MAX parameters. MAXBUF increased.
 
* '''maxdim''': new program that just displays your miriad version and the MAX parameters. MAXBUF increased.
* '''selfcal/mselfcal/mfcal''': to handle the large number (around 10,000) integrations. maxdim.h also updated, see below.
+
* '''selfcal/mselfcal/mfcal''': to handle the large number (around 10,000) integrations. maxdim.h also updated, see below.  Roundoff issues were uncovered mid-March in (m)selfcal, please make sure you have that version.
 +
Although selfcal works fine, mselfcal crashes on freeing up memory (apr 2, 2009), being looked into.  
 
* '''listobs''': handle SZA data [V1.25 and up]
 
* '''listobs''': handle SZA data [V1.25 and up]
 +
 +
A simple pipeline has been developed for PACS, called '''[[drpacs]]'''. A preview can be found in this [http://www.astro.umd.edu/~teuben/carma/pipeline.pdf draft]
  
 
== MIRIAD/SZA data issues ==
 
== MIRIAD/SZA data issues ==
Line 22: Line 58:
 
The procedure to convert SZA data into miriad format is still changing, here are some known issues. Most of these are to be resolved in the new filler that writes data from Dec 11 onwards:
 
The procedure to convert SZA data into miriad format is still changing, here are some known issues. Most of these are to be resolved in the new filler that writes data from Dec 11 onwards:
  
* There are some issues if the data are not time sorted, the '''uvsort''' might fix that, but it seems to mess up (not copy?) the wide band data? The solution is probably to run '''uvwide''' before '''uvsort'''
+
=== Version 1.1 issues (17 Dec and after)  ===
* The '''antpos''' uv variable is not correct. (''should be fixed'')
+
* now have properly dimensioned '''systemp''', though some data don't have '''systemp''' (e.g. pacs 19)
* The '''freq''' variable is missing, '''listobs''' now works around that
+
* some data create an extra hierarchy '''data/eml/mirData/''' in the current directory. Simple manually move it.
* There is no '''version''' keyword in the data, you simply would have to know from timestamps at what stage this file was written and which problems have been resolved. Carma is at version "1.0.3" now. The cookbook keeps a list what issues there were in earlier versions.
+
 
 +
=== Version 1.1 issues (before Dec 17) ===
 +
 
 +
* systemp is erroneously written with the wrong dimension. New version of listobs (V1.26) has been patched to look at wsystemp in this case.
 
* There are no linelength data ('''phasem1'''), but this is a known issue and won't be added for a while.
 
* There are no linelength data ('''phasem1'''), but this is a known issue and won't be added for a while.
* Only wide band data are flagged, you must run
+
 
 +
=== Version 1.0 issues (before Dec 11) ===
 +
 
 +
* Data not always time sorted (''solved in 1.1'').  If the data is not time sorted, mfcal won't work. Rest should be ok. See comments on 3QSO below.
 +
* The '''antpos''' uv variable is not correct. (''fixed in 1.1'')
 +
* The '''freq''' variable is missing, '''listobs''' now works around that (''fixed in 1.1'')
 +
* There is no '''version''' keyword in the data. (''fixed in 1.1'')
 +
* Only wide band data are flagged (''fixed in 1.1''), you must run
 
       uvwide vis=SZA narrow=t
 
       uvwide vis=SZA narrow=t
* 3QSO track (zrct015.3QSO_3C273.3mm.1.mir)
 
** no wide band data (this may be a result of running uvsort, probably '''uvwide narrow=t''' should have been run first
 
** first few integrations after a source change are bad. this may actually be a result of uvsort, To Be Resolved. It is possible that the SMA routine '''qvack''' can fix this, but keep in mind that the number of bad integrations may be not constant (4,5 and 6 of 2" each have been observed)
 
 
* missing NOISE source data.(zrcx252.Arp220_12CO.1.mir.tar)
 
* missing NOISE source data.(zrcx252.Arp220_12CO.1.mir.tar)
* (''now resolved'') prior to about Dec 3, data had the wrong sign in the U-V coordinates. You can solve that by using scale < 1 in gpbuddy. To check if your data has the wrong sign of the phases, check a similarly oriented antenna pair using UV list and see in which quadrant the UV coordinates are, viz.
+
* (''now resolved'') prior to about Dec 3, data had the wrong sign in the U-V coordinates. You can solve that by using scale < 1 in gpbuddy (V1.9). To check if your data has the wrong sign of the phases, check a similarly oriented antenna pair using UV list and see in which quadrant the UV coordinates are, viz.
 
   uvlist vis=SZA select='ant(21)(23),time(15:00,15:01),-source(noise)'
 
   uvlist vis=SZA select='ant(21)(23),time(15:00,15:01),-source(noise)'
 
   uvlist vis=CAR select='ant(2)(4),time(15:00,15:01),-source(noise)'
 
   uvlist vis=CAR select='ant(2)(4),time(15:00,15:01),-source(noise)'
* systemp's are constant, but this is a feature. Need to double check if the array is properly written as systemp(nants,nwide), and not the other way around.
+
* systemp's are constant, but this is a feature. Need to double check if the array is properly written as systemp(nants,nwide), and not the other way around.  
* There was two versions of a dataset (zrct015.3QSO_3c273.3mm.1.mir.tar.gz and zrct015.3QSO_3C273.3mm.1.mir.tar.gz) that have a 30MB difference in size. The latter (upper case C) one is smaller and sorted in time, but has lost the wide band data. Possibly one of these where uvwide narrow=t should have been run before uvsort.
+
* 3QSO track (zrct015.3QSO_3C273.3mm.1.mir).
 +
** no wide band data
 +
** first few integrations after a source change are bad.
 +
There are actually two versions of the 3QSO dataset (zrct015.3QSO_3c273.3mm.1.mir.tar.gz and zrct015.3QSO_3C273.3mm.1.mir.tar.gz) that have a 30MB difference in size. The latter (upper case C) one is smaller and sorted in time, but has lost the wide band data (a feature of uvsort). In the following procedure
 +
  uvwide vis=zrct015.3QSO_3c273.3mm.1.mir narrow=t
 +
  uvsort vis=zrct015.3QSO_3c273.3mm.1.mir out=sza1
 +
  uvwide vis=sza1 out=sza2 nwide=16
 +
all flags are correct, including the bad data after a source change.
  
 
== Reminder on updating MIRIAD (old style) ==
 
== Reminder on updating MIRIAD (old style) ==
Line 85: Line 135:
  
 
* gpbuddy time ranges: if carma data (vis=) and sza data (vis2=) do not share the same time range, erroneous phases can be applied. Sampling can be slighly different, but there are caveats to using a long interval for phaseatm: UV variables only remember the past, selfcal gains are interpolated.
 
* gpbuddy time ranges: if carma data (vis=) and sza data (vis2=) do not share the same time range, erroneous phases can be applied. Sampling can be slighly different, but there are caveats to using a long interval for phaseatm: UV variables only remember the past, selfcal gains are interpolated.
 +
 +
=Memos and papers=
 +
 +
 +
=Draft material and figures=
 +
The summer 2009 PACS analysis is maintained on [http://sites.google.com/site/pacs2009site/ Google Sites]

Latest revision as of 18:57, 17 November 2009

Contents

[edit] PACS Season 2 Testing

PACS Data Flow * John Carpenter *

      • See the Paired Antenna Section of the CARMA Technical pages for task lists and detailed notes.

http://www.mmarray.org/project/PairedAntennas


General Notes

How to generate a CARMA/PACS observing script:

 - PACS observing scripts are generated using the normal web site at http://cedarflat.mmarray.org/observing/scripts/
 - For scripts using pacs, the following changes should be made:
   a) Enter a PACS source for the science target in the "sources" dictionary
   b) Change record time, if needed, in the "limits" dictionary. Typical values will likely be 4-8 seconds.
- PACS calibrators can be found at http://cedarflat.mmarray.org/cgi-bin/calibrators/calibrators.cgi

[edit] Results

[edit] Data Reduction/User Notes

NOTE: To add a page of your own, edit this page and add a line
[[Your page name here]]
or click the internal link button: Ab This will bring the link up in red. If you click on the link, a new page will be created and you can start editing it.


Baseline Issues

[edit] MIRIAD Routines

This section contains PACS notes for Miriad, in particular for those maintaining their own MIRIAD version:

Peter will normally update the OVRO and CARMA machines, as well as the 32 bit version ('astroload miriad') and 64 bit version ('astroload -v 64 miriad') on the UMD machines. For anybody else, check the procedure below if that is what you do:

[edit] New/modified software

A summary of the software components modified for PACS:

  • gpbuddy: writes out (unwrapped) phaseatm from a gain solution (that came from selfcal) [V1.7 and up]
  • gplist: handle 23 ants (SZA files now use 23, CARMA still uses nants=15). Can also list subsets of ants now.
  • uvcal: has new options=atmcal to apply the atmphase to a dataset. Make sure you have a good freqatm value in your dataset (for now)
  • uvwide: SZA data do not have narrow flags, this program was updated for this. [25-nov or later]
  • bldelay: baseline based
  • panda: baseline based
  • maxdim: new program that just displays your miriad version and the MAX parameters. MAXBUF increased.
  • selfcal/mselfcal/mfcal: to handle the large number (around 10,000) integrations. maxdim.h also updated, see below. Roundoff issues were uncovered mid-March in (m)selfcal, please make sure you have that version.

Although selfcal works fine, mselfcal crashes on freeing up memory (apr 2, 2009), being looked into.

  • listobs: handle SZA data [V1.25 and up]

A simple pipeline has been developed for PACS, called drpacs. A preview can be found in this draft

[edit] MIRIAD/SZA data issues

The procedure to convert SZA data into miriad format is still changing, here are some known issues. Most of these are to be resolved in the new filler that writes data from Dec 11 onwards:

[edit] Version 1.1 issues (17 Dec and after)

  • now have properly dimensioned systemp, though some data don't have systemp (e.g. pacs 19)
  • some data create an extra hierarchy data/eml/mirData/ in the current directory. Simple manually move it.

[edit] Version 1.1 issues (before Dec 17)

  • systemp is erroneously written with the wrong dimension. New version of listobs (V1.26) has been patched to look at wsystemp in this case.
  • There are no linelength data (phasem1), but this is a known issue and won't be added for a while.

[edit] Version 1.0 issues (before Dec 11)

  • Data not always time sorted (solved in 1.1). If the data is not time sorted, mfcal won't work. Rest should be ok. See comments on 3QSO below.
  • The antpos uv variable is not correct. (fixed in 1.1)
  • The freq variable is missing, listobs now works around that (fixed in 1.1)
  • There is no version keyword in the data. (fixed in 1.1)
  • Only wide band data are flagged (fixed in 1.1), you must run
      uvwide vis=SZA narrow=t
  • missing NOISE source data.(zrcx252.Arp220_12CO.1.mir.tar)
  • (now resolved) prior to about Dec 3, data had the wrong sign in the U-V coordinates. You can solve that by using scale < 1 in gpbuddy (V1.9). To check if your data has the wrong sign of the phases, check a similarly oriented antenna pair using UV list and see in which quadrant the UV coordinates are, viz.
 uvlist vis=SZA select='ant(21)(23),time(15:00,15:01),-source(noise)'
 uvlist vis=CAR select='ant(2)(4),time(15:00,15:01),-source(noise)'
  • systemp's are constant, but this is a feature. Need to double check if the array is properly written as systemp(nants,nwide), and not the other way around.
  • 3QSO track (zrct015.3QSO_3C273.3mm.1.mir).
    • no wide band data
    • first few integrations after a source change are bad.

There are actually two versions of the 3QSO dataset (zrct015.3QSO_3c273.3mm.1.mir.tar.gz and zrct015.3QSO_3C273.3mm.1.mir.tar.gz) that have a 30MB difference in size. The latter (upper case C) one is smaller and sorted in time, but has lost the wide band data (a feature of uvsort). In the following procedure

  uvwide vis=zrct015.3QSO_3c273.3mm.1.mir narrow=t
  uvsort vis=zrct015.3QSO_3c273.3mm.1.mir out=sza1
  uvwide vis=sza1 out=sza2 nwide=16

all flags are correct, including the bad data after a source change.

[edit] Reminder on updating MIRIAD (old style)

It is assumed you have a CVS enabled release of MIRIAD (i.e. you have files such as $MIR/CVS)

  • MAXDIM: two header files describing maximum sizes were updated: $MIRINC/maxdim.h and $MIRINC/maxdimc.h. Edit this files and modify the MAXBUF parameter in both files to at least 10000000 (4 times the amount is the memory used by programs that use MAXBUF).
 cd $MIRINC
 $EDITOR maxdim.h  maxdimc.h

and if to properly update the CVS status of these files (they changed on Dec 4, 2008) a one time is needed to old MIRIAD installs:

 mkdir tmp
 mv maxdim.h  maxdimc.h tmp
 cvs update
 mv tmp/max*h .
  • CVS: always query your source tree first and look for C (conflicts) and M (locally modified) files. The U (to be updated) files will be new ones you will get:
 cvs -nq update

and if happy with this, issue the real update

 cvs update
  • Old style updates: (you can give these commands from any directory and it assumed you have write permission in $MIR)
 mirboss
 mir.install subs prog

for a blanket re-install of the library and programs. You need this if e.g. maxdim.h was changed, and/or your miriad does not use shared libraries. This is also the safest way to update. On most computers this will only take 2-3 minutes. If however you know exactly what the impact of a new file is, you can pick any of the following examples, which will take 2-3 seconds:

 mir.subs fitsio.for uvio

and

 mir.prog itemize gpbuddy

Notice a file extension is optional, and multiple arguments can be given. Watch the output carefully for fatal compile or link errors.

Either way, a quick way to check if you had no disastrous compile issues, count the number of binaries in $MIRBIN, there should be about 400:

 ls $MIRBIN | wc

The output of a single mir.prog cgdisp command should be clean, and not give any compile or linking problems. If so, you may have either corrupt miriad library, or a misconfigured system to begin with.

  • New style updates: (these can take 10 minutes)
 cd $MIR
 make
 make install

[edit] Notes

  • uvflag:
   uvwide vis=$VIS narrow=t

fixes the narrow band flags by copying them from the wide band flags. This issue will probably disappear when the SZA data filler is updated to write the narrow flags directly.

  • puthd: manually need to put freqatm in the dataset that came out of gpbuddy (phaseatm needs it in uvcal). Ideally selfcal would write the freq0 variable. E.g.
   puthd vis=$VIS/freqatm value=30.1 type=double
  • gpbuddy time ranges: if carma data (vis=) and sza data (vis2=) do not share the same time range, erroneous phases can be applied. Sampling can be slighly different, but there are caveats to using a long interval for phaseatm: UV variables only remember the past, selfcal gains are interpolated.

[edit] Memos and papers

[edit] Draft material and figures

The summer 2009 PACS analysis is maintained on Google Sites

Personal tools