PACS Season 2 Testing
PACS Data Flow * John Carpenter * The following contents are from an email dated 11/8/2009
PACS data will most likely be collected under the same system as last year. Here are my notes on how the data flow should go from prioritizing time in the project database, to taking the data, and to retrieving the data from the CARMA archive at Illinois. There will be a separate email regarding pacs data analysis and testing.
People with specific action items (listed below) include
1. Project database
ACTION ITEMS: - Determine scheduling priorities (SSC) - Add the A/B internal projects to the project database (Doug) - Add priorities for all A/B projects (Nikolaus?) - Keep track of projects which are selfcal or time-critical (Nikolaus) - All A/B projects should have a single password so that anyone can analyze the pacs data. (Doug)
2. Generate the CARMA/PACS observing script
- PACS observing scripts are generated using the normal web site at http://cedarflat.mmarray.org/observing/scripts/ - For scripts using pacs, the following changes should be made: a) Enter a PACS source for the science target in the "sources" dictionary b) Change record time, if needed, in the "limits" dictionary. Typical values will likely be 4-8 seconds.
- PACS calibrators can be found at http://cedarflat.mmarray.org/cgi-bin/calibrators/calibrators.cgi
ACTION ITEMS: - Re-measure 30 GHz fluxes for potential PACS calibrators. A list of possible calibrators within 5 deg of the proposed science targets have been given to Dan Marrone, who plans to measure them this week. A second iteration of calibrators will likely be needed. - Add updated 30 GHz fluxes to PACS catalog (John) - Add pacs calibrators to sza source catalog (Dan?)
3. Running a PACS script
- As was done last year, PACS observing scripts will be run on cedarflat1 or cedarflat4 using the "pacs" keyword. e.g.run('myscript', pacs=True) - The "pacs=True" executes the command /home/szadaq/sza/bin/szaUCommand command='schedule /home/szadaq/sza/array/sch/carmaStart.sch(obsblockID, combine)' where "obsblockID" is the name of the CARMA obsblock (used only for the SZA log files), and "combine" is the combine time in seconds (equivalent to the CARMA record length). The value of "combine" is set to the record length specified in the CARMA observing script. - The sza data will be recorded in the SZA data format.
ACTION ITEMS: - Make sure that the sza bin directory is up-to-date (Erik) - Add the carmaStart.sch script to the sza schedule directory (Erik)
4. Observing sequence
The CARMA script will observe passband, flux, phase, and source calibrators as before. The SZA will observe the same passband, flux, and phase calibrators as the 6/10-m antennas. While the 6/10-m antennas observe the science target, the SZA will observe the PACS calibrator.
ACTION ITEMS: - Test this observing sequence during maintenance day to make sure it still works as intended (John)
5. Ending a PACS script
When the script ends (either gracefully or script crash), the PACS "cleanup" python function is run, which does the following: - Issues the command /home/szadaq/sza/bin/szaUCommand command='schedule /home/szadaq/sza/array/sch/carmaEnd.sch(obsblockID)' - Send an email message to firstname.lastname@example.org with the subject heading "SZA Miriad Request", and the UT start/stop times for the track in the body of the message. For example, the body of the message will be: 08-nov-2009:00:22:08 08-nov-2009:08:00:00 zrc0504.hltau 1 The last argument is the trial number of the miriad file. - This email will create the miriad file for the sza data. The miriad file will be placed in /misc/sdp/sza. - If a CARMA script is restarted, it will re-send the command to re-fill the SZA data when the script ends. - After the sza miriad file is created, Erik's email parser will call a pacs miriad script that will reduce the data.
ACTION ITEMS: - Add the carmaEnd.sch script to the sza schedule directory (Erik?) - Make sure sza miriad files can be overwritten when the carma scripts are restarted (Erik) - Verify that sza can handle multiple email requests to fill sza data for the same obsblock. This can happen when observers start/restart a script several times in a short time span. What is the desired behavior: queue the sza filler commands, or delete the existing commands in the queue? (Erik) - Need to define pacs reduction script (Maryland's pipeline script?),the output, and where the output goes. Once the script is defined, Erik needs to add the reduction command to his email parser. (Ashley/Alberto/John/Erik) - Could also add the miriad command to the CARMA scripts such that the diagnostic plots are created every ~ 15 minutes. This will permit near real-time feedback for the observers. (John)
6. Data transfer
- The pacs data will be available for the Illinois archive.
ACTION ITEMS - Need to verify this still works (Doug)
7. SZA data
ACTION ITEMS - Add line length data to SZA miriad files (Erik) - Scale SZA data by flux calibration factor so that data in the miriad file are in Janskys (Erik) - The time stamps for the SZA miriad files are incorrect. They need to be for the middle of the integration. (Erik)
Data Reduction/User Notes
NOTE: To add a page of your own, edit this page and add a line
[[Your page name here]]
or click the internal link button: Ab This will bring the link up in red. If you click on the link, a new page will be created and you can start editing it.
Peter will normally update the OVRO and CARMA machines, as well as the 32 bit version ('astroload miriad') and 64 bit version ('astroload -v 64 miriad') on the UMD machines. For anybody else, check the procedure below if that is what you do:
A summary of the software components modified for PACS:
- gpbuddy: writes out (unwrapped) phaseatm from a gain solution (that came from selfcal) [V1.7 and up]
- gplist: handle 23 ants (SZA files now use 23, CARMA still uses nants=15). Can also list subsets of ants now.
- uvcal: has new options=atmcal to apply the atmphase to a dataset. Make sure you have a good freqatm value in your dataset (for now)
- uvwide: SZA data do not have narrow flags, this program was updated for this. [25-nov or later]
- bldelay: baseline based
- panda: baseline based
- maxdim: new program that just displays your miriad version and the MAX parameters. MAXBUF increased.
- selfcal/mselfcal/mfcal: to handle the large number (around 10,000) integrations. maxdim.h also updated, see below. Roundoff issues were uncovered mid-March in (m)selfcal, please make sure you have that version.
Although selfcal works fine, mselfcal crashes on freeing up memory (apr 2, 2009), being looked into.
- listobs: handle SZA data [V1.25 and up]
A simple pipeline has been developed for PACS, called drpacs. A preview can be found in this draft
MIRIAD/SZA data issues
The procedure to convert SZA data into miriad format is still changing, here are some known issues. Most of these are to be resolved in the new filler that writes data from Dec 11 onwards:
Version 1.1 issues (17 Dec and after)
- now have properly dimensioned systemp, though some data don't have systemp (e.g. pacs 19)
- some data create an extra hierarchy data/eml/mirData/ in the current directory. Simple manually move it.
Version 1.1 issues (before Dec 17)
- systemp is erroneously written with the wrong dimension. New version of listobs (V1.26) has been patched to look at wsystemp in this case.
- There are no linelength data (phasem1), but this is a known issue and won't be added for a while.
Version 1.0 issues (before Dec 11)
- Data not always time sorted (solved in 1.1). If the data is not time sorted, mfcal won't work. Rest should be ok. See comments on 3QSO below.
- The antpos uv variable is not correct. (fixed in 1.1)
- The freq variable is missing, listobs now works around that (fixed in 1.1)
- There is no version keyword in the data. (fixed in 1.1)
- Only wide band data are flagged (fixed in 1.1), you must run
uvwide vis=SZA narrow=t
- missing NOISE source data.(zrcx252.Arp220_12CO.1.mir.tar)
- (now resolved) prior to about Dec 3, data had the wrong sign in the U-V coordinates. You can solve that by using scale < 1 in gpbuddy (V1.9). To check if your data has the wrong sign of the phases, check a similarly oriented antenna pair using UV list and see in which quadrant the UV coordinates are, viz.
uvlist vis=SZA select='ant(21)(23),time(15:00,15:01),-source(noise)' uvlist vis=CAR select='ant(2)(4),time(15:00,15:01),-source(noise)'
- systemp's are constant, but this is a feature. Need to double check if the array is properly written as systemp(nants,nwide), and not the other way around.
- 3QSO track (zrct015.3QSO_3C273.3mm.1.mir).
- no wide band data
- first few integrations after a source change are bad.
There are actually two versions of the 3QSO dataset (zrct015.3QSO_3c273.3mm.1.mir.tar.gz and zrct015.3QSO_3C273.3mm.1.mir.tar.gz) that have a 30MB difference in size. The latter (upper case C) one is smaller and sorted in time, but has lost the wide band data (a feature of uvsort). In the following procedure
uvwide vis=zrct015.3QSO_3c273.3mm.1.mir narrow=t uvsort vis=zrct015.3QSO_3c273.3mm.1.mir out=sza1 uvwide vis=sza1 out=sza2 nwide=16
all flags are correct, including the bad data after a source change.
Reminder on updating MIRIAD (old style)
It is assumed you have a CVS enabled release of MIRIAD (i.e. you have files such as $MIR/CVS)
- MAXDIM: two header files describing maximum sizes were updated: $MIRINC/maxdim.h and $MIRINC/maxdimc.h. Edit this files and modify the MAXBUF parameter in both files to at least 10000000 (4 times the amount is the memory used by programs that use MAXBUF).
cd $MIRINC $EDITOR maxdim.h maxdimc.h
and if to properly update the CVS status of these files (they changed on Dec 4, 2008) a one time is needed to old MIRIAD installs:
mkdir tmp mv maxdim.h maxdimc.h tmp cvs update mv tmp/max*h .
- CVS: always query your source tree first and look for C (conflicts) and M (locally modified) files. The U (to be updated) files will be new ones you will get:
cvs -nq update
and if happy with this, issue the real update
- Old style updates: (you can give these commands from any directory and it assumed you have write permission in $MIR)
mirboss mir.install subs prog
for a blanket re-install of the library and programs. You need this if e.g. maxdim.h was changed, and/or your miriad does not use shared libraries. This is also the safest way to update. On most computers this will only take 2-3 minutes. If however you know exactly what the impact of a new file is, you can pick any of the following examples, which will take 2-3 seconds:
mir.subs fitsio.for uvio
mir.prog itemize gpbuddy
Notice a file extension is optional, and multiple arguments can be given. Watch the output carefully for fatal compile or link errors.
Either way, a quick way to check if you had no disastrous compile issues, count the number of binaries in $MIRBIN, there should be about 400:
ls $MIRBIN | wc
The output of a single mir.prog cgdisp command should be clean, and not give any compile or linking problems. If so, you may have either corrupt miriad library, or a misconfigured system to begin with.
- New style updates: (these can take 10 minutes)
cd $MIR make make install
uvwide vis=$VIS narrow=t
fixes the narrow band flags by copying them from the wide band flags. This issue will probably disappear when the SZA data filler is updated to write the narrow flags directly.
- puthd: manually need to put freqatm in the dataset that came out of gpbuddy (phaseatm needs it in uvcal). Ideally selfcal would write the freq0 variable. E.g.
puthd vis=$VIS/freqatm value=30.1 type=double
- gpbuddy time ranges: if carma data (vis=) and sza data (vis2=) do not share the same time range, erroneous phases can be applied. Sampling can be slighly different, but there are caveats to using a long interval for phaseatm: UV variables only remember the past, selfcal gains are interpolated.
Memos and papers
Draft material and figures
The summer 2009 PACS analysis is maintained on Google Sites