DrPACS

From CARMA
(Difference between revisions)
Jump to: navigation, search
(Created page with ''''drPACS''' is a simple yet flexible and effective pipeliner for Unix commands.')
 
 
(5 intermediate revisions by one user not shown)
Line 1: Line 1:
'''drPACS''' is a simple yet flexible and effective pipeliner for Unix commands.
+
'''drPACS''' is a simple yet flexible and effective pipeliner for Unix commands. There is a document (currently Appendix A in the PACS cookbook) describing this is more detail,but here is just a teaser to get a feel for what you can do with it.
 +
 
 +
Assuming it has been installed, a '''$DRPACS''' environment variable points to where the code lives, and normally '''$DRPACS/bin''' will have been added to your search path.
 +
 
 +
I keep some more documentation [http://www.astro.umd.edu/~teuben/carma/drpacs/ here]
 +
 
 +
== Simple examples how to start ==
 +
 
 +
'''Step 1''': generate the control Pipefile (hint: it's a Makefile)
 +
 
 +
  pipeline $DRPACS/cat/pipeline.001 > Pipefile
 +
 
 +
  pipeline 4 step1 step2 step3 step4 > Pipefile
 +
 
 +
'''Step 2''': set some parameters
 +
 
 +
  pipepar -c
 +
  pipepar -c project=c0184.3B_108PG2130.13 carmaRefant=2
 +
  pipepar interval=5 carmaRefant=3
 +
 
 +
'''Step 3''': run the pipe
 +
 
 +
  pipe all
 +
  pipe step2
 +
  pipe all
 +
  pipe clean
 +
 
 +
== more advanced examples ==
 +
 
 +
=== Running pipeline in a set of directories ===
 +
 
 +
method 1: old-style c-shell method
 +
 
 +
  foreach dir (`cat dirs.txt`)
 +
    cd $dir
 +
    pipepar foo=3.3
 +
    pipe step3 all
 +
    cd ..
 +
  end
 +
 
 +
method 2: using piperun
 +
 
 +
  piperun dirs.txt 'pipepar foo=3.3 ;  pipe step3 all'
 +
 
 +
 
 +
=== Scatterplot from a set of directories ===
 +
 
 +
  piperun dirs.txt pipepar -v foo -v bar > foo_bar.tab
 +
  tabplot foo_bar.tab
 +
 
 +
=== Parallel pipe ===
 +
 
 +
  piperun -n 4 dirs.txt pipe step2 all
 +
 
 +
=== PACS, in parallel ===
 +
 
 +
  piperun -n 4 -c -o pipe.log dirs.txt 'pipepar -c project=%s showPlots=False; pipe clean all'
 +
 
 +
 
 +
== Advanced usage : persistent pipes ==
 +
 
 +
  pipesetup a=1 b=2 project=test
 +
  pipe all
 +
  pipesave
 +
 
 +
== Advanced usage:
 +
 
 +
== pipe scripts in csh ==
 +
 
 +
 
 +
 
 +
#! /bin/csh -f
 +
#
 +
 
 +
#  (1) define default values in case not given
 +
set a=1
 +
set b=2
 +
 
 +
#  (2) pipeline interface to grab old defaults
 +
pipepar -s csh > tmp$$.par;  source tmp$$.par;  rm tmp$$.par
 +
 
 +
#  (3) poor man's command line processor to override parameters
 +
foreach _arg ($*)
 +
  set $_arg
 +
end
 +
 
 +
#  (4) The Actual Code where the work can be done
 +
echo A=$a  B=$b
 +
 
 +
#  (5) write pipeline parameters back
 +
pipepar a=$a b=$b c=3
 +
 
 +
 
 +
== pipe scripts in python ==
 +
 
 +
#! /bin/env python
 +
#
 +
import parfile, sys
 +
 
 +
a=1
 +
b=2
 +
 
 +
if __name__ == "__main__":
 +
  p = parfile.Parfile('drpacs.def')
 +
  p.argv(sys.argv)
 +
  p.set('a',a)
 +
  if p.has('b'):
 +
      b = p.get('b')
 +
  else:
 +
      p.set('b',b)
 +
  p.set('sum',a+b)
 +
  p.save()

Latest revision as of 07:39, 12 April 2010

drPACS is a simple yet flexible and effective pipeliner for Unix commands. There is a document (currently Appendix A in the PACS cookbook) describing this is more detail,but here is just a teaser to get a feel for what you can do with it.

Assuming it has been installed, a $DRPACS environment variable points to where the code lives, and normally $DRPACS/bin will have been added to your search path.

I keep some more documentation here

Contents

[edit] Simple examples how to start

Step 1: generate the control Pipefile (hint: it's a Makefile)

  pipeline $DRPACS/cat/pipeline.001 > Pipefile
  pipeline 4 step1 step2 step3 step4 > Pipefile

Step 2: set some parameters

  pipepar -c
  pipepar -c project=c0184.3B_108PG2130.13 carmaRefant=2
  pipepar interval=5 carmaRefant=3

Step 3: run the pipe

  pipe all
  pipe step2
  pipe all
  pipe clean

[edit] more advanced examples

[edit] Running pipeline in a set of directories

method 1: old-style c-shell method

  foreach dir (`cat dirs.txt`)
    cd $dir
    pipepar foo=3.3
    pipe step3 all
    cd ..
  end

method 2: using piperun

  piperun dirs.txt 'pipepar foo=3.3 ;  pipe step3 all'


[edit] Scatterplot from a set of directories

  piperun dirs.txt pipepar -v foo -v bar > foo_bar.tab
  tabplot foo_bar.tab

[edit] Parallel pipe

  piperun -n 4 dirs.txt pipe step2 all

[edit] PACS, in parallel

  piperun -n 4 -c -o pipe.log dirs.txt 'pipepar -c project=%s showPlots=False; pipe clean all'


[edit] Advanced usage : persistent pipes

 pipesetup a=1 b=2 project=test
 pipe all
 pipesave

== Advanced usage:

[edit] pipe scripts in csh

#! /bin/csh -f
#
#   (1) define default values in case not given
set a=1
set b=2
#  (2) pipeline interface to grab old defaults
pipepar -s csh > tmp$$.par;  source tmp$$.par;  rm tmp$$.par
#  (3) poor man's command line processor to override parameters
foreach _arg ($*)
 set $_arg
end
#  (4) The Actual Code where the work can be done
echo A=$a   B=$b
#  (5) write pipeline parameters back
pipepar a=$a b=$b c=3


[edit] pipe scripts in python

#! /bin/env python
#
import parfile, sys
a=1
b=2
if __name__ == "__main__":
  p = parfile.Parfile('drpacs.def')
  p.argv(sys.argv)
  p.set('a',a)
  if p.has('b'): 
     b = p.get('b')
  else:
     p.set('b',b)
  p.set('sum',a+b)
  p.save()
Personal tools