TESS:Efficient download

From FITSH
Jump to: navigation, search

An efficient download method for TESS FFIs

Instead of doing a simple

#!/bin/bash

chmod +x tesscurl_sector_1_ffic.sh
./tesscurl_sector_1_ffic.sh

call for these download scripts, implement the following:

#!/bin/bash

sector=1

wget http://archive.stsci.edu/missions/tess/download_scripts/sector/tesscurl_sector_${sector}_ffic.sh

cat tesscurl_sector_${sector}_ffic.sh | \
awk '{ if ( 6<=NF ) print $6; }' | \
sort | \
awk '{ n++; a=a " " $1; if ( n==16 ) { print a; n=0; a=""; } } END { if ( 0<n ) print a; }' | \
while read list; do
       cmd=""
       for fits in $list; do
               test -f $fits && continue
               cmd="$cmd -o $fits https://mast.stsci.edu/api/v0.1/Download/file/?uri=mast:TESS/product/$fits"
       done
       if test -n "$cmd"; then
               t0=`date +%s.%N`
               curl -C - -L $cmd
               t1=`date +%s.%N`
               echo $t0 $t1 | awk '{ printf("Time: %.3f seconds\n",$2-$1); }'
       fi
done

With this optimization, one can achieve a speed-up of a factor of 2.5 ... 3. Just replace the sector=1 line according to the current sector of interest.

Personal tools