Globus Online CLI and Advanced Scripting

Tutorial Agenda

Section 1: CLI walkthrough

1.1 Understand the CLI command structure:

1.2 Map your public ssh key to your Globus Online account:

$ open www.globusonline.org/account/ManageIdentities

1.3 Get help:

$ ssh [email protected]
$ help
$ man endpoint-list

1.4 Interact with endpoints you are authorized to use:

$ endpoint-activate nersc#dtn alcf#dtn
$ endpoint-activate xsede#*
$ endpoint-list *#dtn
$ endpoint-list -v -p nersc#dtn
$ endpoint-add -h
$ endpoint-modify -h
$ exit

1.5 Initiate 1TB transfer:

$ echo "alcf#dtn/intrepid-fs0/users/childers/persistent/datasrc/sdata/10Kfiles10G/10-100files10G/ nersc#dtn/project/projectdirs/mpccc1/childers/data/dest/sdata/cdc9/10-100files10G/ -r" | ssh [email protected] transfer
$ ssh [email protected] details

1.6 Find a task by label, not taskid:

$ ssh [email protected] modify --label gw12 taskid
$ ssh [email protected] details -t label=gw12 |more

1.7 Increase the reliability of transfer submissions:

$ ssh [email protected] transfer --generate-id
$ echo "alcf#dtn/intrepid-fs0/users/childers/persistent/datasrc/sdata/10Kfiles10G/9-100files10G/ nersc#dtn/project/projectdirs/mpccc1/childers/data/dest/sdata/cdc9/9-100files10G/ -r" | ssh [email protected] transfer --taskid=taskid
$ echo "alcf#dtn/intrepid-fs0/users/childers/persistent/datasrc/sdata/10Kfiles10G/9-100files10G/ nersc#dtn/project/projectdirs/mpccc1/childers/data/dest/sdata/cdc9/9-100files10G/ -r" | ssh [email protected] transfer --taskid=taskid

1.8 Play with task foregrounding and backgrounding:

$ ssh [email protected] wait taskid
hit ctrl-c
$ ssh [email protected] scp alcf#dtn:/intrepid-fs0/users/childers/persistent/datasrc/sdata/10Kfiles10G/2-100files10G/ nersc#dtn:/project/projectdirs/mpccc1/childers/data/dest/sdata/cdc9/ -r --label=non_detached_scp
hit ctrl-c
$ ssh [email protected] scp alcf#dtn:/intrepid-fs0/users/childers/persistent/datasrc/sdata/10Kfiles10G/2-100files10G/ nersc#dtn:/project/projectdirs/mpccc1/childers/data/dest/sdata/cdc9/ -r -D --label=detached_scp
Note that other supported task types include rm/delete, mkdir, ls, checksum tests
$ open www.globusonline.org/xfer/ViewTransfers

1.9 Examine event logs:

$ ssh -t dtn01.nersc.gov
$ chmod u= /project/projectdirs/mpccc1/childers/data/dest/sdata/cdc9/
$ ssh [email protected] details taskid
$ ssh [email protected] events -f help
$ ssh [email protected] events -f code,time -O kv taskid | tail -20
$ chmod u=rwx /project/projectdirs/mpccc1/childers/data/dest/sdata/cdc9/
$ open a report describing a prior run
$ ssh [email protected] events -f code,time -O kv taskid | tail -40

1.10 Cancel tasks:

$ ssh [email protected] cancel taskid
$ ssh [email protected] events -f code,time -O kv taskid | tail -10
$ ssh [email protected] details taskid
$ ssh [email protected] details -t taskid | more
$ ssh [email protected] details -t -O kv taskid | grep -c "completion_code=CANCELED"
$ ssh [email protected] details -t -O kv -f completion_code,src_file taskid | grep "completion_code=SUCCEEDED" | more
$ ssh [email protected] details -t -O kv -f completion_code,src_file --succeeded taskid | more

1.11 Innoculate your scripts against future GO version changes:

$ ssh [email protected] versions
$ ssh [email protected] v=transfer/1.2:go-dir-xfer/gw11-cli-tutorial-example endpoint-list

Section 2: Example script inspection

$ open www.globusonline.org/demo-script/

Section 3: Port script to your environment

Raise your hand for help

Section 4: Share interesting script ideas and requirements

Share your ideas during the tutorial, in our new Globus Online forum, or by sending email to [email protected]