Information on the computing environment at NCAR

User Responsibilities on NCAR machines:

Please review the User Resposibilities: https://www2.cisl.ucar.edu/book/export/html/671

User support for the modeling mentors:

Please contact Davide Del Vento (NCAR CISL User Support Group):
E-mail: ddvento@ucar.edu
Phone: 303-497-1233

Practical advice on the computational resources (IBM)

How to logon

In case your login name on your home computer is the same as the login name at NCAR connect via
ssh -Y bluefire.ucar.edu

In case your login name at NCAR is different, connect via
ssh -Y -l login_name bluefire.ucar.edu

GAU computing units

All batch jobs need to specify a project number in their job scripts. On the IBM system Bluefire, the batch queueing system is LSF. It requires the entry

#BSUB -P 35201098                   # project number (required)

It lists our DCMIP Project number: 35201098

Job submission:

The jobscript needs to be submitted via the command

bsub < run.lsf

If your job was successfully submitted the message similar to the one below is written to the screen:

Job <951703> is submitted to queue <regular>.

Advice concerning the job queues and LSF settings:
During the summer school we will have access to a special high priority queue. Before the summer school, please take a look at the list of queues:
http://www2.cisl.ucar.edu/docs/bluefire/running-jobs

The typical queue is 'regular' or 'lrg_regular' in case you need more memory on the node (128 GB instead of 64 GB). The 'regular' queue has a 6-hour wallclock limit. Short test jobs might also be submitted to the 'debug' or 'share' queues. The queue is specified via the LSF setting:

#BSUB -q regular                   # queue

Select the 'Scheduling System' button on the web page listed above to see an example of typical LSF commands that go into the job script.

Special submission queue for DCMIP:

Starting on 7/23/2012 DCMIP has access to a special queue on both Bluefire and Lynx called 'workshop':

#BSUB -q workshop                   # DCMIP queue

It has a 6h wallclock limit, and regular memory (i.e. 64GB per node). From 7/23-7/29/2012 the priority will be set to 90 (i.e. low) which should be ok for testing.
During the workshop from 7/30 onwards the priority will be bumped to 600 (i.e. highest than anything else).

Monitor the jobs:

bjobs -u all

bjobs -u <user_name>

bjobs -q workshop

Kill a job:

bkill <jobnumber>

The jobnumber is dislayed when submitting the bjobs command first.

Helpful commands on the IBM:

List of all running and queued jobs:
batchview

Overview of the allocated space and disk quotas:
spquota

Home, scratch and project space:

The home directory is located at (10 GB limit)
/glade/home/login_name

The scratch space (approx. 25 day lifetime of the files) is
/glade/scratch/login_name

We now (7/10/2012) have access to the DCMIP project space (no automatic deletion after 25 days):
/glade/proj3/mich1098/

You can create a subdirectory for your model there. Please do not write anything into
/glade/proj3/mich1098/noaa_transfer
which is a designated staging area of quality-controlled data before they get transferred to NOAA ESRL and Earth System Grid.

Archiving:

We will backup the quality-controlled DCMIP netcdf data on NCAR's mass storage system HPSS. Instruction on how to access it are posted on the page:
https://www2.cisl.ucar.edu/docs/hpss/hsi-interface

How to monitor the use of GAUs (our computing resource):

Logon to the CISL Portal with your login name and one-time password:
https://cislportal.ucar.edu
and navigate to the GAU usage button

Advice on compiler options and optimization flags:

Please check out the information on the web page:
https://www2.cisl.ucar.edu/docs/bluefire/compiling-and-optimization

Text Editors:

vi
xedit
nedit

Hardware and software characteristics of the High-Performance Computing (HPC) systems:

• Bluefire (IBM Power6 system):

IBM's Symmetric MultiProcessing SMP system, 128 nodes each with:

• 32 procs (on 16 chips)
• 48 nodes have 128GB, 4GB/core (could be undersubscribed, if necessary)
• 69 nodes have 64GB, 2GB/core (could be undersubscribed, if necessary)
• 8 4X InfiniBand DDR links, each capable of 2.5 GB/sec
• 1.3-microsecond MPI latency.

Theoretical total peak computational performance of 77 teraflops.

Compilers:

• IBM XL series with IBM's parallel MPI and OpenMP
• GNU Compiler Collection, including gcc, g++ and gfortran (without MPI)

Compilers versions:

• IBM XL Fortran for AIX, V12.1 - Version: 12.01.0000.0008
• IBM XL C/C++ for AIX, V10.1 - Version: 10.01.0000.0006
• GNU C, C++, Fortran (GCC) v4.5.1 (OpenMP only, no MPI)
• (some older versions are available as well, need to know in advance if needed)

• Lynx, Cray XT5m, 76 compute nodes, each with:

• 12 procs (2 hex-core AMD 2.2 GHz Opteron, Istanbul chips)
• 16GB of memory, 1.33GB/core (could be undersubscribed, if necessary)
• Cray's SeaStar interconnect

Theoretical total peak computational performance of 8.026 teraflops.

Following compilers (all using Cray's MPI 2.0 and/or Cray SHMEM):

• PGI Programming Environment, including Fortran, C and C++
• EKOPATH (formerly “PathScale”) Compiler suite, including Fortran, C and C++
• Intel compiler suite
• GNU Compiler Collection, including gcc, g++ and gfortran

Compilers versions (all are fully fledged OpenMP and MPI)

Internet connections at NCAR:

We provide wired internet connections for the hands-on sessions in the breakout rooms. In addition, we will have access to a wireless guest network in both the FL auditorium and the breakout rooms. The instructions how to connect to the wireless network will be provided in the welcome package on the first day.