Please review the User Resposibilities: https://www2.cisl.ucar.edu/book/export/html/671
Please contact Davide Del Vento (NCAR CISL User Support Group):
In case your login name on your home computer is the same as the login name at NCAR connect via
ssh -Y bluefire.ucar.edu
In case your login name at NCAR is different, connect via
ssh -Y -l login_name bluefire.ucar.edu
All batch jobs need to specify a project number in their job scripts. On the IBM system Bluefire, the batch queueing system is LSF. It requires the entry
#BSUB -P 35201098 # project number (required)
It lists our DCMIP Project number: 35201098
The jobscript needs to be submitted via the command
bsub < run.lsf
If your job was successfully submitted the message similar to the one below is written to the screen:
Job <951703> is submitted to queue <regular>.
Advice concerning the job queues and LSF settings:
During the summer school we will have access to a special high priority queue. Before the summer school, please take a look at the list of queues:
The typical queue is 'regular' or 'lrg_regular' in case you need more memory on the node (128 GB instead of 64 GB). The 'regular' queue has a 6-hour wallclock limit. Short test jobs might also be submitted to the 'debug' or 'share' queues. The queue is specified via the LSF setting:
#BSUB -q regular # queue
Select the 'Scheduling System' button on the web page listed above to see an example of typical LSF commands that go into the job script.
Starting on 7/23/2012 DCMIP has access to a special queue on both Bluefire and Lynx called 'workshop':
#BSUB -q workshop # DCMIP queue
It has a 6h wallclock limit, and regular memory (i.e. 64GB per node). From 7/23-7/29/2012 the priority will be set to 90 (i.e. low) which should be ok for testing.
During the workshop from 7/30 onwards the priority will be bumped to 600 (i.e. highest than anything else).
Monitor the jobs:
bjobs -u all
bjobs -u <user_name>
bjobs -q workshop
Kill a job:
The jobnumber is dislayed when submitting the bjobs command first.
List of all running and queued jobs:
Overview of the allocated space and disk quotas:
The home directory is located at (10 GB limit)
The scratch space (approx. 25 day lifetime of the files) is
We now (7/10/2012) have access to the DCMIP project space (no automatic deletion after 25 days):
You can create a subdirectory for your model there. Please do not write anything into
which is a designated staging area of quality-controlled data before they get transferred to NOAA ESRL and Earth System Grid.
We will backup the quality-controlled DCMIP netcdf data on NCAR's mass storage system HPSS. Instruction on how to access it are posted on the page:
Logon to the CISL Portal with your login name and one-time password:
and navigate to the GAU usage button
Please check out the information on the web page:
IBM's Symmetric MultiProcessing SMP system, 128 nodes each with:
- 32 procs (on 16 chips)
- 48 nodes have 128GB, 4GB/core (could be undersubscribed, if necessary)
- 69 nodes have 64GB, 2GB/core (could be undersubscribed, if necessary)
- 8 4X InfiniBand DDR links, each capable of 2.5 GB/sec
- 1.3-microsecond MPI latency.
Theoretical total peak computational performance of 77 teraflops.
- IBM XL series with IBM's parallel MPI and OpenMP
- GNU Compiler Collection, including gcc, g++ and gfortran (without MPI)
- IBM XL Fortran for AIX, V12.1 - Version: 12.01.0000.0008
- IBM XL C/C++ for AIX, V10.1 - Version: 10.01.0000.0006
- GNU C, C++, Fortran (GCC) v4.5.1 (OpenMP only, no MPI)
- (some older versions are available as well, need to know in advance if needed)
- 12 procs (2 hex-core AMD 2.2 GHz Opteron, Istanbul chips)
- 16GB of memory, 1.33GB/core (could be undersubscribed, if necessary)
- Cray's SeaStar interconnect
Theoretical total peak computational performance of 8.026 teraflops.
Following compilers (all using Cray's MPI 2.0 and/or Cray SHMEM):
- PGI Programming Environment, including Fortran, C and C++
- EKOPATH (formerly “PathScale”) Compiler suite, including Fortran, C and C++
- Intel compiler suite
- GNU Compiler Collection, including gcc, g++ and gfortran
Compilers versions (all are fully fledged OpenMP and MPI)
We provide wired internet connections for the hands-on sessions in the breakout rooms. In addition, we will have access to a wireless guest network in both the FL auditorium and the breakout rooms. The instructions how to connect to the wireless network will be provided in the welcome package on the first day.
DCMIP-2012: News Report on DCMIP
Check out the NCAR CISL News report http://www2.cisl.... More »
cjablono @ 09/20/2012 10:56 a.m.
DCMIP-2012: Photo Gallery
Check out the photos of the Saturday hike on the ... More »
cjablono @ 08/04/2012 5:12 p.m.
DCMIP-2012: Welcome & Ice Breaker Reception
Monday (7/30/2012) 4:30-6:30pm, NCAR Mesa Lab Tree Plaza More »
murphysj @ 07/27/2012 12:16 p.m.
DCMIP-2012: Pre-Workshop Informal Get-Together
Sunday (7/29/2012) at the Baker Street Pub & Grill near ... More »
murphysj @ 07/27/2012 12:13 p.m.