University of Colorado
The ESMF architecture is based on the concept of components. At its simplest, a software component is a code that has a well-defined calling interface and a coherent function. Component-based design is a natural fit for climate and related models, since components are ideally suited for the representation of a system comprised of a set of substantial, distinct and interacting domains, such as atmosphere, land, sea ice and ocean. Further, since Earth system domains are often studied and modeled as collections of sub-processes - radiation and chemistry in an atmosphere, for example - it is convenient to model such applications as a hierarchy of nested components.
Component-based software is also well-suited for the manner in which multi-component climate and weather models are developed and used. The multiple domains and processes in a model are usually developed as separate codes by specialists. The creation of a viable climate application requires the integration, testing and tuning of the pieces, a scientifically and technically formidable task. When each piece is represented as a component with a standard interface and behavior, that integration, at least at the technical level, is more straightforward. Similarly, standard interfaces help to foster interoperability of components, and the use of components in different contexts. This is a primary concern for modelers, since they are motivated to explore and maintain alternative versions of algorithms (such as different implementations of the governing fluid equations for the atmosphere), whole physical domains (such as oceans), parameterizations (such as convection schemes), and configurations (such as standalone versions of physical domains).
There are two types of components in ESMF, Gridded Components (ESMF_GridComp) and Coupler Components (ESMF_CplComp). Gridded Components represent the scientific and computational functions in a model, and Coupler Components contain the operations necessary to transform and transfer data between them. Both Gridded and Coupler Components are implemented in the Fortran interface as derived types with associated modules. ESMF itself does not currently contain prefabricated Gridded or Coupler Components - the user must write them. The ESMF documentation and source distribution provides tools and examples to guide users through this task.
Each major physical domain in an ESMF climate or weather model is represented as an ESMF Gridded Component with a standardized calling interface and arguments. Physical processes or computational elements, such as radiative processes or I/O, may also be represented as Gridded Components. ESMF Components can be nested, so that parent components can contain child components with progressively more specialized processes or refined grids.
As a climate or weather model steps forward in time, the physical domains represented by Gridded Components must periodically transfer interfacial fluxes. The operations necessary to couple Gridded Components together may involve data redistribution, spectral or grid transformations, time averaging, and/or unit conversions. In ESMF, a Coupler Component encapsulates these interactions.
Coupler Components share the same standard interfaces and arguments as Gridded Components. The key data structure in these interfaces is the ESMF_State object, which holds the data to be transferred between components. Each Gridded Component is associated with an import State, containing the data required for it to run, and an export State, containing the data it produces. Coupler Components arrange and execute the transfer of data from the export States of producer Gridded Components into the import States of consumer Gridded Components. The same Gridded Component can be a producer or consumer at different times during model execution.
There is no single, generic Coupler Component for all ESMF applications. Modelers write Coupler Component internals using ESMF classes bundled with the framework. These classes include methods for time advancement, data redistribution, calculation of interpolation weights, application of interpolation weights via a sparse matrix multiply, and other common functions. ESMF does not currently offer tools for unit transformations or time averaging operations, so users must manage these operations themselves.
Coupler Components can be written to transform data between a pair of Gridded Components, or a single Coupler Component can couple more than two Gridded Components. Multiple couplers may be included in a single modeling application. This is a natural strategy when the application is structured as a hierarchy of components. Each level in the hierarchy usually has its own set of Coupler Components.
It is not necessary to rewrite the internals of model codes to implement coupling using ESMF. Model code attaches to ESMF standard component interfaces via a user-written translation layer that connects native data structures to ESMF data structures. The steps in adopting ESMF are summarized by the acronym PARSE:
In the next two sections, we expand on these steps. The first three steps (PAR) focus on wrapping user code in ESMF Components. The last two (SE) concern coupling components together.
The first step, preparing user code, involves deciding what elements of the application will become Gridded Components. At this time many climate and weather modeling groups wrap major physical domains (land, ocean, etc.) as Gridded Components, and expect to wrap atmospheric physics and dynamics as Gridded Components in the future. A few applications, such as the GEOS-5 model at NASA, wrap sub-processes such as radiation as Gridded Components as well. A key consideration is what elements of the model are expected to be exchanged or used in multiple contexts; these elements are good candidates for component interfaces. Once Gridded Components are identified, the user must split each of them cleanly into initialize, run, and finalize sections, each callable as a subroutine. These subroutines can have multiple phases; for example, run part one and run part two.
This step also involves analyzing the data flow between components: what fields need to be transferred, what transformations are required between components, how frequently fields must be transferred, and what the data dependencies are. This analysis should give the user a good idea of what Coupler Components will be required, and what operations they should contain. In general, this first step takes the longest.
The next step is adapting native data structures to ESMF. Modelers have two main options when deciding on how to wrap their native data structures using ESMF classes. The first option is to implement coupling operations in index-space using the ESMF_Array class. This is a general, multi-dimensional, distributed array structure that can be used to store virtually any kind of real or integer data, and can also represent ghost cells. The user must either copy or reference their native data to an Array object. With this approach, the framework does not hold information about grid coordinates. Interpolation weights must be provided by an external package such as SCRIP. The weights are applied via an Array sparse matrix multiply operation. This option provides a lot of flexibility, but compromises some interoperability since there must be implicit assumptions by the producers and consumers of data about how data is organized and what the data represents.
The second option is to represent native data structures in physical space, by wrapping them in ESMF_Field objects. In this approach the ESMF_Grid class stores coordinate information, and the ESMF_Field class associates the grid with the native data array. Since ESMF holds information about the discretization of the data in physical space, it can calculate the required interpolation weights. ESMF currently support bilinear and higher order interpolation calculations, in up to three dimensions. This strategy is a bit more involved than setting up coupling in index-space using Arrays, since detailed information about component grids must be expressed in framework-defined data structures. However, it enhances interoperability by making the assumptions about how data is organized, and what it represents, more explicit, and readily available to the Coupler Component.
Many applications contain multiple physical fields that share the same physical domain. ESMF Arrays that are distributed in the same fashion, and ESMF Fields that share the same Grid, can be represented in a compact way using ESMF_ArrayBundle and ESMF_FieldBundle objects.
All data exchanged between components is stored in ESMF import and export State objects. These are simple containers that hold ESMF Arrays, ArrayBundles, Fields, and FieldBundles. Once native data structures have been associated with ESMF data types, they must be added by the user to the appropriate State objects. At this point the user code is quite close to the required ESMF interfaces. A remaining task is to wrap native calendar and time information into an ESMF_Clock object. The ESMF Clock holds information about start time, stop time, time step, and calendar type, and enables the user to set alarms related to specific events. The modeler may also choose to use the ESMF_Config object to store configuration parameters. Config is a straightforward utility that enables the application to read labels and values from a text file.
The resulting user component methods, for initialize, run and finalize, have ESMF data structures at the calling interface, and look like this example initialize subroutine:
subroutine myOcean_Init(gridComp, importState, exportState, clock, rc) type(ESMF_GridComp) :: gridComp type(ESMF_State) :: importState type(ESMF_State) :: exportState type(ESMF_Clock) :: clock integer, intent(out) :: rc ! Wrapping layer in which native arrays are extracted from model ! data structures, and referenced or copied into ESMF Arrays, ! Array Bundles, Fields, or Field Bundles. References to these ! objects are then placed into import and export States. ! Scientific content of initialize routine goes here. rc = ESMF_SUCCESS end subroutine myOcean_Init
The third step, registering user methods, is relatively simple. In it the user-written part of a Gridded or Coupler Component is associated with an ESMF_GridComp or ESMF_CplComp derived type through a special SetServices routine. This is a routine that the user must write, and declare public.
Inside the SetServices routine the user calls ESMF SetEntryPoint methods that associate the standard initialize/run/finalize ESMF Component methods with the names of their corresponding user code subroutines. For example, a user routine called myOcean_Init} might be associated with the standard initialize routine for a Gridded Component named myOcean. The sequence of calls is outlined below.
First the Gridded Component is created. This happens in the application layer, one level above the Gridded Component code. This layer may be a relatively small driver program, or an ESMF Component itself. The highest level of a hierarchical ESMF application can be thought of as the ``cap''. Templates and examples are provided within ESMF to show how the driver is structured. The application driver would contain code similar to this:
type(ESMF_GridComp) :: oceanComp oceanComp = ESMF_GridCompCreate(name="myOcean", rc=rc) call ESMF_GridCompSetServices(comp=oceanComp, & subroutineName=mySetServices, rc=rc)
Here mySetServices is the user given name of the public Component routine that is responsible for setting the initialize, run and finalize entry points for oceanComp. If the Fortran subroutine names of the user's initialize, run, and finalize methods were myOcean_Init, myOcean_Run, and myOcean_Final, respectively, the mySetServices method would contain the following calls:
call ESMF_GridCompSetEntryPoint(comp=oceanComp, & subroutineType=ESMF_SETINIT, subroutineName=myOcean_Init, rc=rc) call ESMF_GridCompSetEntryPoint(comp=oceanComp, & subroutineType=ESMF_SETRUN, subroutineName=myOcean_Run, rc=rc) call ESMF_GridCompSetEntryPoint(comp=oceanComp, & subroutineType=ESMF_SETFINAL, subroutineName=myOcean_Final, rc=rc)
These calls link the two pieces of the component: the Gridded Component derived type provided by the framework and the methods provided by the user. The result is that ``myOcean'' model can be dispatched by a driver or by a parent component in a generic way. Note that the create and destroy operations for components are not linked to user code; they act only on the component derived type.
Like the ESMF_GridCompCreate() and ESMF_GridCompSetServices() calls, the initialize, run, and finalize methods are invoked from a driver or parent component using standard ESMF-defined Component methods. They would follow the ESMF\_GridCompSetServices() call shown previously:
call ESMF_GridCompInitialize(gridcomp=oceanComp, ..., rc=rc) call ESMF_GridCompRun(gridcomp=oceanComp, ..., rc=rc) call ESMF_GridCompFinalize(gridcomp=oceanComp, ..., rc=rc)
The omitted arguments indicated by ``...'' are the optional importState, exportState, and clock arguments. The State arguments are necessary to import and export data to and from the Component. The Clock argument provides a means to synchronize the simulation time between different model Components.
A very simple ESMF application might involve an application driver cap, a parent Gridded Component, two child Gridded Components that require inter-component data exchange, and two Coupler Components. Calls cascade so that when, for example, the initialize routine of a parent component is called, it in turn calls the initialize routines of all its children.
The next step, following the ``PARSE'' approach, involves writing the Coupler code to do the scheduling, synchronizing, and sending of data between Components. A sequence similar to that shown for the Gridded Component oceanComp would be followed in order to create and register methods for a second Gridded Component atmComp and a Coupler Component oceanToAtmCpl.
Assuming atmComp needs the temperature Field produced by oceanComp, the Coupler Component is responsible for the correct data flow. If both Gridded Components define the temperature Field on the same physical grid, but with their own custom distribution, a simple Field redistribution can be used. Otherwise, if the physical grids are different, an interpolation is necessary.
The required pre-computations for coupling are typically carried out during the Coupler's initialize phase, storing the complete exchange pattern in an ESMF_RouteHandle object.
type(ESMF_RouteHandle) :: routehandle call ESMF_FieldRedistStore(srcField=oceanTempField, & dstField=atmTempField, routehandle=routehandle, rc=rc)
Here the oceanTempField and atmTempField are Fields from the ocean Component's export State and the atmosphere Component's import State, respectively. The actual data exchange between these Field objects takes place during the Coupler's run phase.
call ESMF_FieldRedist(srcField=oceanTempField, & dstField=atmTempField, routehandle=routehandle, rc=rc)
The last step, execution, combines all the pieces into a complete ESMF application. The sequencing is specified in the application driver or parent Component. In the following example, the ocean is run first; the ocean to atmosphere coupler communicates the ocean export State to the atmosphere import State; the atmosphere runs; the coupler communicates the atmosphere export State to the ocean import State. This loop repeats until the stop time of the Clock is reached.
do while (.not. ESMF_ClockIsStopTime(clock=clock, rc=rc)) call ESMF_GridCompRun(gridcomp=oceanComp, importState=oceanImportState, & exportState=oceanExportState, clock=clock, rc=rc) call ESMF_CplCompRun(cplcomp=oceanToAtmCpl, importState=oceanExportState, & exportState=atmImportState, clock=clock, rc=rc) call ESMF_GridCompRun(gridcomp=atmComp, importState=atmImportState, & exportState=atmExportState, clock=clock, rc=rc) call ESMF_CplCompRun(cplcomp=atmToOceanCpl, importState=atmExportState, & exportState=oceanImportState, clock=clock, rc=rc) call ESMF_ClockAdvance(clock=clock, rc=rc) enddo
Two alternative modes of coupling that have been recently introduced into ESMF are coupling multiple executables and ``direct'' coupling. In order to couple multiple executables - wholly separate programs - ESMF has been collaborating with the InterComm project. Through an extension of ESMF Array class methods, users can translate ESMF Arrays into their InterComm equivalent, and perform sends and receives of data to InterComm applications. This work is currently supporting a space weather application in which an ESMF atmosphere is coupling to a non-ESMF ionosphere and upper atmosphere application via InterComm.
Direct coupling was introduced as a way to initiate a data exchange without needing to first return to a Coupler Component interface. The data exchange is arranged within a Coupler Component, usually at initialization time, but it can be invoked from deep within a Gridded Component. This is useful for many modeling situations, including tightly linked physical processes and asynchronous I/O.