Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

 

Time step: 2 sec; duration 6 hours.
Forcing: Prescribed surface fluxes and radiative cooling.
Case description: Starts early morning when no clouds present. About 2 hours into simulation, shallow convection develops gradually growing into mid-level convection with the transition to deep convection by the simulation end.


Idealized GATE Simulation of Convection over Tropical Atlantic ("GigaLES")
Based on average forcing and sounding from GATE Phase III observations (30 August - 19 September 1074, Tropical Atlantic);
Forcing: SST, horizontal advective tendencies of s and q; mean wind nudged to observed; radiative heating prescribed; surface fluxes - interactive.
Domain: 2048x2048x256 grid points, or 205x205x27 km3 (horizontal grid spacing 100m);
Vertical grid spacing: 50m below 1km, 50-100m @1-5km; 100m @5-18km; 100-300m above;
Time step: 2 sec, duration: 1 day;
Initialization: random small-amplitude noise in temperature near the surface;


WAJEX Simulation
23 July - 15 September 1999: 52 days, Kwajalein Atoll, Marshall Islands.
Forcing: SST, horizontal advective tendencies of s and q; large-scale vertical velocity; mean wind nudged to observed.
Radiation and surface fluxes - interactive.
Domain: 256x256x64 grid points, or 256x256x27 km3, time step: 10 sec, duration: 52 days

…...

 

How to run SAM?

User's Guide describes how to run the model in more details than this wiki page. It is available under DOC/

SAM6.10_guide.pdf

SAM6.9_guide.pdf

SAM6.7_guide.pdf


Compile the model

0.0 One-time setup

Register in http://www.cesm.ucar.edu/models/register/register.html

Edit ~/.bashrc to include following lines (for CentOS7):

module load intel/17.0.4-fasrc01 impi/2017.2.174-fasrc01 netcdf/4.1.3-fasrc02

0.1 Copy a clean version of SAM to your home directory

SAM6.10.9 with isotope is in ~pblossey/tmp/SAMUW-2018-10-04/

SAM6.9.5 with LPDM is in ~xwei/SAM6.9.5_LPDM/

SAM6.8.2 with LPDM and isotope is in ~dchan/Atmospheric_Models/sam_lpdm1.0_clean/

 

Microphysics with isotope is mainly supported by Peter Blossey.

Lagrangian Particle Dispersion Model (LPDM) is developed by Kuang's group.

Starting 6.10, alternative subgrid-scale (SGS) schemes are available. (see Changes_log/)

Starting 6.9, alternative advection (ADV) schemes are available. (see Changes_log/)

 

[Note from the authors]

SAM only takes 356Mb of your hard drive, and copying will not take a long time.

 

Not sure if the directories in the source code needs to be manually modified in this cleaned version… Check

I don't think so, just copy everything over with no modification

1. Edit SRC/domain.f90 

You can choose whether to do 3D runs, and change the number of grids in horizontal direction and number of layers in vertical direction here: 

      integer, parameter :: YES3D = 1  ! Domain dimensionality: 1 - 3D, 0 - 2D

      integer, parameter :: nx_gl = 384 !384 ! Number of grid points in X

      integer, parameter :: ny_gl = 64  !384 ! Number of grid points in Y

      integer, parameter :: nz_gl = 96

Here, you can also choose to change the number of subdomain in each horizontal direction:

      integer, parameter :: nsubdomains_x  = 32  ! No of subdomains in x

      integer, parameter :: nsubdomains_y  = 2  ! No of subdomains in y

Please note that:

!  * nx_gl and ny_gl should be a factor of 2,3, or 5 (see User's Guide)

!  * if 2D case, ny_gl = nsubdomains_y = 1 ;

!  * nsubdomains_x*nsubdomains_y = total number of processors

!  * if one processor is used, than  nsubdomains_x = nsubdomains_y = 1;

!  * if ntracers is > 0, don't forget to set dotracers to .true. in namelist

!  * nsubdomains_x * nsubdomains_y should be a factor of both nx_gl & ny_gl

[Note from the authors]

nx_gl and ny_gl should be factors of 2,3, or 5 only (see User's Guide)

ncpu|nx_gl & ncpu|ny_gl, or ncpu|nz_gl (see User's Guide)

if 2D case, ny_gl = nsubdomains_y = 1; ncpu|nz_gl (see User's Guide)

nsubdomains_x*nsubdomains_y = total number of processors  

2. To set the Lagrangian particle dispersion model, go to SRC/lpdm_mod.f90

 integer, parameter :: lpdm_top0  = 70  ! top level below which particles are initially released, mirror reflection b.c.

 integer, parameter :: lpdm_num   = 700 ! 4000  !900!960  ! particles in one column, theoretically, lpdm_num should

be divided exactly by lpdm_top0, as particles are uniformly distributed over the entire domain if height is used as the vertical coordinate.

 

To run lpdm model, you need to specify parameters in the prm file under your specific case directory

lpdm_do =  .true ,  ! true or false, set it to be true if you want to include particle model

lpdm_ndt = 4,   ! frequency to output particle location

lpdm_sub=64,  ! total number of processors used, should be consistent with the total number of cores specified

[Note from the authors]

LPDM model is not a built-in SAM module, it is developed by Kuang group and available upon request if you want to use a newest SAM version.

To support lpdm, nx_gl and ny_gl has the upper limit of 999

To support lpdm, nsubdomain_x *nsubdomain_y has the upper limit of 214 (or edit source code)


For the current LPDM model, it does not have the restart function, I will add this function in the future. Therefore if your run stops

before it finishes, you will need to start from the beginning. REMEMBER: Remove the existing lpdm output first!!

3. Delete the Object files 

 

rm OBJ/*

4. In the Build file, one can:

a. Set the output directory:

setenv SAM_SCR /n/home10/dchan/holy_kuang/SAM_output/   #TODO

b. Choose the radiation and microphysics scheme:

# ----------------------------------

# specify radiation directory in SRC

setenv RAD_DIR RAD_CAM

#setenv RAD_DIR RAD_RRTM

 

 

# specify microphysics directory in SRC

 setenv MICRO_DIR MICRO_LINISO

# setenv MICRO_DIR MICRO_THOMPISO

# setenv MICRO_DIR MICRO_SAM1MOM

# setenv MICRO_DIR MICRO_M2005

c. [Note from the authors]

For case that has multiple source codes, choose the version you intend to use:

setenv SAM_DIR  `pwd`

setenv SAM_OBJ  $SAM_SCR_OBJ/OBJ               # todo

setenv SAM_SRC  `pwd`/SRC                      # todo

5. Build the model using the command:

./Build

[Note from the authors]

a. The compelling does not take more than 3 minutes to finish on Odyssey

b. This step will return an exe file: 

 

SAM_[Name of Radiation Scheme]_[Name of Microphysics Scheme]

For example SAM_RAD_CAM_MICRO_M2005

c. Whenever the source code has been modified, rebuild of the model is required, except for the customized modules (e.g. lpdm).



Setup a Case and Run

Cases are directly listed under the SAM directory, for example, those highlighted yellow below: 

ARMIOP  BOMEX  CaseName  DYNAMO  Makefile  OBJ      RCE   RUNDATA  SRC_ISO         

 

bin     Build  DRYPBL    logs    MC3E      OBJ_ISO  RICO  SRC      runscript.fast


1. To set the case name, go to file CaseName, for example we use RCE

RCE


2. To make your case, first go to the directory of the case to run (here RCE)

A case folder should have these following files:

grd  lsf lst  prm rad sfc snd  trc

 

 grd - specifies the vertical grid. This file contains the height of the scalar levels in meters starting from the levels near the surface. You don’t have to specify all the levels though. The algorithm will continue building the grid levels above the last one specified in the grd file assuming the grid spacing between two last specified levels to be a spacing between all the grid level above the last one specified in the grd file. 


 

prm - the namelist for the model parameters.  


snd – initial-sounding file. This file contains soundings for several time points. The model will linearly interpolate the initial sounding in time; therefore, at least two time samples are needed. If only one sounding is available, still, the snd file should contain at least two identical soundings separated in time for the model to initialize correctly (see the BOMEX case, for example). 


lst - the file that specifies which horizontally averaged statistics should be saved into the statistics file. Only the fields flaged by 1 in the beginning of each line will be computed and saved. Don’t change the second numbers though. The statistics file for each case will be written in the case directory and will have an extension .stat . The file name will be as ‘case-directory-name’_’caseid’.stat, where caseid is the string identifier of the case set up in the prm namelist file (see below). It can later be converted into the netcdf file using the stat2nc utility contained in the UTIL directory (that directory’s source files should also be compiled using the supplied Makefile and GNU-make command directly).  


While the files above are absolutely essential, the following files are optional in the sense that, in principle, the model can be configured to run without them:


lsf - large-scale forcing file. It specifies the large-scale temperature and vapor tendencies, as well as large-scale wind that simulated mean wind may be nudged (relaxed) to. (snd will be overwritten by lsf)

 

 

sfc – contains the time evolution of prescribed SST, and surface fluxes. 

 
 

rad - prescribed radiation heating rates.


trc - trace gases; used only when interactive radiation is used.

For more details, please see the User's Guide.


3.  sbatch runscript.fast

Make sure that the multiple of nsubdomain_x and nsubdomain_y matches the total number of cores when submitting your job.

Post Processing

Copy files in /n/home10/dchan/bin directory to your $HOME/bin directory

bin2D2nc  bin2D2nc_all.sh  bin3D2nc bin3D2nc_all.sh  stat2nc


1. To convert the 3D output files into NC files

cd [output directory]/OUT_3D 

nohup bin3D2nc_all.sh *.bin3D &


# Merging variables into one file:
ncrcat -v QN *nc 20180417_QN.nc


Troubleshooting1: coordinate variables will be mixed, if the version of bin3D2nc and the version of SAM is not the same.

Troubleshooting2: for data >2G, some of the variables might be missed. Solution: editting UTIL/SRC/bin3D2nc.f. Around line 135:
Original:    err = NF_CREATE(filename, NF_CLOBBER, ncid)
Change to:     err = NF_CREATE(filename, OR(NF_CLASSIC_MODEL,NF_NETCDF4), ncid)
(Reference: https://www.unidata.ucar.edu/software/netcdf/fortran/docs/nc_f77_interface_guide.html#f77_NF_CREATE)
cd back to UTIL/, type "make" to recompile. Make sure you overwrite your old bin3D2nc with the new one.

2. To convert the STAT files into NC files 

 cd  [output directory]/OUT_STAT

stat2nc RCE_trail.stat

3. LPDM: convert outputs into Matlab .mat files

Here is a Matlab function for reference:

 

function function_WND_convert_lpdm(seg,dir_load,Casename,caseid,ttt)
   addpath('/n/home10/dchan/Matlab_Tool_Box/');

   for i = 1:numel(ttt)


   tt = ttt(i);
   disp(['starting tt: ' num2str(tt-1)]);


   dir_save = dir_load;
   cmon = '0000';
   cmon(:,end-size(num2str(tt-1),2)+1:end) = num2str(tt-1);
   fnn1=[dir_load,Casename,'_',num2str(caseid),'_xyz',cmon];
   fid1=fopen(fnn1);
   clear('xyz','x','y','z');

   fread(fid1,1,'int32');

   xyz =fread(fid1,seg,'int32');
   fread(fid1,1,'int32');
   clear('x','y','z')
   z = rem(xyz,1000);
   xyz = fix (xyz/1000);
   y = rem(xyz,1000);
   xyz = fix (xyz/1000);
   x = xyz;

   disp (['Saving! Time Step ',num2str(tt-1),' ...']);
   clear('position')
   position(:,1) = x;
   position(:,2) = y;
   position(:,3) = z;
   save([dir_save,Casename,'_',caseid,'_time',num2str(tt-1),'.mat'],'position','-v7.3');
   fclose(fid1);
   end
end

Check what you get!

Randomized convection and cold pools in an RCE setup

Advanced modifications

To Create Warm or Cold Bubbles 

For some idealized convection experiments, you want to initialize a warm or cold bubble within the domain, SAM has the capacity to do so easily, the bubble shape can be specified in SRC/setperturb.f90

Then in your prm namelist, change the following things:

         perturb_type =2., ! change perturb_type to indicate this will be a bubble perturbation

         bubble_x0 =0., ! center of bubble in the x direction

         bubble_y0 =0., ! center of  bubble in the y direction

         bubble_z0 =1000., ! center of bubble in the z direction

         bubble_radius_hor = 2000., ! horizontal length scale

         bubble_radius_ver = 500., ! vertical length scale

         bubble_dtemp = 1.0, ! a cold/warm bubble

         bubble_dq= 0.2*1e-3 ! a moist/dry bubble

You can release many bubbles at the same time by including a customized module in the SRC code.

To Include Customized Modules

If you want to include your customized module, for example, customized_mod.f90 to achieve more specific goals, add this to your SRC code. A few more things need to be done:

a. Include this new module in your SRC/main.f90

b. Define all new variables in the SRC/setparm.f90

c. Specify variable types in SRC/vars. f90 if needed

d. Modify SRC/write_fields2D.f90, SRC/write_fields3D.f90 and statistics.f90 files if you want to output those new variable values

Source Code Framework

(Briefly describe the framework of the source codes, basically in the order of when they are called)

SAM uses moist static energy and total water mixing ratio as prognostic variables.

Before looping

1. main.f90: This is the main file. It loads parameters, claims variables, set up initial conditions & forcings and loops over time to integrate forward. All other f90 files (subroutines, variable claims) are called in main.f90. 

2. setdata.f90: Initialize the soundings, calculate the pressures and heights of the vertical grids. Print out the one-dimension vertical structure of height, rho, pressure, qt etc. in the log file. It will use the 1976 Standard Atmosphere if soundings are missed.

3. diagnose.f90: If the case run is restarted from a run before, diagnose.f90 uses the information from restart file to deduce other variables such as temperature, liquid water content. SAM uses a all-or-nothing microphysics scheme. There is no super saturation.

4. read_all.f90: Read restart file in ../RESTART/. The new run will start from the end time of the old one regardless of what day you set in prm.

5. set_forcing.f90: Read forcings (including nudging profile from snd, large-scale tendencies from lsf and surface forcing from sfc).

When Looping

6. kurant.f90: Decide the size of the current time step from the KFL condition. If dt/4 from your prm file is larger than KFL requires, SAM will exit.

7. buoyancy.f90: Update the vertical velocity by buoyancy acceleration.

8. forcing.f90: Apply forcings to the prognostic variables. It will call subsidence.f90 if wls in lsf file is not zero. subsidence.f90 calculates the vertically advective tendencies of MSE and Qt from large-scale vertical velocity virtually and with which updates MSE and Qt.

9. nudging.f90: Nudge MSE, Qt, u and v to the corresponding sounding from snd file by the rates set in prm (tauls).

10. damping.f90: Suppress turbulence for one third layers in the top.

11. ice_fall.f90: Cloud ice will fall down with speed suggested by Heysfield (2003).

12. advect_mom.f90: Momentum advection.

13. surface.f90: Applies surface heat flux, moisture flux and drag to the lowest layer.

   Options in prm file.

(1) LES, CEM: By choosing LES the surface is assumed homogeneous. CEM, on the contrary, calculates surface flux point by point with velocity at the corresponding point of the lowest layer.

(2) OCEAN, LAND: Specifies the type of the surface.

(3) SFC_FLX_FXD: If yes, then surface fluxes are read from sfc file. If not, they are calculated from by oceflx.f90 or landflx.f90.

14. coriolis.f90: Applies coriolis forcing to u, v, w. The coriolis parameter is decided by lat from prm file.

15. ADV_MPDATA/advect_scalar3D.f90: Advects scalars (Qt, MSE) with anti-diffusive flux scheme.

16. MICRO_SAM1MOM/microphysics.f90:

Subroutine micro_precip_fall(): Assign different terminal speeds to rain, snow and graupel.

Subroutine micro_proc(): Calculates the transfer rate from cloud droplet to rainfall.

17. radiation.f90: If dolongwave or doshortwave is true, this will be executed. Otherwise if doradforcing is true, rad file will be used.

18. stepout.f90: Prints out the stat file, 2D and 3D file, and updates the restart file.