US Reanalysis Workshop Summary

Following the USCLIVAR Reanalysis workshop (Nov 2010, Baltimore MD), a short summary of the proceedings and recommendations was developed. It was published in AGU EOS last week.

The aim of the workshop was to bring together separate disciplines working on data assimilation and reanalysis in separate disciplines, to discuss the needs for and direction to integrated Earth system analysis (IESA). The strengths and weaknesses of the current reanalyses were discussed, as well as potential improvements to each of the major components of reanalyses (model, data assimilation and observations). Despite the current number of reanalyses, they remain in demand, and still have room for improvement, especially in the representation of present climate.

April 2011 Precipitation Extremes

During last April, record or near-record precipitation occurred across a large section of the United States (from the Midwest through the Ohio Valley). This rain, and likely snow melt from the Northern Great Plains, are contributing to the current prolonged and severe flooding along the Mississippi River and its delta. In addition, a prolonged drought of varying degrees persists across the Gulf Coast states.

MERRA precipitation (color shaded) with CPC gauge observations (black contour) time averaged for April 2011. (units: mm/day)

This week, MERRA data for April 2011 was released at the MDISC, roughly two weeks behind real time. Preliminary comparison with the CPC gauge data shows that MERRA precipitation is generally weaker than observed especially in southern Missouri, though the maximum in Pennsylvania is an overestimate. Because the reanalysis system is strongly constrained by observations, the weather systems that produce the rain, and hence the occurrence of rain events, are faithfully reproduced. The physical process of producing the precipitating water then leads to the error in the data product (assuming that the rain gauges capture the extent of the precipitating mass). It is worthwhile to note that MERRA does not assimilate precipitation observations over land, as in NARR.

Giovanni for Quick look and data download

Recently a question about some regional climate data came in, it needed to be independently checked. It seemed like a good opportunity to make use of the Giovanni portal into the MERRA data. Many of the MERRA data collections are available to Giovanni, even 2D one hourly data. MERRA data is directly linked through the Giovanni page, but also from the MERRA data holdings page, which may be more desirable since there is more documentation about what is in each MERRA collection.

Once at the MERRA/Giovanni page, there are options to choose variable(s), region, level and time. There are several hardwired calculation/visualization options in which the data can be presented. Depending on the size of the request, there may be a few minutes before the results are available. However, once they are there is options to download the visualizations, or the data. Both source data and processed data can be retrieved. At this time, HDF, NetCDF and ASCII formats are supported for download.

The FTP Subset is the best way for large requests, and OpenDAP/GDS still serves a purpose for some on the fly live calculation jobs needing flexibility. However, Giovanni fills the middle ground in data visualization and retrieval. It certainly is worth a test drive, and may be a useful option in many cases. Given the range of what can be done with MERRA data, the wide array of ways to access the data should fit most needs.

MERRA Special Collection

Papers are now available at the AMS Online Journals MERRA Special Collection. The MERRA Overview by Rienecker et al. should be considered as the fundamental citation for the MERRA project and data set.

A Community Resource: reanalyses.org

There have been numerous discussions in meetings over the last year or so regarding the multitude of new reanalyses available. These have been generally concerned with users time and effort in deciding which reanalysis best suits any particular users needs. This is mostly true, though some features of reanalyses are equally well represented. The utility of reanalyses is very broad, and has been the key in there continuing development. However, it is nearly impossible for a developing center to sufficiently document all possible features of a reanalysis in reasonable time to provide access to the community. So, there is some effort incurred, for intercomparisons, to be the first researcher on any given topic utilizing all these reanalyses.

It takes time to document a system, but in time we can characterize well the systems. For MERRA, we have produced documentation, the MERRA Atlas, a www page listing the most recent papers (that we know of) and also this blog. However, there is much much more available. How can communicate collective knowledge about all the reanalysis systems? Consider also that communication in this sense is a two-way street. The users of reanalyses have valuable information as well. Complicating matters, we'd like this information with short turnaround time, minimizing duplicative efforts.

While peer review journals provide the formal record, the explosion of internet tools allows for much more rapid transfer of information. Wikipedia has an entry and definition of reanalysis. But the research community has much more demanding needs for knowledge. Given the problem and difficulty of any solution, we can use the internet for documentaiton of not only the developing centers understanding of the various reanalyses, but also the much more vast user community.

So, reanalyses.org has been set up to provide a place for information and research to be posted more quickly than peer review printing (though, early online release that some journals are adopting is a great positive step). This site is intended to be a forum for all aspects of reanalyses and the two way street we need to keep up on the rapid developments. The site is also an experiment, in a way. Consider that it will only grow and thrive with active participation of developing centers, expert researcher and new users! Have a look around reanalyses.org, and if you have some unique comparison or evaluation to share, request an account and add a link or report!

Publications Page growing

With the holidays, travel and deadlines it has been difficult to put together regular snippets of interesting results. The hope was to summarize the papers coming out but there are quite a few and a lot of useful information. I hope to get back to that in the coming weeks.

In the mean time, it is important to share the information in a timely fashion, and the speed that the internet and electronic publishing permits is much greater now than anytime before. The GMAO is collecting information and manuscripts with permission of the authors on our www site. The MERRA Publications page is growing well, most manuscripts are presently submitted for publication and some already accepted. In addition, while most of the papers are written by GMAO staff, quite a few have been authored outside the GMAO, without a GMAO co-author. Most papers have some critical review of the realism of the system. It is important for the development of the systems to account for the strengths and weaknesses, and is a challenge to improve the system while keeping the processes that are already well represented. Shared knowledge would be critical to this development.

Included on the page is the general overview description of the project: Rienecker, M.M., et al., 2011. MERRA - NASA's Modern-Era Retrospective Analysis for Research and Applications. J. Climate (submitted). Check the page linked above for the latest status on the paper. At this time it is still being reviewed.

Latest Data being reprocessed

Earlier this year, a compiler was upgraded on our supercomputing platform. This upgrade has apparently introduced a problem into the post processing program that compresses and prepares data files for transfer to the DISC where the data are publicly accessible. So that: all MERRA data after data date June 1, 2010 will be replaced and is currently no longer accessible at the DISC. This includes Monthly means from May 2010 on, but not the May 2010 data.

For those that may have downloaded data for the period June 1 - August 31, 2010, consider very carefully whether to continue using the data. So far, the only variables we have confirmed are corrupted are the roughness lengths for momentum and heat. We have no reason to think that other variables are not affected, hence the recall. The GMAO does not recommend the use of this data if you downloaded it before this recall.

The MERRA system and archive data are not affected by this issue. The system continues to run in near real time. Once the post processing code is fixed, the archive data will be reprocessed and posted to the DISC.

Late Oct 29 Update: The code has been fixed and tested. Corrected files are being reprocessed and sent to the DISC.

MERRA Climate Atlas

The MERRA Climate Atlas http://gmao.gsfc.nasa.gov/ref/merra/atlas/ has been opened and linked from the homepage. Numerous figures compare MERRA with previous and current reanalyses as well as global observation data when available. As is mentioned in the Introduction, new figures and comparisons are still planned. Comments and suggestions on the Atlas content are welcome.

MERRA Status and some images

Most of the MERRA analysis and results that have come along are intended for publication but not yet submitted. So, we are holding off posting them here until peer reviews are in. Quite a few are going to a special collection at the Journal of Climate (Deadline Oct 30). In addition, between the upcoming IESA workshop in Baltimore (Nov 1-3), AGU Fall Meeting and AMS Annual Meeting, there are a lot of interesting abstract and work to write on. So, expect more posts on the science being done with MERRA in the coming months.

The system continues at about 2 months behind present time available for download.

Here are some nice trajectory images and animations produced at the Scientific Visualization Studio, and written up for the Earth Observatory Image of the day.

Last Call for IESA workshop (more info in previous post)

Evaluation of Reanalyses - Developing an Integrated Earth System Analysis (IESA) Capability November 1-3, 2010, Sheraton Hotel, Baltimore, Maryland (Inner Harbor)

NOTE: The government perdiem rate in Baltimore has dropped to $144 for FY2011. If you have already made a reservation ,your room rate will be adjusted accordingly. If you have any questions or problems reserving at the new rate, you can contact Jill Reisdorf at 303-497-8636 or reisdorf at ucar.edu

Online registration available - abstract deadline 30 September 2010

Agenda: http://www.usclivar.org/Reanalysis2010.php

(Update) USCLIVAR IESA Workshop

Note that Travel Support and deadlines have been included:

Evaluation of Reanalyses - Developing an Integrated Earth System Analysis (IESA) Capability
November 1-3, 2010,
Sheraton Hotel, Baltimore, Maryland (Inner Harbor)
Online registration available - abstract deadline 10 September 2010
Sponsors: NOAA, NASA, NSF, and U.S. CLIVAR

Workshop objectives:
*Assess strengths and limitations of the new recent U.S. reanalyses and suggest where improvements of reanalysis products can be made; identify additional studies necessary to further elucidate the fidelity and usefulness of recent U.S. reanalyses;
*Develop definitions and identify goals of U.S. efforts leading to the forthcoming generation of integrated Earth system analyses (IESA);
*Develop diagnostics to quantitatively assess needed improvements in IESA products; and
*Demonstrate applications of reanalyses in climate and weather that would further highlight needed improvements in reanalysis products

The workshop welcomes contributions in the following areas:
*Assessment of the scientific strengths and limitations of the latest reanalyses, including for example: NASA MERRA, NOAA CFS, NOAA-CIRES 20th Century Reanalysis, NOAA GFDL coupled, SODA, ECCO.
*Applications of and requirements for reanalyses to support climate and weather research
*Efforts to move forward toward integrating (either directly through coupled assimilation or through co-varying products) multiple components of the Earth system (Atmosphere, Ocean, Land, Cryosphere, Chemistry, Carbon, etc.) in reanalyses.

The workshop is intended to bring together reanalysis developers and reanalysis users, and also to bring together various discipline specific analysis systems (including atmosphere, ocean, sea ice, land, and carbon). The anticipated result from the workshop would be improving two-way communications among the reanalysis developers and the climate and weather community, and increasing coordination among developers of the reanalyses for different components.

TRAVEL SUPPORT FOR THE REANALYSIS/IESA WORKSHOP, Nov 1-3, 2010
Please note there is a limited amount of funding available to offset travel to support students, postdocs, and/or early career scientists (receiving a PhD after Jan 1, 2008) from US institutions. If you are interested in support, please provide your name, institution and the type/amount of support you are requesting along with a vita and abstract for presentation at the workshop. A short letter confirming your affiliation should also be provided if you are a student. Please be aware federal employees are NOT eligible to receive funding.
Please submit this information by September 30, 2010.

AGU Session on Reanalyses

Please consider presenting at or attending session A20 "Progress and Uncertainty in Reanalysis Datasets" at the AGU 2010 Fall Meeting , 13-17 December 2010, San Francisco, California, USA.

By synthesizing observations from various conventional and satellite observing systems, reanalysis datasets provide comprehensive long-term descriptions of the climate system that are used in numerous researches and applications. Recently five new global reanalyses (ECMWF ERA-interim, Japanese 25-year ReAnalysis (JRA-25), NASA Modern Era Retrospective-analysis for Research and Applications (MERRA), NOAA-CIRES 20th Century Reanalysis (20CR), NOAA-NCEP Climate Forecast System Reanalysis (CFSR)) based on advanced assimilation systems from different agencies became available to the research community. The goal of this session is to advance the awareness of the progress and uncertainty in these new reanalyses, so as to benefit the research based on reanalysis datasets.

Presentations on the introduction, evaluation, inter-comparison, and application of reanalysis datasets are all welcome.

You may submit your abstract online at:
http://www.agu.org/meetings/fm10/program/abstract_submissions.php
The deadline to submit an abstract is 02 September 2010 — 23:59 Eastern Daylight Time (EDT).

Session A20 convener
Junye Chen, University of Maryland, jchen at essic.umd.edu
Michael Bosilovich, NASA/GMAO, Michael.G.Bosilovich at nasa.gov
Phillip Arkin, University of Maryland, parkin at essic.umd.edu
Wesley Ebisuzaki, NOAA/NCEP, wesley.ebisuzaki at noaa.gov

New results

Sincere apologies for the lack of postings on new results. We are pushing forward with quite a few papers on MERRA, and they are nearly ready for peer review submission. Most of what has been published here before was either accepted, or some straight forward analysis that would likely not be submitted. As the peer reviews come in and these manuscripts get accepted, more results will be shared here.

By the same token, the GMAO is very interested in research using MERRA and plans to host a listing of journal articles using MERRA data. While we scan the journals regularly, we appreciate notification of submitted and accepted papers. There are several contacts through the GMAO web page to do that.

20CR Status

For long term climate applications, 20CR should be a good option. Hopefully we will compare and contrast with MERRA soon. Here is the latest status from 20CR:

14 July 2010

Dear Colleagues,

Global four-times-daily atmospheric and surface fields spanning 1871 to 2008 from the Twentieth Century Reanalysis Project version 2 dataset are now available. They can be obtained in GRIB format via our partners at the National Center for Atmospheric Research http://dss.ucar.edu/datasets/ds131.1, and are also available at http://www.esrl.noaa.gov/psd/data/gridded/data.20thC_ReanV2.html in netCDF format, courtesy of our partners at the NOAA Earth System Research Laboratory, Physical Sciences Division (PSD), and University of Colorado CIRES/Climate Diagnostics Center (CDC). They will also become available via our partners at NOAA’s National Climatic Data Center (NCDC) through the NOAA Operational Model Archive and Distribution System (NOMADS) http://nomads.ncdc.noaa.gov in the near future. Beta composite plotting tools for 4-times-daily, daily and monthly means are now available courtesy of Cathy Smith of NOAA/ESRL/PSD and CU/CIRES/CDC at http://www.esrl.noaa.gov/psd/data/20thC_Rean/.

Additionally, pre-generated synoptic maps showing the ensemble mean analysis and analysis uncertainty of Sea Level Pressure and 500 hPa height over the Northern Hemisphere and Southern Hemisphere and for Sea Level Pressure and 850 hPa zonal wind in the tropics are available at http://www.esrl.noaa.gov/psd/data/20thC_Rean/hem_images.html.

Example animations of the Galveston hurricane of 1900 are available courtesy of NCAR at http://dss.ucar.edu/datasets/ds131.1/docs/galveston/Galveston1900.avi and for the Science on a Sphere visualization platform courtesy of Cathy Smith and Beth Russell at http://sos.noaa.gov/datasets/Atmosphere/reanalysis.html

A brief description of the project is given below. Please also visit the project home page at http://www.esrl.noaa.gov/psd/data/20thC_Rean/. A journal article describing the dataset has been submitted to the Quarterly Journal of the Royal Meteorological Society at their invitation:
Compo, G.P., J.S. Whitaker, P.D. Sardeshmukh, N. Matsui, R.J. Allan, X. Yin, B.E. Gleason, R.S. Vose, G. Rutledge, P. Bessemoulin, S. Brönnimann, M. Brunet, R.I. Crouthamel, A.N. Grant, P.Y. Groisman, P.D. Jones, M.C. Kruk, A.C. Kruger, G.J. Marshall, M. Maugeri, H.Y. Mok, Ø. Nordli, T.F. Ross, R.M. Trigo, X.L. Wang, S.D. Woodruff, S.J. Worley, 2009: The Twentieth Century Reanalysis Project. Quart. J. Roy. Meteor. Soc., submitted. Available online at http://www.esrl.noaa.gov/psd/people/gilbert.p.compo/20CRv2_Compoetal2010.pdf

Please feel free to contact the Project Leads, Gil Compo compo at colorado.edu and Jeff Whitaker Jeffrey.S.Whitaker at noaa.gov, with questions.

USCLIVAR Reanalysis Workshop Nov 1-3

Within the last year or so, and in addition to the availability of MERRA data, ECMWF Interim, NCEP CFSR and NOAA 20CR have made data available. Each system has its own strengths and weaknesses. USCLIVAR has initiated a workshop this year. One of the foci is to evaluate these new reanalyses in a climate sense, but also to determine what is needed for the integration of Earth system components (IESA, Integrated Earth System Analysis). In addition to the atmospheric reanalyses, there will be sessions for various disciplinary reanalyses (ocean, land, cryosphere, chemistry and aerosols), and how to couple them. This should be an excellent opportunity for the research community to interact with the reanalysis developers and begin to define the needs for subsequent reanalyses.

Evaluation of Reanalyses - Developing an Integrated Earth System Analysis (IESA) Capability
November 1-3, 2010,
Sheraton Hotel, Baltimore, Maryland (Inner Harbor)
For registration:
http://www.regonline.com/Checkin.asp?EventId=872029
Additional Information:
http://www.usclivar.org/Reanalysis2010.php

MERRA Downloads

The GES DISC has been tracking MERRA downloads, and there are a few interesting occurrences. In the first 5 months of 2010, the average download is 24Tb per month, and the total for 2010 is nearly equal to all of 2009. Since the purpose of the Data Subsetter is to help reduce the volume for faster downloads (and is the most popular method of access), we also look at number of files. On average, the number of files per month being downloaded in 2010 is 2x that in 2009. This is to be expected. The long time series became connected in August of 2009, and MERRA just caught present in April. Mostly, this is a sign that MERRA is getting out there. It's also not surprising that most downloads are in the US, but the access from the international community is growing as well, and Brazil has downloaded the most MERRA files outside the US to date.

Data through March 2010 is available on line, and MERRA processing has nearly reached the end of May 2010.


Figure 1 MERRA Monthly Downloads in Tb per month.

Figure 2 MERRA downloads in total number of files (in Millions) by country (Top 10).

Converting MERRA data to Grib2

Forwarded from Wesely Ebisuzaki:

Hi,

   I wanted the MERRA dataset (HDF/GDS) in grib2 format so g2grb.gs
was written.
g2grb allows you to write grib2 files from grads.

Testers wanted:
 requirements:  latest version of wgrib2 installed
                know grib2
                send results to wesley.ebisuzaki at noaa.gov

instructions: http://www.cpc.noaa.gov/products/wesley/g2grb.html


          Wesley Ebisuzaki

GEO on Continuing Reanalyses

GEO has published an article supporting the reanalyses development and international collaboration and coordination. Reanalyses represent a significant confrontation between the models and observations, and can be used to improve both, and produce a research quality value-added data product. While CFSR, Interim, 20CR and MERRA are all being released about the same time, they were not planned that way. So, there is a unique opportunity to have a community-wide check on our technological development in the models, assimilation, observations quality and computational resources (which affect the resolution and numerics). Generally, there will be advantages and disadvantages to each of these, which will take some time and effort to sort out. While they will be different, the important point will be is if they are making improvements on the representation of the Earth system.

In order to make progress, continuity and traceability in the system development will be crucial, but can only be attained if the respective centers have regular plans to produce the next round. If too many years go by, the next systems will be so different isolating and understanding the reasons for differences will not be possible. To little time, and the next reanalysis will not represent much difference, and the community will begin to feel the workload updating to the latest. Ideally, the next reanalyses should utilize all the information gained on the quality of the observations from the previous reanalyses, while advancing the representation of physical processes and data assimilation.

Effect of the Eyjafjallajokull eruption

Aircraft observations are a substantial part of the reanalysis system, especially over the oceans and regions where other high altitude conventional observations are infrequent. So that, the eruption of Eyjafjallajokull volcano will have many effects on reanalyses in general. Increased concentrations of aerosols can effect the remote sensing of atmospheric properties (Mount Pinatubo is the typical historical case). The aerosols will affect the radiance observations, and we will be monitoring MERRA's variational bias correction and assimilation of the radiance observations. It's also worthwhile to note that the MERRA system does not include interactive aerosols or aerosol assimialtion (currently under development), but rather incorporates only prescribed mean seasonal cycle variations.

The commercial aircraft restrictions that occurred also have affected the aircraft observing data set.The figures below show the number and locations of aircraft observations in a 6 hour window from ASDAR data system, before and after the major eruption that disrupted air travel.

It appears that the numbers of flights are beginning to recover, and the altitude of the eruption is on the order of 10km, so that the effects should be short lived, assuming the activity subsides.

Before: 12425 obs in 6 hours (green are used in the assimilation, red indicates qc rejection)
After: 6038 obs in 6 hours

MERRA real time production schedule

Stream three has finished March 2010, signifying a transition to real time production schedule. Production will lag real time by approximately 3 weeks, allowing collection of all the observations used in the retrospective periods. We are planning one week to review and quality assure each new month's data files, before being sent to the GES DISC. So, Each new month of MERRA data will be available online roughly 1 month behind present time, and will be released in 1 month segments.

Jan-Mar 2010 are currently being reviewed and should be available on the DISC within the next couple of days.