Output Data Review

We are currently reviewing the MERRA File Specification Document, and considering new variables and diagnostics that should be incorporated into future experiments and reanalyses. This is directed toward information that needs to be captured during run time, as opposed to diagnostics that can be post processed from existing data. For example, to get Max/Min temperature from MERRA would be the maximum/minimum hourly averaged temperature, in the current configuration. We will implement for future analyses hourly maximum and hourly minimum temperatures, so that the instantaneous Max/Min T can be captured from the system.

Any suggestions would be welcome. Please include any computations that may be non standard, and reasons or applications of the suggested diagnostic.

Apologies for letting the blog become stagnant. While I have seen a lot of interesting work with reanalyses and MERRA, finding the time to work up a worthwhile post is remarkably challenging.

Reanalysis Conference Report

The Report of the 4th International Conference on Reanalyses (May 7-11, 2012, Silver Spring MD) is presently available from the World Climate Research Programme.

Return of the MERRA Blog

As many readers will understand, the demands on time can be many, and there is never enough time to do all that you want to do. Being spread thin for some time, the MERRA blog fell into neglect. However, with a lot of interesting things going on, from research with MERRA to formulating plans for subsequent reanalyses (and noticing in the last month a fair number of hits from around the world), I will be trying hard to find time to make regular posts.

The original purpose of the blog was to 1) follow the development of the MERRA system and eventually the production of data and then 2) follow the research that was being done with MERRA and other reanalyses. #2 never fully materialized, but there is a lot of work being published on reanalyses lately.  So, for the near future, summaries of published research or perhaps topical discussions involving several papers/reports will likely be the regular topic. However, new reanalysis development is on the horizon, as well.

Mostly, I figure that the hits on the MERRA blog are coming from internet searches for hard to find information, now that the MERRA overview paper is printed. This is an important issue, as users need to know if a reanalysis is applicable to a research topic, and its strengths and weaknesses. So, perhaps this can open the communication, just a little more.

MERRA Special Collection

Papers are now available at the AMS Online Journals MERRA Special Collection. The MERRA Overview by Rienecker et al. should be considered as the fundamental citation for the MERRA project and data set.

Latest Data being reprocessed

Earlier this year, a compiler was upgraded on our supercomputing platform. This upgrade has apparently introduced a problem into the post processing program that compresses and prepares data files for transfer to the DISC where the data are publicly accessible. So that: all MERRA data after data date June 1, 2010 will be replaced and is currently no longer accessible at the DISC. This includes Monthly means from May 2010 on, but not the May 2010 data.

For those that may have downloaded data for the period June 1 - August 31, 2010, consider very carefully whether to continue using the data. So far, the only variables we have confirmed are corrupted are the roughness lengths for momentum and heat. We have no reason to think that other variables are not affected, hence the recall. The GMAO does not recommend the use of this data if you downloaded it before this recall.

The MERRA system and archive data are not affected by this issue. The system continues to run in near real time. Once the post processing code is fixed, the archive data will be reprocessed and posted to the DISC.

Late Oct 29 Update: The code has been fixed and tested. Corrected files are being reprocessed and sent to the DISC.

MERRA Climate Atlas

The MERRA Climate Atlas http://gmao.gsfc.nasa.gov/ref/merra/atlas/ has been opened and linked from the homepage. Numerous figures compare MERRA with previous and current reanalyses as well as global observation data when available. As is mentioned in the Introduction, new figures and comparisons are still planned. Comments and suggestions on the Atlas content are welcome.

New results

Sincere apologies for the lack of postings on new results. We are pushing forward with quite a few papers on MERRA, and they are nearly ready for peer review submission. Most of what has been published here before was either accepted, or some straight forward analysis that would likely not be submitted. As the peer reviews come in and these manuscripts get accepted, more results will be shared here.

By the same token, the GMAO is very interested in research using MERRA and plans to host a listing of journal articles using MERRA data. While we scan the journals regularly, we appreciate notification of submitted and accepted papers. There are several contacts through the GMAO web page to do that.

20CR Status

For long term climate applications, 20CR should be a good option. Hopefully we will compare and contrast with MERRA soon. Here is the latest status from 20CR:

14 July 2010

Dear Colleagues,

Global four-times-daily atmospheric and surface fields spanning 1871 to 2008 from the Twentieth Century Reanalysis Project version 2 dataset are now available. They can be obtained in GRIB format via our partners at the National Center for Atmospheric Research http://dss.ucar.edu/datasets/ds131.1, and are also available at http://www.esrl.noaa.gov/psd/data/gridded/data.20thC_ReanV2.html in netCDF format, courtesy of our partners at the NOAA Earth System Research Laboratory, Physical Sciences Division (PSD), and University of Colorado CIRES/Climate Diagnostics Center (CDC). They will also become available via our partners at NOAA’s National Climatic Data Center (NCDC) through the NOAA Operational Model Archive and Distribution System (NOMADS) http://nomads.ncdc.noaa.gov in the near future. Beta composite plotting tools for 4-times-daily, daily and monthly means are now available courtesy of Cathy Smith of NOAA/ESRL/PSD and CU/CIRES/CDC at http://www.esrl.noaa.gov/psd/data/20thC_Rean/.

Additionally, pre-generated synoptic maps showing the ensemble mean analysis and analysis uncertainty of Sea Level Pressure and 500 hPa height over the Northern Hemisphere and Southern Hemisphere and for Sea Level Pressure and 850 hPa zonal wind in the tropics are available at http://www.esrl.noaa.gov/psd/data/20thC_Rean/hem_images.html.

Example animations of the Galveston hurricane of 1900 are available courtesy of NCAR at http://dss.ucar.edu/datasets/ds131.1/docs/galveston/Galveston1900.avi and for the Science on a Sphere visualization platform courtesy of Cathy Smith and Beth Russell at http://sos.noaa.gov/datasets/Atmosphere/reanalysis.html

A brief description of the project is given below. Please also visit the project home page at http://www.esrl.noaa.gov/psd/data/20thC_Rean/. A journal article describing the dataset has been submitted to the Quarterly Journal of the Royal Meteorological Society at their invitation:
Compo, G.P., J.S. Whitaker, P.D. Sardeshmukh, N. Matsui, R.J. Allan, X. Yin, B.E. Gleason, R.S. Vose, G. Rutledge, P. Bessemoulin, S. Brönnimann, M. Brunet, R.I. Crouthamel, A.N. Grant, P.Y. Groisman, P.D. Jones, M.C. Kruk, A.C. Kruger, G.J. Marshall, M. Maugeri, H.Y. Mok, Ø. Nordli, T.F. Ross, R.M. Trigo, X.L. Wang, S.D. Woodruff, S.J. Worley, 2009: The Twentieth Century Reanalysis Project. Quart. J. Roy. Meteor. Soc., submitted. Available online at http://www.esrl.noaa.gov/psd/people/gilbert.p.compo/20CRv2_Compoetal2010.pdf

Please feel free to contact the Project Leads, Gil Compo compo at colorado.edu and Jeff Whitaker Jeffrey.S.Whitaker at noaa.gov, with questions.

USCLIVAR Reanalysis Workshop Nov 1-3

Within the last year or so, and in addition to the availability of MERRA data, ECMWF Interim, NCEP CFSR and NOAA 20CR have made data available. Each system has its own strengths and weaknesses. USCLIVAR has initiated a workshop this year. One of the foci is to evaluate these new reanalyses in a climate sense, but also to determine what is needed for the integration of Earth system components (IESA, Integrated Earth System Analysis). In addition to the atmospheric reanalyses, there will be sessions for various disciplinary reanalyses (ocean, land, cryosphere, chemistry and aerosols), and how to couple them. This should be an excellent opportunity for the research community to interact with the reanalysis developers and begin to define the needs for subsequent reanalyses.

Evaluation of Reanalyses - Developing an Integrated Earth System Analysis (IESA) Capability
November 1-3, 2010,
Sheraton Hotel, Baltimore, Maryland (Inner Harbor)
For registration:
http://www.regonline.com/Checkin.asp?EventId=872029
Additional Information:
http://www.usclivar.org/Reanalysis2010.php

MERRA real time production schedule

Stream three has finished March 2010, signifying a transition to real time production schedule. Production will lag real time by approximately 3 weeks, allowing collection of all the observations used in the retrospective periods. We are planning one week to review and quality assure each new month's data files, before being sent to the GES DISC. So, Each new month of MERRA data will be available online roughly 1 month behind present time, and will be released in 1 month segments.

Jan-Mar 2010 are currently being reviewed and should be available on the DISC within the next couple of days.

Update to Data Streams

We are updating MERRA data to permit longer spinup of streams 2 and 3. The announcement and how it affects data downloads follows. Data files formally available in the MERRA data streams are being renamed "SPINUP_*" and will also be available for download. Most research and applications for the data will only require use of the "Mainstream" MERRA data.

Updating of the MERRA data products site to contain only Mainstream data (http://disc.sci.gsfc.nasa.gov/mdisc/data-holdings/merra-mainstream-and-spinup-data) will begin Thursday, March 11, 2010. A two step procedure will be followed to achieve this. In step one, all current  products for the years 1989 through 1992 and 1998 through 2000 will be deleted. It is strongly advised that the MERRA data for the two aforementioned periods not be accessed until this data update is complete.  In step two,  
 products will be used to repopulate these data years. We anticipate that it will take about two weeks to complete the data update. The remaining year
s of the data and all the MERRA data services will still be available during this time period. Once the update activities are completed, a new notification will be sent out and posted on the web. We thank you for your patience during this period.  If you have any questions or concerns please send an email to the GES DISC User Services (help-disc@listserv.gsfc.nasa.gov).


So when the update is complete, Stream 1 will contain 1979 through 1992, Stream 2 will have 5 years spinup (4 available for download) containing 1993 through 2000, and stream 3 will have 4 years spinup (3 available for download) containing 2001 through present.

2009 data should become available during the week of March 15.

MERRA Turns 30

Data availability at the DISC now extends for thirty years 1979 through 2008. This data is all online, and open to access. Production is presently in May 2009, and so should catch up to real time around the end of February. MERRA will continue into the future, as long as it is feasible to do so. At this point, we do not have a set production terminus.

CFSRR

Some News which may interest those who read these pages:

On 1/14/2010 6:16 PM, Suranjana.Saha@noaa.gov wrote:
> Dear Colleagues
>
> I am happy to inform you that the CFS Reanalysis is now complete for 31
> years (1979-2009) and the paper has just been submitted to BAMS.
>
> You can download it at the CFS website at:
>
> http://cfs.ncep.noaa.gov
>
> The first set of CFSR data will be made available to the public at NCDC
> and NCAR on Feb 1, 2010. The complete high resolution datasets will be
> distributed in the next 6 months or so.
>
> Thanks
> Suru Saha

MERRA Turns 30

31DEC2008 has just been produced by the reanalysis system. This extends the MERRA Mainstream to 30 Years. 2008 is still being checked for quality, but should be posted online at the MDISC within a week or so.

At approximately 1 year per six weeks, 2009 should be produced by the end of February. MERRA production will make a transition then, where we expect data to be released monthly. The production is delayed from real time to allow ingest of the most complete observations, but also to quality check the data files.

Main stream and spinup data

The original plan for MERRA production was to run in three streams, optimizing the computers processes. The original three streams were 1) 1979-1988, 2) 1989-1997 and 3) 1998-present. These were initialized with 2 years of coarse resolution analysis, followed by 1 year at the native (1/2 degree) resolution. Recently, streams 1 and 2 caught up to the beginning of the respective subsequent streams, connecting the time series. Streams 1 and 2 were continued providing overlapping data with the beginning of the next streams, in order to test the variance of the system and the viability of the initialization. At present, each of the overlapping periods are now 3 years duration (1989-1991, and 1998-2000).

We have been evaluating the initial conditions and the continued spinup of streams 2 and 3. In general, there are few differences in the meteorology between the overlapping data. In fluxes (such as precipitation), we do not a small difference at the beginning which gets smaller in time. The differences would likely not affect any scientific results. Slightly larger differences can be seen in slowly varying states, such as the root zone wetness.

At present we are documenting these differences and will provide a report on the overlapping period. In the mean time, this letter is provided to users to alert them that we will be changing the transition times of the streams to utilize the additional data produced at the ends of stream 1 and 2. This does not invalidate the data currently available, but is considered only a minor scientific revision. Once the data is provided to the DAAC and prepared for user access, the new streams and transitions will be as follows:

• Stream 1 1979-1991
• Stream 2 1992-2000
• Stream 3 2001-present

Connecting these streams will be presented to users as the primary MERRA data, or the “main stream” data. Access to Stream 2 1989-1991 and Stream 3 1998-2000 data will still be provided, but the data will be considered secondary and called “spinup” data. This will effectively increase the spinup period for Streams 2 and 3 to 4 years of native resolution analysis.

We anticipate the transition from the original streams to this extended spinup configuration to occur in February 2010, nearly coincident with Stream 3 catching up to real time. Regular updates on this will be made after the holidays.

Have a wonderful Holiday Break!!

Status and some recent analysis

Data at the download site are now continuous from 1979 through December 2006. To date, the data volume is approaching 70Tb, and in September alone, 5.7 million files totaling 91Tb of data were served.

Production continues toward real time, and June 2007 is halfway complete. The throughput is approximately 1 year every 6 weeks, so the production should catch up to real time in early 2010.

An overview of MERRA, including some recent results were presented at Purdue University Department of Earth and Atmospheric Sciences. This includes some evaluation of the global water and energy clycles and processes, however, the analysis continues and a peer review manuscript is in preparation. So, use these results accordingly.

AMS Annual Meeting, Jan 2010, Atlanta

The absttract deadline for the 24th Conference on Hydrology has been extended to Aug 10. Of note, abstracts regarding "Hydrometeorological representation and applications of reanalyses" are sought. The full list of conferences, themes and topics can be found at the AMS WWW page.

Further Production Update

Stream 2 has past the end of Dec 1997 and stream 3 has past the end of Dec 2005, so that we now have a time series from Jan 1979 through Dec 2005. The 1997 and 2005 data are still being assessed for quality, and should be posted at the MDISC site by then of the month or no later than early August.

Production update

Stream 1 has completed Jan 1979 through Dec 1988, catching up to the beginning of Stream 2. The reamining Stream 1 data will be released to the the download site within a week or so. Further, we are conituing Stream 1 for two additional years in order to study the initialization of Stream 2, and evaluate the transition of the data between the two streams.

Stream 2 should catch up to the begining of Stream 3 by the end of July. We we conduct a similar evatuation of the transtion between streams there as well.

Stream 3 production is on hold, pending a fix to allow for a format change in some of hte input data. It will resume ASAP.

AMS Annual Meeting - MERRA Short Course

Today, January 11, we hosted the short course on MERRA and data access. We had 15 attendees from a variety of backgrounds, research and applications, universities and government. Also, there were a range of experiences, some familiar with reanalyses some no prior experience. Our objective was to provide the basic understanding of the system, how we validate and use the data in research and how to access the data (with traditional methods, and newer online software access).

The day started with and overview of the project and the GEOS5 data assimilation system by Michele Rienecker. Michael Bosilovich presented an overview of the validation prior to starting the reanalysis and the current description of the hydrological cycle and global energy budget. Steve Berrick described the access to the data and the various portals at the MDISC. Arlindo da Silva gave a wide ranging presentation on how many different software packages can access MERRA online data.

We were pleased to have Alan Betts give a lunch time presentation covering much of the work he has done over the last 10 years working with ECMWF reanalyses data. The afternoon was reserved for some hands on data analysis and processing activities. We provided digital handouts including many of the presentations but also some software and data that the attendees could run on rented laptops (or their own).

The first hands-on exercise was reproducing some of Alan's figures of ERA land atmosphere interactions except with MERRA data. Next Arlindo da Silva discussed the regridding and reformatting of reanalyses data with the theme of "Look-Alike" imitation. In other words, making MERRA look like NCEP reanalyses (or any other reanalysis) for comparison or reading into existing software applications.

One theme of the meeting was processing data online, not downloading data, but producing the answer with online utilities. This was primarily through GrADS Data Servers (GDS) . The Look-Alike hands-on activity included a walkthrough where participants created MERRA data files from the online data servers using a command line utility (lats4d). Following that, Michael Bosilovich showed examples of using serverside calculations to improve the efficiency of online GDS calculations.

Lastly, Dana Ostrenga of the GSFC MDISC demonstrated the Giovanni access and evaluation of MERRA data, including the along track (satellite track) utility soon to be released. This will allow comparison of MERRA vertical sections compared to A Train data, such as Cloud Sat.

We are currently preparing the materials (including software and presentations) for WWW distribution and will post a message here when they are ready. The networking and online data servers performed well during these exercises.