Climate Observations: Data Quality Control and Time Series Homogenization

By Peter Domonkos, Róbert Tóth and László Nyitrai

Climate Observations: Data Quality Control and Time Series Homogenization pulls together the different phases of the production of high-quality climatic datasets, allowing interested readers to obtain a coherent picture on the complexity and importance of this task. There are several new methods of time series homogenization, each very complex and fast developing. The thematic discussion of the production of high quality climatic datasets provides the opportunity to reduce errors, including the careful installation of meteorological instruments, the application of strict observing rules and inspections, and the use of sophistically developed statistical software to detect and remove errors or biases.

This book is intended for professionals working on climate data management at the national meteorological services, for the users of observed climatic data, and for students and researchers studying atmospheric and climate science.

Members of the Royal Meteorological Society are eligible for a 35% discount on all Developments in Weather and Climate Science series titles. See the RMetS member dashboard for the discount code.

• Describes the research tasks and tools for which the reliability and accuracy of climatic data is particularly important
• Includes case studies to provide real-world context to the research presented in the book
• Features benchmark datasets that have been used for testing the stable operation and efficiency of homogenization methods
• Explains the use of semiautomatic quality control software, recently developed effective homogenization methods, their testing, and related new concepts and statistical tools

• Language: English
• Publisher: Elsevier Science
• Release date: Nov 15, 2022
• ISBN: 9780323904889

Peter Domonkos

Peter Domonkos is a Hungarian climatologist living in Spain since 2009. He is expert on statistical climatology, analysis of extreme climatic events, data quality control and time series homogenization. He is member of the Hungarian Meteorological Society and secretary of ESPERE (Environmental Science for Everybody Round the Earth). Between 2009 – 2015 Dr. Domonkos was a researcher of the University Rovira i Virgili (Tarragona, Spain), and is a free researcher since then. He has been developed an automatic homogenization method (ACMANT), which was found to be one of the most accurate methods by various international test experiments. Between 2013 and 2015 he led 4 international trainings on time series homogenization, sponsored by the World Meteorological Organization. He has 104 printed scientific publications.

Book preview

Introduction

We need observed weather and climate data to know how to prepare for outdoor activities. Weather forecasters need much more observed data to make reliable weather predictions. And climate science needs even more observed data, that are dense in space and time, to understand the rules of weather and climate processes, to produce reliable climate predictions, and provide professional support to applied climatological research. Of course, not only the amount of observed data, but also their accuracy is important. In addition, observed climatic data are expected to be fairly comparable both spatially and temporally, as they are often used jointly, or to analyze spatial and temporal climatic gradients. The temporal comparability needs a long and uninterrupted series of climate observations performed at the same sites, in the same environment, and following fixed observation rules. On the other hand, fair spatial comparability needs networks of adequately dense and uniformly equipped observing sites, and the application of common observing rules.

In the history of instrumental climate observations, the development of accuracy, spatial density, and the uniformity of observing rules are gradual. Together with the increasing spatial density of observing stations, statistical methods have been developed to optimize the information provided by the data for climate change and climate variability estimations. These statistical methods have two main groups: data quality control and time series homogenization. In some European cities, time series of instrumental temperature and precipitation observations have been recorded from as early as the 18th century, but the earliest records are rarely used for temporal gaps of the observations and other data quality problems. From the second half of the 19th century, national meteorological institutes were established in many countries, and they provide spatially coordinated climate observations of high-level professional standards. In 1950, the creation of the World Meteorological Organization (WMO) within the United Nations brought new possibilities to the worldwide regularization of climate observations and climate data management. Within the WMO, the World Climate Data Monitoring Program (WCDMP) was launched in the 1980s, which released 86 documents for improving the uniformity and accuracy of climate observations, giving recommendations for data rescue, quality control, time series homogenization, and other data management issues, and giving advice on the use of observed data in the analysis of climate variability and climatic extreme events.

This book has three main topics: climate observations, data quality control, and time series homogenization. The intention and practical efforts to produce more accurate data and spatially and temporally more correctly comparable data is the common line, which connects these rather different topics. The time series homogenization (referred briefly to as homogenization) takes the largest place in the book. The detailed presentation of homogenization issues has four reasons: their high potential impact on the accuracy of climate trend estimations, the fast methodological development of homogenization in the most recent decades, the scientific complexity of the topic, and the lack of alternative sources regarding thematically ordered and sufficiently detailed scientific presentations of this topic.

The complete presentation of climate observations would have a larger extent than the entire book, so, we focus on the land surface observations and give a less detailed presentation of the other segments of climate observations. The reason for this distinction is that long time series of spatially dense and accurate observations are accessible first for land surface observations; hence, the highest level of spatial and temporal comparability can generally be achieved for such climatic records. Note that although the observations of chemical components of the atmosphere are also climate observations, we skip this subtopic due to the large methodological differences in comparison with recording and analyzing the more traditionally observed climatic elements relating directly to weather variations.

Since the late 1980s and 1990s, both the scientific community and the public are more aware about the threats of accelerating global warming, and the importance of possessing accurate and correctly comparable climatic data seems obvious. However, the intention to improve data accuracy as much as possible is far older than the recognition of the severity of global warming issues. It took months for a climate observer aspirant in the 20th century to learn the correct performance of instrumental and subjective observations. An aspirant observer was regularly accompanied by an experienced observer who explained hundreds of rules of the correct observation practices. Certain rules sometimes seemed too minute. However, the best climate observers would rather record 100 unimportant details than to miss only 1 of the important ones. We would not have long, high-quality climate time series without their work. I believe that a book presenting the best practices of high-quality climate data production is more than a source of information. It is also a homage to the participants involved in the production of high-quality climatic databases, namely engineers and manufacturers of precise meteorological instruments; officials charged with data collection and data archiving; students who digitized a huge amount of data to save them for future generations; upper air wind observers who followed visually the trajectory of a balloon, keeping their eyes glued to the optical instrument (theodolite) for 30–60 min under different weather conditions; the brave men who traveled to icy polar regions to perform climate observations and discover new details of the Earth’s climate; residents who never traveled anywhere, as they performed climate observations at the same observation site on each day of the year without interruptions for holidays.

The book has 10 chapters. Chapter 1 presents the land surface observations. Both instrumental and subjective observations are discussed, but the presentation of the instrumental observations for some key climatic elements, i.e., temperature, precipitation total, wind speed, wind direction, relative humidity, atmospheric pressure, and sunshine duration, is detailed more than that of the other climatic elements. From the 1990s, the traditional meteorological instruments have been changed to automatic weather stations (AWS) in most countries of the world, and we present both the traditional and AWS instrumentations.

In Chapter 2, we deviate from the main topic of the book, which deals with data on land surface observations, and present the other segments of climate observations. In this chapter, the radiosonde observations are presented in more detail, as they provide insight to the three-dimensional behavior of some essential climatic elements like temperature, humidity, and airflow structures. Then the role of meteorological satellites and radars in climate observations is discussed. Although these remote sensing observations are generally less accurate than the local observations, they are indispensable for climate monitoring in the less populated regions of the Earth, and also for monitoring some special climate properties.

In Chapter 3, the main topic is the quality control of the observed data. The sources of data errors, the indicators that suggest the likely occurrence of data errors, and the different quality control procedures are discussed with several examples.

From Chapter 4 to Chapter 9, the principal topic is the time series homogenization. Homogenization examines persistent biases; this topic is more complicated than the quality control, for the added time dimension. In Chapter 4, the general concepts of homogenization are presented, and the connection between quality control and homogenization is discussed.

In Chapters 5–7, the rules and options for relative homogenization are presented. This is the most important group of homogenization methods. In relative homogenization, persistent non-climatic biases of a candidate series of observed climatic values are detected by spatial comparisons between the candidate series and time series of neighboring stations. There is a large variety of relative homogenization methods. Despite the scientific complexity of the topic, the main rules and conclusions are easy to understand, while the detailed descriptions serve partly as justifications and partly to offer a unique intellectual adventure for interested readers.

In Chapter 8, the most frequently used homogenization methods are presented. In addition, some modern homogenization methods showing high accuracy in method comparison tests are also presented.

Chapter 9 is dedicated to the topic of efficiency tests of homogenization methods. The accuracy of homogenization methods can be tested on synthetically developed test datasets. We discuss with examples why such tests are important, which characteristics make a test dataset appropriate for testing, and which factors limit the potential accuracy of homogenization methods.

Chapter 10 presents examples where the accessibility and accuracy of observed climatic data are highly important. Data accuracy is not less important for the elaboration of climate predictions and climate change scenarios than for revealing Earth's climate in the past, and in this chapter we show the reasons why.

The book is supplemented with an Appendix that describes some basic statistical concepts and relations. Readers can consult this Appendix at any time if they feel the necessity to renew such knowledge when reading the main chapters of the book.

The major part of the material in this book is my own collection. For the presentation of AWS instrumentation and upper air observations, I received the material from two of my former colleagues, Róbert Tóth and László Nyitrai, Hungarian Meteorological Service, research fellows in the areas of land surface observations and upper air observations, respectively, and we wrote these sections together.

Chapter 1: Land surface observations

Abstract

The observation program of a typical synoptic station is described both in manual weather station (MWS) and automated (AWS) modes. First, the selection of observing sites and the preparation of instrument land where the meteorological instruments are placed are described. The installation, using, and maintenance rules of instruments are presented. Time schedule of reading instruments and the automatic recording with mechanical and electric instruments are described. More details are dedicated to the essential climate variables, which often have long series of adequate quality and spatial density, and they are temperature, humidity, precipitation amount, wind direction and wind speed, atmospheric pressure, and sunshine duration. The observation of other climate elements including visual observations is briefly presented. Traceability assurance and instrument calibration issues are presented. The most typical sources of observation errors are discussed.

Keywords

Observing station; Observing rules; Temperature; Humidity; Precipitation; Wind measurement; Atmospheric pressure; Sunshine duration; AWS; MWS

In this chapter, the main instruments and methods of land surface observations are presented. We go through the activities of surface observing stations focusing most on climatic elements, which have long records in numerous observing sites. These are in harmony with the essential climate variables (ECV) determined by the Global Climate Observing System (https://gcos.wmo.int) for the near surface part of the atmosphere: temperature, water vapor, precipitation, wind speed and direction, atmospheric pressure, and surface radiation budget (Bojinski et al., 2014). Both the traditional, manual observations and those with automated instruments are presented. In most cases of detailed descriptions, and for the essential climate variables always, the presented methodologies characterize the professional observations organized by the Hungarian Meteorological Service (HMS), as there I was observer in the 1980s, and thus I have personal experiences from that observing network. In Hungary, the regular instrumental climate observations started in 1781 in Buda (part of the later capital Budapest), and the national meteorological institute (later Hungarian Meteorological Service) started its operation as early as in 1870. The performance of Hungarian climate observations has been following high international standards; therefore, I believe that the presentation of the Hungarian observation practices adds a particular value to the content of this chapter.

Readers interested in a wider and more general presentation of climate observations may consult the relevant and openly accessible World Meteorological Organization (WMO) issue (WMO, 2018a).

1.1: Global system of weather and climate observations

Climatic elements can be observed with watching subjectively the weather processes or with reading meteorological instruments. For instance, cloud types or rain duration can be observed directly by eyes, while the observation of atmospheric pressure or radiation total is unimaginable without meteorological instruments. For many other climatic elements, subjective observations can provide rough estimations only: An observer may feel that the temperature is high or low, may see the traces that lot of rain have been fallen, etc., but such estimations are insufficient for the quantitative description of weather and climate. In climate observations, the use of meteorological instruments is general from the 18th century for observing temperature, precipitation amount and atmospheric pressure, although the number of observing sites was very small before 1850. From the second half of the 19th century, observing networks became denser, the instrumental observations of further climatic elements have been established, and the unification of observation rules has begun with the foundation of national meteorological institutes in many developed countries.

The climatic record surface air temperature = 20.0°C refers to a physical state of the air near the ground surface. Ideally, a given record should mean the same at any place of the world, and the meaning should be independent from the time of the observation. Many efforts have been dedicated to unify observing rules, first by the national meteorological institutes and then by the WMO, but some geographical differences have been remained for traditions, political reasons, and also for maintaining long series of observations without methodological changes. An obstacle of unifying observing rules internationally was and has remained that while such unifications favor the geographical comparability, might do harm in the temporal comparability of climate records of an observing site. Further problem is the natural differences of ground surface. Unified observing rules cannot always be provided for regions of deserts, forests, permafrost areas, etc.

Around 1990 the transition to the use of automatic weather stations (AWS) started, and the developed countries finished this transition in or before the first decade of the 21st century. Note, however, that manual observations are still performed for some climatic elements. In our review about climate observations, the instrumentations of both the manual weather stations (MWS) and AWS are presented, but we cannot present the diversity of instruments applied in different countries and different eras. We limit the presentation of climate observations to some typical MWS (AWS) instruments used in the 1980s (around 2020) in a central European country, Hungary. These or very similar instruments with the same or nearly the same installations are applied in many other countries of the world.

This chapter is generally dedicated to the presentation of the land surface climate observations, but before starting that, here we review briefly the whole system of climate observations.

•Land surface observations: These observations are performed by observers or by AWSs. The instruments are usually placed about 1–2 m height above the ground surface. Often a large number of climatic elements are observed in a given observing site, and the climate records can be longer than 100 years.

•Marine climate observations: Marine observations are performed in ships, by using buoys, or with remote sensing from meteorological satellites. Though ship observations have long history, we dedicate little space to marine observations for two reasons: ship observations provided a strongly uneven spatial coverage of marine climate records, and the spatial and temporal comparability of data is generally poorer than for land surface data for the spatially and temporally changing conditions of marine observations. In the next chapter, some satellite-based marine observations will be briefly discussed (Section 2.7), while for readers interested in ship and buoy observations we recommend WMO (2018b) and Hemsley (2015).

•Radiosondes: A radiosonde comprises an electric thermometer, an electric hygrometer and a transmitter. After the release of the radiosonde from its host station, it elevates up to about 30 km height in 60–90 min by a balloon filled with hydrogen or helium, and it monitors and transmits the physical properties of the atmosphere around it. Radiosondes are used from about the middle of the 20th century, and the most accurate and spatio-temporal coherent observations of the troposphere and lower stratosphere are provided by them.

•Meteorological satellites: Surface and upper air properties are monitored from meteorological satellites since the 1960s. Sensors of meteorological satellites intercept the natural electromagnetic radiation emitted from the earth surface and atmosphere, and the evaluation of intensity distribution according to radiation wavelengths provides the transformation from detected radiation to observed climatic values. A large variety of climate variables and climate indicator surface properties are observed by them all over the Earth.

•Meteorological radars: Similarly to satellites, radars are modern remote sensing tools in meteorology. They emit electromagnetic waves, which reflect from raindrops, snowflakes and ice particles of clouds. Weather radars detect the development, position and intensity of thunderstorms, hailstorms, the icing conditions in cloudy areas, and also area average precipitation amounts can be measured by them. Upper air winds can also be observed by radars.

Radiosonde, satellite and radar observations will be presented in Chapter 2. Turning back to the presentation of land surface observations, the first thing to be enhanced is that different kinds of observing stations exist with different instrumentations and schedules of observations. AWSs allow continuous observation of climatic elements, but continuous observations were performed also in the MWS era in synoptic stations. One important role of synoptic stations was to provide continuous observations and frequent data transmissions for weather forecasts and weather alarm systems. The observer of an MWS coded the majority of the actually observed climate characteristics in every hour, and emitted a report having comprised a series of digits.

A part of the climate observations are often organized out of the national meteorological institutes, as climate observations with special objectives serve hydrological, military, agronomical purposes, or monitoring local climates in cities or coastal areas. Such observation programs and the data management might be organized jointly with meteorological institutes, but these issues vary according to countries and often also according to historical periods. In several countries, the meteorological and hydrological managements or the civil and military services are unified institutionally. However, the unification of data obtained by different kinds of observation managements might reduce the spatial and temporal comparability of climate records, or at least the unification needs special attention in the data management procedures.

In Hungary, 24 synoptic stations operated in the 1980s with the continuous observation of many weather and climate elements, while the program of further 60 principal climatological stations were usually limited to the observation of precipitation, temperature, humidity and significant weather events like fog, thunderstorm, etc. Beyond the institutionally organized network, voluntary observers of precipitation observing stations contributed to the spatially dense observation of precipitation and significant weather events. With the installation of AWSs, the division to synoptic stations and principal climatological stations ceased, but the spatial density of observations still have differences according to climatic elements. The main reason of these differences is the differences in the degree of the natural spatial variability according to climatic elements.

In 2020, 125 AWSs were operated by HMS, and further 142 by General Directorate of Water Management (GDWM). In the majority of the GDWM stations, only precipitation, snow cover, and snow water content measurements are performed, although in a few of them several other climatic elements are also observed. There is a close collaboration between HMS and GDWM, HMS helps in the professional control and maintenance of the GDWM instruments, and the observed data of GDWM are shared with HMS.

Image 1

Reference observing station of Hungarian Meteorological Service in Budapest Pestszentlőrinc (2021). The instrument nearest to the camera is a lightning detector.

The observed data of AWSs are coded and transmitted from the observing sites into the HMS center in every 10 min by the AWS computers and a Web application system installed in the center. The fast and high quality fulfillment of coding and transmission is clearly more assured with the new AWS computers and modern transmission channels than with any earlier system.

Precipitation observations need the highest station density, firstly for its outstanding importance in hydrology, water management and agriculture, and secondly for the generally high spatial variation of the fallen precipitation. In Hungary, the system of voluntary observers satisfies this need. Voluntary observers are trained civil observers working for precipitation observation stations. The number of these stations decreased since the middle of the 20th century from 800 to 430. Around 2020, approximately half of the precipitation observing stations have already been automated, and their data were collected by the HMS automatic data transmission system. From the still operating manual precipitation observing stations, the observers sent daily reports of the amount and form of the fallen precipitation. Voluntary observers are encouraged to send a report immediately when an extraordinary weather event has been occurred (e.g., heavy rain, hailstorm, etc.). The data sent from precipitation observing stations are subjected to professional quality control in the same way as any other observed climatic data.

In the observing network of HMS, the visual observation of cloudiness and significant weather events has not completely ceased, but such observations serve more weather forecasts than climatological use. Around 2020, 14 professional observers performed visual observations.

1.2: Site selection and installation of instruments

Observed climatic characteristics are generally expected to be representative for the region of the observing site. Therefore, as far as it is possible, flat areas with spatially uniform surface use and vegetation cover are selected to be observing sites. However, as we are also interested in learning the climate of coastal or mountainous areas, several exceptions occur in practice where the data of observing sites characterize more the climate of specific locations than the regional climate. The spatial representativeness of observed data depends also on the observed climatic elements. For instance, the spatial representativeness is generally high (∼ 100 km) over planes for temperature, irradiation and atmospheric pressure when local weather phenomena like showers, fogs, etc. do not disturb the spatial uniformity of weather characteristics, while the spatial variability is the highest for cloudiness and precipitation amount.

Professional meteorological instruments, either of MWS or AWS, are installed in an enclosed area of "instrument land," far from any local influence (e.g., high buildings, smoke source, roads with traffic). The recommended size of instrument land is at least 25 m² × 25 m², but note that the size may depend from the distance from potentially disturbing nearby objects, observation program of the station, and sometimes also from practical limits. The surface of the instrument land is covered by short grass where the soil and climate allow grasslands, or remains bare in the reverse case.

In selecting the places of individual instruments, distance keeping from the fence of the instrument land and possible other nearby objects must be considered. Radiometers are the most demanding instruments regarding their locations, as any limitation of the incoming radiation by the shadow of surrounding objects would directly affect the observed radiation. The location of wind measuring instruments also needs distinguished attention, and for them the height above other nearby objects has key importance. Wind speed and wind direction are affected by the local geographical unevenness of the surface and any natural or human made objects being in the way of airstreams. Therefore, it is recommended to place the wind measuring instruments at least a few meters above the top of all nearby objects. Thermometers and hygrometers also can be affected by local distortions of airstreams or the radiation of nearby objects, but the impacts of these factors are generally low in an appropriately selected instrument land. More importantly, thermometers and hygrometers must be placed to a clearly higher level than the surface of plants within the instrument land, and they must be protected against direct radiation effects and the direct effects of weather phenomena like rain, snow, ice deposition, etc.

Summarizing, the following factors influence the spatial representativeness of the observed climatic data:

•Geographical characteristics of the site, like exposure, distance from water bodies and distance from the nearest high objects;

•Preparation and maintenance of instrument land;

•Location of meteorological instruments within the instrument land;

•Height of meteorological instrument above the surface and above other nearby objects;

•Protection of certain instruments from direct weather effects.

1.3: Manual and automated observations

Until the 1990s, relatively few AWSs were installed in the world, mainly in places hardly accessible for local inhabitants (Hartl et al., 2020). In other places, the inclusion of human observers facilitated a higher quality and completeness of observations, and MWSs were more economic than AWSs. With the development of more precise and more economic automatic tools this situation was gradually changed, and AWSs became the best and most economic observation tools around 1990. In Hungary, the automatization of synoptic and principal climatological stations was completed between 1990 and 2000. In many synoptic stations, both AWS and MWS observations were performed during some years of the transition to AWS mode, and these parallel measurements help to eliminate inhomogeneities from the time series of climate records.

Image 2

Manned weather station in northwestern Hungary (Sopron) according to an old photograph. In the right, there is a Stevenson screen, in which the thermometers and hygrometers were placed. Bottom in the right a traditional precipitation gauge stands. On the left side cables are exposed to observe possible ice depositions from air (hoar, rime or icing rain). Note the large distances between the station building and the meteorological instruments.

Traditional precipitation gauges are still in use in several precipitation observing stations. The withdrawal of observer staff from synoptic stations and principal climatological stations was gradual and completed about 2015 leaving only one professional observer to observe the cloudiness and weather phenomena over 7–10 sites of the western part of country by web cameras, as well as one observer in the eastern part of the country for 7 sites. When significant weather phenomena (e.g., fog, thunderstorm, snowfall) are expected, all the 14 official observers are instructed to carry out local visual observations. Beyond the principal change that the contribution of human observers was minimized, some other important changes are related to the automatization of observations: (i) New instruments with new sensors are used; (ii) Mechanical recording instruments are no longer used; (iii) Accuracy of observed physical quantities has generally been improved; (iv) A part of the visual observations have been mechanized, while another part of them are simplified or even abandoned; (v) Timings of observations have become accurate; (vi) In recording daily temperature maximums and minimums AWSs consider the 24 h period of calendar days; (vii) Calculations and coding have been computerized; (viii) Data recording and data transmission to the host institute have been modernized. All these changes may influence the temporal comparability of climate records; therefore, we discuss them more.

(i)New instruments with new sensors are used. The new sensors are often electric tools, i.e., their operation is either related to the changes of some electric properties as a response to the changes of meteorological conditions, or to the emission/absorption of electromagnetic waves. The new sensors are generally more accurate, although some exceptions occur. The new sensors are usually smaller and characterized by shorter response time than the sensors of the MWS instruments. This change is generally favorable, but might affect the temporal comparability between old and new observations. Note that AWSs may include instruments with nonelectric sensors, and in this case, the instrument is supplied with a transducer to provide digital recording.

(ii)Mechanical recording instruments are no longer used. In the MWS era, the continuous recording of some climate elements was solved by mechanical recording instruments. They operated with pens and ink drawing graphs on a chart. The chart was fitted over the surface of a clock-driven revolving drum.

A mechanical barograph from 1930 (Wikimedia).

The pens moved vertically in function of the transmitted signs of the sensors, while the drum and the chart moved horizontally around the vertical axis of the drum. The drum made a whole circle within 1 day or 1 week, hence daily or weekly charts were produced. In most cases, these instruments were less accurate than the base instruments of the station, due to the errors in mechanical transmission to the pens and the friction between the pen and the paper of the chart. The records of these instruments served to control the correct operation of the base instruments and to provide details about the temporal changes of the observed climate variable, but they did not substitute the readings of the base instruments in predetermined timings. With the transition to AWSs, the continuous recording of climate observations is solved with more modern and accurate