Probability Theory and Mathematical Statistics for Engineers

By V. S. Pugachev

Probability Theory and Mathematical Statistics for Engineers focuses on the concepts of probability theory and mathematical statistics for finite-dimensional random variables.

The book underscores the probabilities of events, random variables, and numerical characteristics of random variables. Discussions focus on canonical expansions of random vectors, second-order moments of random vectors, generalization of the density concept, entropy of a distribution, direct evaluation of probabilities, and conditional probabilities. The text then examines projections of random vectors and their distributions, including conditional distributions of projections of a random vector, conditional numerical characteristics, and information contained in random variables.

The book elaborates on the functions of random variables and estimation of parameters of distributions. Topics include frequency as a probability estimate, estimation of statistical characteristics, estimation of the expectation and covariance matrix of a random vector, and testing the hypotheses on the parameters of distributions. The text then takes a look at estimator theory and estimation of distributions.

The book is a vital source of data for students, engineers, postgraduates of applied mathematics, and other institutes of higher technical education.

• Language: English
• Publisher: Elsevier Science
• Release date: Jun 28, 2014
• ISBN: 9781483190501

Book preview

Probability Theory and Mathematical Statistics for Engineers

V.S. PUGACHEV

Institute of Control Sciences, Academy of Sciences of the USSR, Moscow, USSR

Table of Contents

Cover image

Title page

Other Titles of Interest

Copyright

PREFACE

Chapter 1: PROBABILITIES OF EVENTS

Publisher Summary

1.1 Random phenomena

1.2 Statistical approach to the description of random phenomena

1.3 Direct evaluation of probabilities

1.4 Operations with events

1.5 Axioms of probability theory

1.6 Conditional probabilities

1.7 Probabilities of complex events

1.8 Repeated trials

1.9 Poisson distribution

Chapter 2: RANDOM VARIABLES

Publisher Summary

2.1 General definitions. Discrete random variables

2.2 Continuous random variables. Density of a random variable

2.3 Generalization of the density concept

2.4 Distribution function

2.5 Entropy of a distribution

Chapter 3: NUMERICAL CHARACTERISTICS OF RANDOM VARIABLES

Publisher Summary

3.1 Expectation

3.2 Characteristics of the scatter

3.3 Second-order moments of random vectors

3.4 Canonical expansions of random vectors

3.5 Other numerical characteristics of random variables

3.6 One-dimensional normal distribution

Chapter 4: PROJECTIONS OF RANDOM VECTORS AND THEIR DISTRIBUTIONS

Publisher Summary

4.1 Distributions of projections of a random vector

4.2 Conditional distributions of projections of a random vector

4.3 Conditional numerical characteristics

4.4 Characteristic functions of random variables

4.5 Multi-dimensional normal distribution

4.6 Information contained in random variables

Chapter 5: FUNCTIONS OF RANDOM VARIABLES

Publisher Summary

5.1 Moments of functions of random variables

5.2 Distribution function of a function of a random variable

5.3 Density of a function of a random variable

5.4 Limit theorems

5.5 Information contained in transformed random variables

Chapter 6: ESTIMATION OF PARAMETERS OF DISTRIBUTIONS

Publisher Summary

6.1 Main problems of mathematical statistics

6.2 Estimation of statistical characteristics

6.3 Frequency as a probability estimate

6.4 Estimation of the expectation and variance of a random variable

6.5 Estimation of the expectation and covariance matrix of a random vector

6.6 Testing hypotheses about parameters of distributions

Chapter 7: ESTIMATOR THEORY

Publisher Summary

7.1 General properties of estimators

7.2 Main methods for finding estimators

7.3 Recursive estimation of the root of a regression equation

7.4 Recursive estimation of the extremum point of a regression

Chapter 8: ESTIMATION OF DISTRIBUTIONS

Publisher Summary

8.1 Estimators of densities and distribution functions

8.2 Approximate representation of distributions

8.3 Testing hypotheses about distributions

8.4 Statistical simulation methods

Chapter 9: STATISTICAL MODELS, I

Publisher Summary

9.1 Mathematical models

9.2 Regression models

9.3 Estimation of regressions

9.4 Testing hypotheses about regressions

9.5 Analysis of variance

Chapter 10: STATISTICAL MODELS, II

Publisher Summary

10.1 Models described by difference equations

10.2 Estimation of random variables determined by difference equations

10.3 Factor models

10.4 Recognition models

10.5 Decision-making models

APPENDICES

MAIN NOTATIONS

REFERENCES

INDEX

Other Titles of Interest

AKHIEZER and PELETMINSKII

Methods of Statistical Physics

BOWLER

Lectures on Statistical Mechanics

Journals

Automatica

Computers and Mathematics with Applications

Journal of Applied Mathematics and Mechanics

Reports on Mathematical Physics

USSR Computational Mathematics and Mathematical Physics

Copyright

Copyright © 1984 Pergamon Press Ltd.

All Rights Reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means: electronic, electrostatic, magnetic tape, mechanical, photocopying, recording or otherwise, without permission in writing from the publishers.

First edition 1984

Library of Congress Cataloging in Publication Data

Pugachev, V. S. (Vladimir Semenovich)

Probability theory and mathematical statistics for engineers.

statistika.

Includes bibliographical references.

1. Probabilities. 2. Mathematical statistics.

I. Title.

QA273.P8313 1984 519.1 82–13189

British Library Cataloguing in Publication Data

Pugachev, V. S.

Probability theory and mathematical statistics for engineers.

1. Probabilities 2. Mathematical statistics

I. Title

519.2 QA273

ISBN 0-08-029148-1

Printed in Hungary by Franklin Printing House

PREFACE

THE original Russian book is based on the lecture courses delivered by the author to students of the Moscow Aviation Institute (Applied Mathematics Faculty) during many years.

The book is designed for students and postgraduates of applied mathematics faculties of universities and other institutes of higher technical education. It may be useful, however, also for engineers and other specialists who have to use statistical methods in applied research and for mathematicians who deal with probability theory and mathematical statistics.

The book is intended first of all for specialists of applied mathematics. This fact determined the structure and the character of this book. A sufficiently rigorous exposition is given of the basic concepts of probability theory and mathematical statistics for finite-dimensional random variables, without using measure theory and functional analysis.

The construction of probability theory is based on A. N. Kolmogorov’s axioms. But the axioms are introduced only after studying properties of the frequencies of events and the approach to probability as an abstract notion which reflects an experimentally observable regularity in the behaviour of frequencies of events, i.e. their stability. As the result of such an approach the axioms of probability theory are introduced as a natural extension of properties of the frequencies of events to probabilities.

Almost everywhere throughout the book, especially in studying mathematical statistics, vector random variables are considered without preliminary studying scalar ones. This intensifies the applied trend of the book because in the majority of practical problems we deal with multi-dimensional random vectors (finite sets of scalar random variables).

In order to employ the presented methods to direct practical problems using computers, references are given throughout the book to standard programs given in IBM Programmer’s Manual cited in sequel as IBM PM.

Besides the foundations of probability theory an exposition is given of all the basic parts of mathematical statistics for finite-dimensional random variables.

Apart from routine problems of point and interval estimation and general theory of estimates, the book contains also the stochastic approximation method, multi-dimensional regression analysis, analysis of variance, factor analysis, the theory of estimation of unknown parameters in stochastic difference equations, the elements of recognition theory and testing hypotheses, elements of statistical decision theory, the principles of statistical simulation (Monte Carlo) method.

While translating the book into English many improvements were made in the exposition of the theory and numerous misprints were corrected. In particular, new sections were added to Chapters 2, 3, 4, 5 devoted to the fundamental notions of the entropy of random variables and the information contained in them (Sections 2.5, 3.6.5, 4.5.10, 4.6 and 5.5).

In Chapter 1 the basic properties of frequencies of events are considered and a frequency approach to the notion of probability is given. The cases are considered where the probabilities of events may directly be calculated from the equiprobability of different outcomes of a trial. After that the notion of an elementary event is given. The basic axioms of probability theory are formulated, the notions of probability space, probability distribution, conditional probability, dependence and independence of events are introduced and the basic formulae, directly following from the axioms including the formulae determining binomial and polynomial distributions, are derived. Then the Poisson distribution is derived.

In Chapter 2 random variables and their distributions are considered. The basic characteristics of the distributions of finite-dimensional random variables, i.e. a density and a distribution function, are studied. It is shown that a density as a generalized function, containing a linear combination of δ-functions, exists for all three types of random variables encountered in problems of practice, i.e. continuous, discrete and discrete–continuous. An example is given of a random variable which has not a density of such a type. The notions of dependence and independence of random variables are introduced. Finally the notion of entropy is given and the main properties of entropy are studied.

In Chapter 3 the numerical characteristics of random variables are studied. First the definition of an expectation is given and basic properties of expectations are studied. Then the definitions of the second-order moments are given and their properties are studied. After this the moments of any orders of real random variables are defined. Besides the moments the notions of a median and quantiles for real scalar random variables are given. The chapter concludes with the study of the one-dimensional normal distribution.

Chapter 4 is devoted to the distributions and conditional distributions of projections of a random vector. The expressions for the density of a projection of a random vector and the conditional density of this projection, given the value of the projection of the random vector on the complementary subspace, are derived in terms of the density of the random vector. Some examples of dependent and independent random variables are given and the relation between the notions of correlation and dependence is discussed. Conditional moments are defined. Characteristic functions of random variables and the multi-dimensional normal distribution are discussed. The notions of mean conditional entropy and of amount of information about a random variable contained in another random variable are given.

In Chapter 5 the methods for finding the distributions of functions of random variables, given the distributions of their arguments, are studied. Here we consider a general method for determining the distribution functions of functions of random variables, three methods for determining the densities, i.e. the method of comparison of probabilities, the method of comparison of probability elements and the δ-function method, as well as a method for finding the characteristic functions and the moments method. The proof of the limit theorem for the sums of independent, identically distributed, random variables is given. The basic distributions encountered in mathematical statistics are derived in the numerous examples showing the application of the general methods outlined. The last section is devoted to studying the effects of transformations of random variables on the amount of information contained in them.

In Chapter 6 the statement of the basic problems of mathematical statistics, i.e. the problem of estimation of unknown probabilities of events, distributions of random variables and their parameters is given at first. Then the basic modes of convergence of sequences of random variables are considered. The general definitions concerning estimates and confidence regions are given; also the basic methods for finding confidence regions for unknown parameters are studied. After this a frequency as the estimate of a probability and estimates of moments determined by sample means are studied. The chapter concludes with the studying of the basic methods of testing hypotheses about distribution parameters.

The general theory of estimates of distribution parameters and basic methods for finding the estimates, i.e. the maximum likelihood method and moments method, are outlined in Chapter 7. Recursive estimation of the root of a regression equation and the extremum point of a regression by means of stochastic approximation method are studied.

Chapter 8 is devoted to the basic methods for estimation of densities and distribution functions of random variables and the methods for approximate analytical representation of distributions. The methods for testing hypotheses about distributions by the criteria of K. Pearson, A. N. Kolmogorov and N. V. Smirnov are studied and the estimation of distribution parameters by means of minimum χ² method is considered. In the last section of the chapter a summary of a statistical simulation method is given as a technique for approximate calculations and a method for scientific research.

In Chapter 9 statistical regression models are studied. The general method for determining the mean square regression in a given class of functions, in particular linear mean square regression, is studied at first. Then the methods for estimation of linear regressions (regression analysis) and the methods for testing hypotheses about regressions are given. Finally the bases of variance analysis theory are derived from the general theory of designing linear regression models.

Statistical models of other types are studied in Chapter 10. At first the models described by difference equations, in particular autoregression models, are considered. A method for estimation of sequences of random variables determined by difference equations and unknown parameters in difference equations is discussed as well as application of this method to linear and non-linear autoregression models. Then some methods for designing factor models (elements of factor analysis) and recognition models are studied. The similarity is demonstrated of some recognition problems and problems of testing hypotheses about distribution parameters. In the last section a short summary of elements of statistical decision theory (methods for designing the models of decision-making processes) is given.

The Harvard system of references is used in the English translation of the book. The author does not pretend in any way to provide a complete list of literature references in the field concerned. In the list only those sources are given which are cited in the text.

.

Only a short summary is given of basic methods of modern mathematical statistics of finite-dimensional random variables in Chapters 6–10.

For a deeper and more complete study of mathematical statistics one may be recommended to read the books by H. Cramér (1946), M. G. Kendall and A. Stuart (1976, 1977, 1979), S. Wilks (1962), C. R. Rao (1973), T. W. Anderson (1958) and the books on various parts of mathematical statistics, to which in Chapters 6–10 references are given.

In order to study the mathematical foundations of probability theory we advise the books by M. Loève (1978), J. Neveu (1965) and P. L. Hennequin and A. Tortrat (1965).

For information about the notions and theorems from various parts of mathematics used in the book we advise the book by Korn and Korn (1968). For recalling linear algebra the reader may use the books by Gantmacher (1959), Lancaster (1969), Noble and Daniel (1977) and Wilkinson (1965). For recalling mathematical analysis the book by Burkill and Burkill (1970) may be used.†

Sections 2.5, 3.6.5, 4.5.10, 4.6 and 5.5 devoted to the notions of entropy and information contained in random variables, Section 8.4 devoted to the statistical simulation method and Chapters 9 and 10 have been written with the active assistance of I. N. Sinitsyn who has also helped me to edit the whole Russian manuscript. Without his help the book probably would not appear so soon. I consider it my pleasant duty to express my sincere gratitude to I. N. Sinitsyn for his invaluable assistance.

I express also my gratitude to I. V. Sinitsyna for her excellent translation of the book into English and for typing and retyping various versions of Russian and English manuscripts.

I owe also many thanks to Professor P. Eykhoff for his kind collaboration with me as a co-editor of the English translation of the book resulting in considerable improvement of the English version of the book.

I wish to acknowledge my gratitude to N. I. Andreev and N. M. Sotsky for their valuable remarks and discussions which promoted considerable improvement of the book, to N. S. Belova, A. S. Piunikhin, I. D. Siluyanova and O. V. Timokhina for their assistance in preparing the Russian manuscript for press, to M. T. Yaroslavtseva for the assistance in preparing for press the Russian manuscript and the last three chapters of the English manuscript, to S. Ya. Vilenkin for the consultations on computational aspects of the methods outlined in the book and for the organization of computer calculations for a number of examples.

I owe also my gratitude to I. V. Brůža, Eindhoven University of Technology, who carefully checked the list of references at the end of the book, corrected it, and converted it into a form suitable for English-speaking readers.

V.S. PUGACHEV,     Moscow

December, 1980

---

†We recommend Russian readers also the Russian books by Fichtenholz (1964), Nikol’skij (1977) and Smirnow (1979, vols. 1,2) for recalling mathematical analysis and the books by Golovina (1974), Mal’cev (1978) and Smirnow (1979, vol. 3, Pt. 1) for recalling linear algebra.

CHAPTER 1

PROBABILITIES OF EVENTS

Publisher Summary

This chapter discusses the probabilities of events. The irregular oscillations (vibrations) of an aircraft flying in a turbulent atmosphere also represent a random phenomenon. These oscillations of a plane are because of random gusts of wind in a turbulent atmosphere. All theory of modern complex systems and control processes is based on statistical methods. Probability theory is the foundation for the reliability theory of technical systems and for many other applied scientific theories. The chapter presents several examples to provide an overview of probabilities of complex events. Like any phenomena, random phenomena are caused by quite definite reasons. All phenomena of the external world are interrelated and influence each other (the law of phenomenological interdependence). Therefore, each observable phenomenon is causally related with innumerable other phenomena and its pattern of development depends on the multiplicity of the factors. It is, therefore, impossible to trace all these innumerable relations and to investigate their actions. When some phenomenon is studied, only a limited number of basic factors affecting the pattern of the phenomenon can be established and traced. A number of secondary factors is neglected. This gives an opportunity to study the essence of a phenomenon deeply and to determine its regularity.

1.1 Random phenomena

1.1.1 Examples of random phenomena

In his daily life man meets with random phenomena at every step. There is no process without them. The simplest example of random phenomena are the measurement errors. We know that absolutely accurate measurements do not exist. While measuring repeatedly the same object, for instance when we weigh it many times with an analytical balance, we always receive similar but different results. It may be explained by the fact that the result of every measurement contains a random error and the results of different measurements contain different errors. It is impossible to predict what will be the error of a given specific measurement or to determine it after measuring. If we make an experimental study of some phenomenon and represent the results by a graph it is seen that the points, if sufficient in number, never lie on a single curve but are subject to random scatter. This scatter is explained both by measurement errors and the action of other random causes.

A second example of random phenomena is missile scatter. The missiles never get in the same point even when you aim at the same point. One would think the conditions for all the shots are the same. But the missiles follow different trajectories and arrive at different points. It is impossible to predict in what point a given missile arrives. One of the reasons for this is the fundamental impossibility to measure exactly the parameters of atmospheric conditions at all the points of the missile trajectory exactly at the time instants at which the missile will pass these points. The aerodynamic forces and their moments acting on the missile depend on these parameters and cause the uncertainty of the hit point of the missile.

As a third example of random phenomena we can point to the failures of various technical equipment. In spite of the high quality of modern engineering there are sometimes failures of some devices. The failure of a device is a random phenomenon. It is impossible to predict if it fails or not and, if it fails, to predict the instant of a failure.

The noises in radio-receivers also belong to the random phenomena. The so-called ether is always satiated with various electromagnetic radiations. The electric discharges in the atmosphere, the movement of atmospheric electricity, working equipment created by man and so on act as sources of such radiations. Therefore tuning cannot prevent outside radiations to make noises in a receiver. The more remote the transmitter is the more noises distort the received signals. It becomes apparent that at the same time we hear in the radio-set the received signals and crackles as well. This well-known phenomenon also represents a random phenomenon because it is impossible to predict when and what outside electromagnetic radiation will get into the radio-set. It is fundamentally impossible to avoid the outside radiations in the radio-sets since they are destined for receiving faint electromagnetic radiations.

The irregular oscillations (vibrations) of an aircraft flying in a turbulent atmosphere also represent a random phenomenon. These oscillations of a plane are due to random gusts of wind in a turbulent atmosphere.

1.1.2 Nature of random phenomena

Like any phenomena random phenomena are caused by quite definite reasons. All phenomena of the external world are interrelated and influence each other (the law of phenomenological interdependence). Therefore each observable phenomenon is causally related with innumerable other phenomena and its pattern of development depends on the multiplicity of the factors. It is therefore impossible to trace all these innumerable relations and to investigate their actions. When some phenomenon is studied only a limited number of basic factors affecting the pattern of the phenomenon can be established and traced. A number of secondary factors is neglected. This gives an opportunity to study the essence of a phenomenon deeply and to determine its regularity. At the same time acting in such a way man impoverishes the phenomenon, makes it schematic. In other words, an observable phenomenon is substituted by its suitable simplified model. In consequence of this any law of science reflects the essence of an observable phenomenon, but it is always considerably poorer than the true phenomenon. No law can characterize a phenomenon comprehensively, in plentitude and variety. The deviations from regularity caused by joint action of an innumerable variety of neglected factors we call random phenomena.

If we make an experimental study of any phenomenon with the purpose of obtaining its regularities we have to observe it many times under equal conditions. By equal conditions we mean equal values of all numerical characteristics of controlled factors. All uncontrolled factors may be different. Consequently the action of the controlled factors will be practically the same under different observations of the same phenomenon. This fact reflects the regularities of a given phenomenon. Random deviations from the regularities caused by the action of uncontrolled factors are different for different observations and it is fundamentally impossible to predict beforehand what they will be in a given concrete case.

The importance of chance in different phenomena is different. Random deviations from regularities (laws) in some phenomena are so small that they may be neglected. But there are such phenomena in which no regularities at all are apparent and the chance is of main importance. An example of such a phenomenon is the motion of a small particle of solid material in a liquid suspension known as Brownian motion. The collisions with a very large number of moving molecules cause the particle to follow a quite unsystematic path without any apparent regularity. In phenomena of this kind the random behaviour is itself a regularity.

Under multiple observations of random phenomena one may notice definite regularities in themselves. After studying these regularities one can control to a certain extent such random phenomena, restrict their effects, predict the results of their actions and even use them in practical activities. Thus one can design measuring systems possessing the maximal available accuracy, radio-receivers with the minimal level of noises, control systems for aircraft which provide the highest possible accuracy of navigation or the least action of perturbation on the aircraft. One may also design technical systems possessing a given reliability.

1.1.3 Mass random phenomena

It is clear that regularities of random phenomena become manifest only through repeated observation. Hence it follows that one can study only those random phenomena which are observable, at least potentially, indefinitely many times. Such random phenomena are called mass random phenomena. It should be noticed that for studying these phenomena there is no strict necessity that they be really observable many times. Having studied the regularities of elementary random phenomena and, on the basis of this study having constructed the corresponding theory, one may then theoretically study more complicated random phenomena including those which are not directly observable (but which are conceivably observable arbitrarily many times). Thus, for instance, in the process of designing a space vehicle intended for only one flight one may investigate the reliability of the whole complex of equipment needed for the flight, make the calculation in such a way that all equipment would work with certainty without failures. The strength of science consists in its ability to discover and theoretically forecast new phenomena based on a few simple concepts deduced from direct observations.

1.1.4 Object of probability theory

The branch of mathematics which studies the laws of mass random phenomena is called probability theory. The methods of probability theory, called probability or statistical methods, enable us to make calculations allowing for some practical inferences concerning random phenomena to be arrived at. Like any applied science the probability theory requires initial experimental data for calculations. The part of probability theory which studies the methods for handling empirical data and deducing from them necessary inferences is called mathematical statistics.

The probability theory is a powerful instrument of scientific research and therefore it finds a great number of diverse applications in various fields of science and engineering. The areas of its applications are continuously extending. In the last century probability theory was applied to the theory of measurements, to the theory of artillery fire and to physics. In our century the field of applications of probability theory was gradually extended to aerodynamics and hydrodynamics, radio engineering, control theory, flight dynamics, communication theory, construction mechanics, machines and mechanisms theory, theory of sea waves and ship motions, meteorology and many other fields of knowledge. It is difficult to name any branch of science which does not exploit the probability methods. Probability theory became the main research instrument in the modern theory of control processes and theoretical radio engineering. All theory of modern complex systems and control processes is based on statistical methods. Probability theory is the foundation for reliability theory of technical systems and for many other applied scientific theories.

This process of continuous extension of application areas of probability theory is quite natural and easy to explain. The point is that at the beginning of the development of every branch of science man is going to discover the main laws of this science and he is content with a rather rough coincidence of calculation results with empirical data. Besides, at the initial stage experimental equipment is primitive and cannot provide a high accuracy of measurements. In the development of science the requirements to the accuracy of calculations are constantly raising, experimental equipment is improving and random phenomena which might be neglected at the beginning of the development of a given branch of science become more and more apparent. As a result of this the old theory begins to diverge from experimental data and the necessity to apply probability theory arises. Invariably in all cases probability theory gives new means of describing phenomena more precisely and providing the coincidence of the results of theoretical calculations with experimental data. It happened at the beginning of the thirties with turbulence theory in aerodynamics and in the forties with automatic control theory and radio engineering, as well as with other applied scientific theories.

The peculiarity of probability methods consists in the fact that they consider an investigated phenomenon as a whole; they study the results of joint action of all causal relationships which are impossible to trace if each is taken separately.

1.2 Statistical approach to the description of random phenomena

1.2.1 Trial, event, random variable

The departing points for the construction of probability theory, as for any other theoretical science, are some experimental facts which underly the corresponding abstract notions. To speak about these experimental facts it is necessary to introduce some terms.

The observation of any phenomenon under some complex of conditions and actions, which have to be strictly fulfilled every time while repeating a given experiment, we shall call a trial. The observation of the same phenomenon under another complex of conditions and actions will be another trial.

The results of a trial may be characterized qualitatively and quantitatively.

A qualitative characteristic of a trial consists in the registration of some fact, i.e. in determining whether the results of a trial possess some property or not. Any such a fact is called an event. We say that an event appeared (occurred) or an event did not appear (occur) as a result of a trial.

Examples of events are: failure of a device in a given interval of time, the impact of a space vehicle with a meteorite, the gain or the loss in a game, the receiving of m gains in n games.

The events are denoted by capital Latin letters, usually by the initial letters, for example, A, B, C.

A quantitative characteristic of a trial consists in determining the values of some variables obtained as a result of a trial. Such variables, which can assume different values as the result of a trial in such a way that it is impossible to predict these values, are called random variables.

Examples of random variables are: errors and results of measurement, the time interval of faultless functioning of a device or a system, the height and the weight of a randomly chosen person, the coordinates of the point at which a meteorite will hit a space vehicle, the number of gains in n games.

We shall denote random variables by capital letters mainly from the end of the Latin alphabet, and their possible values by corresponding small letters. For example, random variables we shall denote by X, Y, Z and their specific values which can be attained as a result of a trial x, y, z respectively. These values are also called sample values or realizations of the random variables X, Y, Z.

With any random variable one can connect various events. A typical event, connected with a random variable, is that as a result of a trial this random variable will assume some value belonging to a given set. Such an event is shortly called the occurrence of a random variable in a given set.

1.2.2 Frequency of an event

It is natural to compare the events with respect to the fact how often every event appears when the trial is repeated. If, when the trial is repeated, one event appears more often than another one says that the first event is more probable than the other. It is clear that for the comparison of events it is neccessary to suppose that the given trial may be repeated indefinitely many times. Later on we shall say in short "n trials are performed instead of the given trial is repeated n times".

The ratio of the number of appearances of an event to the total number of the trials is called the frequency of this event. Thus if an event A appears m times in n trials then its frequency in this series of trials is equal to m/n.

1.2.3 Conditional frequencies

Sometimes we have to determine the frequency of an event under the additional condition that some other event has occurred. In order to determine the frequency of the event A under the condition that the event B occurred it is necessary to take into account not all performed trials but only those in which the event B occurred.

Thus if the number of all trials is n, B appeared in m trials and A appeared together with B in k trials, then the frequency of the event A under the condition that B has occurred is equal to k/m. As a rule this frequency does not coincide with the frequency of the event A calculated for all n trials.

The frequency of an event A, calculated with accounting only those trials where another event B appeared, is called the conditional frequency of the event A relative to the event B.

1.2.4 Properties of frequencies

An event is called impossible and is denoted by ø, if it cannot occur as a result of a trial. An event is certain and is denoted by Ω, if it cannot fail to occur when a trial is performed.

The events A1,…, An are called exclusive in a given trial if no two of them can appear together as a result of this trial. Examples of exclusive events are: catching an infection, having contact with somebody suffering from influenza, and not catching; appearance of one, two, three dots when rolling dice once.

Two events exclusive in one type of trial may be non-exclusive in another. For example, gain and loss in one game are exclusive events, but they are non-exclusive if we consider two games as one trial.

Taking into account these definitions we can go on studying the main properties of frequencies of events.

(1) The frequency of any event is a non-negative number not exceeding unity, while the frequency of an impossible event is equal to zero, and the frequency of a certain event is equal to unity.

(2) The frequency of appearance of any one of exclusive events is equal to the sum of their frequencies. It follows immediately from the fact that the number of appearances of a complex event, which consists of the appearance of exclusive events, is equal to the sum of numbers of appearances of these events.

(3) The frequency of simultaneous appearance of two events A and B is equal to the frequency of one of them multiplied by the conditional frequency of the other. To prove this it is sufficient to note that if A appeared m times in n trials, and the event B appeared l times together with A, and k – l times without A then the frequency of their simultaneous appearance is l/n, the frequency of A is m/n and the conditional frequency of B relative to A is l/m.

It is obvious that if the frequency of an event in a given series of trials is 0 (or 1) it does not imply that the event is impossible (certain). Thus, for instance, if heads did not appear in five tossings of the coin it does not mean that the appearance of heads is impossible.

1.2.5 Probability of an event

An outstanding experimental fact, the main regularity which is observed in mass random phenomena, is the stability of frequencies of events in great numbers of trials. In a small number of trials the frequency of an event assumes quite randomly different values, but in an indefinitely large number of trials it tends to become stable close to some value which is typical for a given event.

Let a trial be repeated indefinitely and the frequency of an event be calculated after every trial, taking into account the total number of trials performed. At the beginning, when the number of trials is small, it is noticed that the random result of every trial essentially changes the frequency of an event. But as the number of trials is increasing the influence of the result of every new trial is decreasing. Thus, for instance, in the result of the thousandth trial the change of the frequency is less than 0.001. The frequency tends to become stable, settling near a particular value.

The stability of frequencies of events gives the grounds to assume that every event is connected with some number, the probability of this event, at which its frequency tends to become stable. Thus, for instance, the frequency of appearance of heads in a series of coin tossings must evidently stabilize around ½. Therefore the probability of heads appearance is ½.

We shall denote by P(A) the probability of an event A. Certainly this does not exclude the using of abbreviated notations, for instance, P(A) = p and so on.

The notion of probability of an event is basic in probability theory and therefore there is no need to define it. It represents the result of an abstraction necessary for construction of any theory. Distracting from complex and inessential variations of a frequency when the number of trials is increasing indefinitely and emphasizing the main and essential regularity which is observed in random phenomena—the stability of a frequency—we introduce the abstract notion of the probability of an event.

The probability of an event in a given trial is its objective characteristic. It has a quite definite value independently of whether we are going to perform trials or not.

In the same way the empirical notion of a conditional frequency gives rise to the abstract notion of a conditional probability. The conditional probability of an event A relative to an event B is denoted P(A | B).

1.2.6 Sample mean

The main problem of experimental study of random variables is to determine how the experimental points are distributed on the numerical axis, in the plane or in the space.

The set of values of observed random variables which are received as a result of a trial is called a sample.

Having received a sample it first of all is necessary to determine the position of the values of the random variables on the numerical axis and their scatter, i.e. the size of the region occupied by them.

The mean arithmetic value of a random variable is usually taken as the characteristic of the position of experimental points and is called a sample mean.

Suppose that a random variable X assumed the values x1,…, xn as a result of n trials. Then the sample mean is determined by

(1.1)

1.2.7 Sample variance and sample mean square deviation

The mean arithmetic value of the squares of deviations of experimental values of a random variable from the sample mean is usually taken as the experimental characteristic of the scatter of values of a scalar random variable.

If as a result of n trials the random variable X assumed the values x1,…, xn then its sample variance is determined by

(1.2)

The sample variance has the dimension of the square of a random variable and is difficult to represent visually. Therefore the sample mean square (or standard) deviation .

For practical purposes formula (1.2) is brought into a more convenient form. Remembering that according to (1.1)

we may rewrite (1.2) in the form

(1.3)

If necessary, for avoiding differences of large numbers, an arbitrary number in the centre of the interval occupied by experimental values may be subtracted from all the values x1,…, xn (the rule of false zero).

1.2.8 Least-squares method

When studying jointly several random variables then, besides their sample means and variances, it is necessary to determine some characteristics of dependence between them.

The least-squares method is generally used for searching approximate dependences between variables studied experimentally.

Suppose it is required to find the dependence between the observed variables x and y ), depending on certain parameters c1,…, cN (for instance, a linear combination of N definite functions with indefinite coefficients) and choose those parameters in such a way that the sum of squares of the errors with respect to the approximate formula y = φ(x; c1,…, cN) in all experimental points be minimal:

This is the least-squares method.

For minimizing δ one may use various methods for finding the function extremum, depending on the way the function is defined and on the complexity of its calculation. In particular, one may use the conventional method of equating to zero the first derivatives of δ with respect to c1,…, cN and solving the equations obtained with respect to c1,…, cN, with subsequent studying of the behaviour of function δ in the vicinity of the obtained solution.

1.2.9 Sample covariance and sample correlation coefficient

Suppose that the random variables X and Y occurred as pairs of values x1, y1;…,; xn, yn in n trials. In order to find a suitable characteristic of dependence between the random variables X and Y we shall fit, by the least-squares method, the best linear expression for Y in terms of X of the form

(1.4)

are the sample means of the random variables X and Y respectively.

To determine the coefficient c in (4) by the least-squares method we equate to zero the derivative with respect to c of the sum of squares of the errors in all experimental points

After solving the equation obtained relative to c we find using (1.2)

The quantity

(1.5)

is called the sample covariance of the random variables X and Y. As a measure of interdependence between X and Y we take the dimensionless quantity

(1.6)

which is called the sample correlation coefficient of the random variables X and Y.

Since

we may rewrite (1.5) as

(1.7)

This formula is generally used in practice. If necessary, in order to avoid differences of close numbers, we subtract from the coordinates of all experimental points the coordinates of some point in the middle of the region occupied by the points.

, d*x, d*y, k*xy, σ*x, σ*y, r*xy serve as a rough characteristic of the distribution of the experimental points on the plane. Similarly the distribution of experimental points in m-dimensional space, obtained as a result of joint observations of m random variables, may be characterized by sample means, variances (mean squares deviations) and covariances (correlation coefficients).†

1.2.10 Histogram

In order to receive a more complete idea about the distribution of experimental points the region which is occupied by them is usually partitioned into intervals (rectangles, parallelepipeds) and the frequencies of occurrences of the points in these intervals (rectangles, parallelepipeds) are calculated. Dividing these frequencies by the lengths of the respective intervals (areas of rectangles, volumes of parallelepipeds) we find the relative densities of the experimental points in the respective parts of the region occupied by them. The distribution of experimental points obtained in such a way may be represented graphically. For this purpose we draw at the base of each interval the rectangle whose height is equal to the value of the relative density of the experimental points in this interval (Fig. 1.1). The step line thus obtained is called a histogram.

FIG. 1.1

For constructing a histogram it is recommended to choose intervals (rectangles, parallelepipeds) in such a way that each interval contains at least 10 points (to arrange this one may take intervals of different lengths). It is easy to understand that a histogram can be obtained only for a sufficiently large number of experimental points (for a one-dimensional histogram no less than 100).†

1.2.11 Grouped sample

To simplify the calculation of sample means, variances and covariances when they are determined simultaneously with the construction of histograms all experimental points in a given interval (rectangle) are usually considered as coinciding with its centre. A new sample obtained in such a way is called a grouped sample.

If we use a grouped sample we may write formulae (1.1)–(1.3), (1.5) and (1.7) in the form

(1.8)

where r is the number of intervals containing experimental points on each of the x, y rr the frequencies of occurrences of points in the intervals of xμv (μ, v = 1,…, r) the frequencies of occurrences of points in the rectangles into which the plane xy is partitioned.†

The experimental characteristics of random variables just considered underlie the respective abstract notions of probability theory exactly in the same way as the frequency of an event underlies the abstract notion of the probability of an event.

1.3 Direct evaluation of probabilities

1.3.1 Equiprobable outcomes of a trial

The example of a trial with coin tossing mentioned in Section 1.2.5 shows that the probabilities of some events are easily evaluated. Let us consider the general scheme of such trials. Assume that a trial has n possible outcomes so that one and only one of these n outcomes appears in every realization of this trial, and that there is no reason to assume that at an unlimited repetition of this trial one of the outcomes may appear more often than any other. In this case the probability of every outcome is obviously equal to 1/n as their frequencies must stabilize at the repetition of the trial near the same number and their sum must be equal to 1. In other words, the given trial has n equiprobable outcomes. In our example with coin tossings we have two such equiprobable outcomes—the appearance of heads and the appearance of tails, and the probability of each of them is ½.

1.3.2 Scheme of chances

We assume that at n equiprobable outcomes of a trial we are interested