(COMP 233) Probability and Statics - Probability, Random variables and PMF&PDF&CDF

[ Probability Theory ]

 

Probability theory is the study of uncertainty. It is about the concepts from probability theory for deriving machine learning algorithms and delves(=examines) into a branch of analysis known as measure theory.

1      Elements of probability

 

-       Sample Space  Ω : The set of all the outcomes of a random experiment.

-       Sets of Events(=event space) : F: A set whose elements A ∈ F(called events) are subsets of Ω

-       Probablity measure : A function P : F -> R that satisfies the following properties,

                        - P(A) ≥ 0, for all A ∈ F - P(Ω) = 1

- If A1, A2, . . . are disjoint events (i.e., Ai ∩ Aj = ∅ whenever i '= j), then

  P(∪iAi) = ∑P(Ai)

 

=>  ‘Axioms of Probability’ = Sample space/Sets of Events / Probability measure

 

=> Properties :

- If A ⊆ B =⇒ P(A) ≤ P(B).

- P(A ∩ B) ≤ min(P(A), P(B)).

- (Union Bound) P(A ∪ B) ≤ P(A) + P(B).

- P(Ω \ A) = 1 − P(A).

- (Law of Total Probability) If A1, . . . , Ak are a set of disjoint events such that ∪k i=1Ai = Ω, then "k i=1 P(Ak) = 1.

 

1.1  Conditional probability and independence

Let B be an event with non-zero probability.

The conditional probability of any event A given B is defined as,

 

 

In other words, P(A|B) is the probability measure of the event A after observing the occurrence of event B.

 

To events are called independent if and only if P(A∩B) = P(A)P(B).

->  B does not have any effect on the probability of A. (vice versa)

 

[ Probability Theory ] Probability theory is the study of uncertainty. It is about the concepts from probability theory for deriving machine learning algorithms and delves(=examines) into a branch of analysis known as measure theory. 1      Elements of probability -       Sample Space  Ω : The set of all the outcomes of a random experiment. -       Sets of Events(=event space) : F: A set whose elements A ∈ F(called events) are subsets of Ω -       Probablity measure : A function P : F -> R that satisfies the following properties,                         - P(A) ≥ 0, for all A ∈ F - P(Ω) = 1 - If A1, A2, . . . are disjoint events (i.e., Ai ∩ Aj = ∅ whenever i '= j), then   P(∪iAi) = ∑P(Ai) =>  ‘Axioms of Probability’ = Sample space/Sets of Events / Probability measure => Properties : - If A ⊆ B =⇒ P(A) ≤ P(B). - P(A ∩ B) ≤ min(P(A), P(B)). - (Union Bound) P(A ∪ B) ≤ P(A) + P(B). - P(Ω \ A) = 1 − P(A). - (Law of Total Probability) If A1, . . . , Ak are a set of disjoint events such that ∪k i=1Ai = Ω, then "k i=1 P(Ak) = 1. 1.1  Conditional probability and independence Let B be an event with non-zero probability. The conditional probability of any event A given B is defined as,       In other words, P(A|B) is the probability measure of the event A after observing the occurrence of event B. To events are called independent if and only if P(A∩B) = P(A)P(B). ->  B does not have any effect on the probability of A. (vice versa)   2 Random variables   A random variable X is a function X : Ω -> R^2. Typically, we will denote random variables using upper case letters X(w) or more simply X ( where the dependence on the random outcome w is implied). We will denote the value that a random variable may take on using lower case letters x.   ð CDF/ PMF / PDF   2.1 Cumulative distribution functions : CDP -> In order to specify the probablility measures used when dealing with random variables, it is often convenient to specify alternative functions (CDFs, PDFs, and PMFs) from which the probability measure governing an experiment immediately follows.   Properties - 0 ≤ FX(x) ≤ 1. - limx→−∞ FX(x) = 0. - limx→∞ FX(x) = 1. - x ≤ y =⇒ FX(x) ≤ FX(y).     2.2  Probability mass functions : PMF   ->When a random variable X takes on a finite set of possible values ( X is a discrete random variable), a simpler way to represent the probability measure associated with a random variable is to directly specity the probability of each value that the random variable can assume.   Properties - 0 ≤ pX(x) ≤ 1. - " x∈V al(X) pX(x) = 1. - " x∈A pX(x) = P(X ∈ A).   2.3 Probability density functions : PDF -> For some continuous random variables, the cumulative distribution function Fx(s) is differentiable everywhere. In there cases, PDF is the derivative of the CDF.   Properties - fX(x) ≥ 0 . - # ∞ −∞ fX(x) = 1. - # x∈A fX(x)dx = P(X ∈ A)