Create, transform, and summarize custom random variables with distribution functions (analogues of p*(), d*(), q*(), and r*() functions from base R), all of which called “pdqr-functions”. General idea is to work with manually created distributions which can be described in four interchangeable functional ways.

Typical usage is to:

  1. Create pdqr-function from sample or data frame (with new_*() family), and/or convert from some other existing distribution function (with as_*() family).
  2. Make necessary transformations with form_*() family.
  3. Compute summary values with summ_*() family.

Two types of pdqr-functions, representing different types of distributions, are supported:

  • Type “discrete”: random variable has finite number of output values. Pdqr-function is explicitly defined by the collection of its values with their corresponding probability. Usually is used when underlying distribution is discrete (even if in theory there are infinite number of output values).
  • Type “continuous”: random variable has infinite number of output values in the form of continuous random variable. It is explicitly defined by piecewise-linear density function with finite support and values. Usually is used when underlying distribution is continuous (even if in theory it has infinite support and/or density values).

Implemented approaches often emphasize approximate and numerical solutions:

  • All distributions assume finite support (output values are bounded from below and above) and finite values of density function (density function in case of “continuous” type can’t go to infinity).
  • Some methods implemented with simulation techniques.

Note that to fully use this package, one needs to be familiar with basics of probability theory (concepts such as probability, distribution, density, etc.).

This README covers the following topics:

Installation

‘pdqr’ is not yet on CRAN. You can install the development version from GitHub with:

Quick examples

Generate a sample from a distribution defined by some reference sample:

Compute winsorized mean:

Compute and visualize distribution of difference of sample means:

Compute and visualize 95% highest density region for mixture of normal distributions:

# Create a list of pdqr-functions
norm_list <- list(
  as_d(dnorm), as_d(dnorm, mean = 2, sd = 0.25), as_d(dnorm, mean = 4, sd = 0.5)
)

# Form a mixture with custom weights
norm_mix <- form_mix(norm_list, weights = c(0.6, 0.2, 0.2))

# Compute 95% highest density region
(norm_hdr <- summ_hdr(norm_mix, level = 0.95))
#>          left      right
#> 1 -1.82436425 2.53094176
#> 2  3.19653241 4.79331185

# Visualize
plot(norm_mix, main = "95% highest density region for normal mixture")
region_draw(norm_hdr)

Create with new_*()

All new_*() functions create a pdqr-function of certain class (“p”, “d”, “q”, or “r”) and type (“discrete” or “continuous”) based on sample or data frame (with appropriate structure):

  • Sample input is converted into data frame of appropriate structure that defines distribution (see next list item). It is done based on type. For “discrete” type it gets tabulated with frequency of unique values serving as their probability. For “continuous” type distribution density is estimated using density() function if input has at least 2 elements. For 1 element special “dirac-like” pdqr-function is created: an approximation of single number as triangular distribution with very narrow support (1e-8 order of magnitude).
  • Data frame input should completely define distribution. For “discrete” type it should have “x” and “prob” columns for output values and their probabilities. For “continuous” type - “x” and “y” columns for points, which define piecewise-linear continuous density function. Columns “prob” and “y” will be automatically normalized to represent proper distribution: sum of “prob” will be 1 and total square under graph of piecewise-linear function will be 1.

All information about distribution that pdqr-function represents is stored in its “x_tbl” metadata: a data frame describing distribution with format similar to data frame input of new_*() functions. One can get it using meta_x_tbl() function.

Pdqr class correspond to the following functions describing distribution:

  • P-function is a cumulative distribution function. Created with new_p().
  • D-function is a probability mass function for “discrete” type and density function for “continuous”. Created with new_d(). Generally speaking, it is a derivative of distribution’s p-function.
  • Q-function is a quantile function. Created with new_q(). Inverse of distribution’s p-function.
  • R-function is a random generation function. Created with new_r(). Generates a random sample from distribution.

For more details see vignette about creating pdqr-functions.

Create pdqr-function from sample

# Treat input as discrete
(p_mpg_dis <- new_p(mtcars$mpg, type = "discrete"))
#> Cumulative distribution function of discrete type
#> Support: [10.4, 33.9] (25 elements)
(d_mpg_dis <- new_d(mtcars$mpg, type = "discrete"))
#> Probability mass function of discrete type
#> Support: [10.4, 33.9] (25 elements)
(q_mpg_dis <- new_q(mtcars$mpg, type = "discrete"))
#> Quantile function of discrete type
#> Support: [10.4, 33.9] (25 elements)
(r_mpg_dis <- new_r(mtcars$mpg, type = "discrete"))
#> Random generation function of discrete type
#> Support: [10.4, 33.9] (25 elements)

  # "x_tbl" metadata is the same for all `*_mpg_dis()` pdqr-functions
head(meta_x_tbl(p_mpg_dis), n = 3)
#>      x    prob cumprob
#> 1 10.4 0.06250 0.06250
#> 2 13.3 0.03125 0.09375
#> 3 14.3 0.03125 0.12500

# Treat input as continuous
(p_mpg_con <- new_p(mtcars$mpg, type = "continuous"))
#> Cumulative distribution function of continuous type
#> Support: ~[2.96996, 41.33004] (511 intervals)
(d_mpg_con <- new_d(mtcars$mpg, type = "continuous"))
#> Density function of continuous type
#> Support: ~[2.96996, 41.33004] (511 intervals)
(q_mpg_con <- new_q(mtcars$mpg, type = "continuous"))
#> Quantile function of continuous type
#> Support: ~[2.96996, 41.33004] (511 intervals)
(r_mpg_con <- new_r(mtcars$mpg, type = "continuous"))
#> Random generation function of continuous type
#> Support: ~[2.96996, 41.33004] (511 intervals)

  # "x_tbl" metadata is the same for all `*_mpg_con()` pdqr-functions
head(meta_x_tbl(p_mpg_con), n = 3)
#>            x              y        cumprob
#> 1 2.96996269 0.000114133557 0.00000000e+00
#> 2 3.04503133 0.000125168087 8.98202438e-06
#> 3 3.12009996 0.000136934574 1.88198694e-05

# Output of `new_*()` is actually a function
p_mpg_dis(15:20)
#> [1] 0.18750 0.31250 0.34375 0.40625 0.46875 0.56250

  # Random generation. "discrete" type generates only values from input
r_mpg_dis(10)
#>  [1] 10.4 33.9 26.0 19.2 24.4 18.7 17.8 18.7 15.5 21.4
r_mpg_con(10)
#>  [1] 35.7089680 33.3634914 31.1541225 15.6482281 20.8554689 13.7275908
#>  [7] 11.3686051 16.7155604 20.6710197 20.5132727

# Special case of dirac-like pdqr-function, which numerically approximates
# single number with distribution with narrow support
(r_dirac <- new_r(3.14, "continuous"))
#> Random generation function of continuous type
#> Support: ~[3.14, 3.14] (2 intervals)
meta_x_tbl(r_dirac)
#>            x         y cumprob
#> 1 3.13999999         0     0.0
#> 2 3.14000000 100000001     0.5
#> 3 3.14000001         0     1.0

Convert with as_*()

Family of as_*() functions should be used to convert existing distribution functions into desired class (“p”, “d”, “q”, or “r”). Roughly, this is a new_*() family but with function as an input.

There are two main use cases:

  • Convert existing pdqr-functions to different type.
  • Convert (create) pdqr-function based on some other user-supplied distribution function.

For more details see vignette about converting pdqr-functions.

Existing pdqr-functions

Converting existing pdqr-function to desired type is done straightforwardly by changing function’s class without touching the underlying distribution (“x_tbl” metadata is the same):

Other distribution functions

Another important use case for as_*() functions is to convert some other distribution functions to be pdqr-functions. Except small number of special cases, output of as_*() function will have “continuous” type. The reason is because identifying exact values of distribution in discrete case is very hard in this setup (when almost nothing is known about the input function). It is assumed that if user knows those values, some new_*() function with data frame input can be used to create arbitrary discrete pdqr-function.

General conversion algorithm is as follows:

  • If user didn’t supply support, detect it using algorithms targeted for every pdqr class separately. If input function belongs to a certain set of “honored” distributions (currently, it is all common univariate distributions from ‘stats’ package), support is detected in predefined way.
  • Using detected support, data frame input for corresponding new_*() function is created which approximates input function. Approximation precision can be tweaked with n_grid (and n_sample for as_r()) argument.

Note that output is usually an approximation of input due to the following facts:

  • Output density has piecewise-linear nature, which is almost never the case for input function.
  • Possible infinite tails are removed to obtain finite support. Usually output support “loses” only around 1e-6 probability on each infinite tail.
  • Possible infinite values of density are linearly approximated from neighborhood points.

Transform with form_*() and base operations

Concept of form functions is to take one or more pdqr-function(s) and return a transformed pdqr-function. Argument method is used to choose function-specific algorithm of computation.

For more details see vignette about transforming pdqr-functions.

form_*() family

There are several form_*() functions. Here are some examples:

# Transform support of pdqr-function with `form_resupport()`. Usually useful
# for dealing with bounded values.
d_smpl <- new_d(runif(1000), type = "continuous")
d_smpl_bounded <- form_resupport(d_smpl, support = c(0, 1), method = "reflect")

plot(d_smpl, main = "Estimating density of bounded quantity", col = "black")
lines(d_smpl_bounded, col = "blue")
  # Refernece uniform distribution
lines(as_d(dunif), col = "red")

Summarize with summ_*()

Concept of summary functions is to take one or more pdqr-function(s) and return a summary value. Argument method is used to choose function-specific algorithm of computation.

For more details see vignette about summarizing pdqr-functions.

Basic summary

Regions

Distributions can be summarized with regions: union of closed intervals. They are represented as data frame with rows representing intervals and two columns “left” and “right” with left and right interval edges respectively.

Separation, classification, and ROC curve

Function summ_separation() computes a threshold that optimally separates distributions represented by pair of input pdqr-functions. In other words, summ_separation() solves a binary classification problem with one-dimensional linear classifier: values not more than some threshold are classified as one class, and more than threshold - as another. Order of input functions doesn’t matter.

summ_separation(my_d, my_d_2, method = "KS")
#> [1] 0.637042714
summ_separation(my_d, my_d_2, method = "F1")
#> [1] -6.02372908e-05

Functions summ_classmetric() and summ_classmetric_df() compute metric of classification setup, similar to one used in summ_separation(). Here classifier threshold should be supplied and order of input matters. Classification is assumed to be done as follows: any x value not more than threshold value is classified as “negative”; if strictly greater - “positive”. Classification metrics are computed based on two pdqr-functions: f, which represents the distribution of values which should be classified as “negative” (“true negative”), and g - the same for “positive” (“true positive”).

With summ_roc() and summ_rocauc() one can compute data frame of ROC curve points and ROC AUC value respectively. There is also a roc_plot() function for predefined plotting of ROC curve.

Similar packages

  • “distrXXX”-family of packages: S4 classes for distributions.
  • distr6: Unified and Object Oriented Probability Distribution Interface for R written in R6.
  • distributions3: Probability Distributions as S3 Objects.
  • fitdistrplus: Help to Fit of a Parametric Distribution to Non-Censored or Censored Data.
  • Probability Distributions CRAN Task View has more examples of packages intended to work with probability distributions.