deltatest {deltatest} | R Documentation |
Two Sample Z-Test for Ratio Metrics Using the Delta Method
Description
Performs two sample Z-test to compare the ratio metrics between two groups using the delta method. The Delta method is used to estimate the variance by accounting for the correlation between the numerator and denominator of ratio metrics.
Usage
deltatest(
data,
formula,
by,
group_names = "auto",
type = c("difference", "relative_change"),
bias_correction = FALSE,
alternative = c("two.sided", "less", "greater"),
conf.level = 0.95,
na.rm = FALSE,
quiet = FALSE
)
Arguments
data |
data.frame containing the numerator and denominator columns of the ratio metric, aggregated by randomization unit. It also includes a column indicating the assigned group (control or treatment). For example, if randomizing by user while the metric is click-through rate (CTR) per page-view, the numerator is the number of clicks per user, and the denominator is the number of page views per user. |
formula |
expression representing the ratio metric. It can be written in
three styles: standard formula |
by |
character string or symbol that indicates the group column. If the
group column is specified in the |
group_names |
character vector of length 2 or |
type |
character string specifying the test type. If |
bias_correction |
logical value indicating whether correction to the
mean of the metric is performed using the second-order term of the Taylor
expansion. The default is |
alternative |
character string specifying the alternative hypothesis,
must be one of |
conf.level |
numeric value specifying the confidence level of the interval. The default is 0.95. |
na.rm |
logical value. If |
quiet |
logical value indicating whether messages should be displayed
during the execution of the function. The default is |
Value
A list with class "htest"
containing following components:
statistic |
the value of the Z-statistic. |
p.value |
the p-value for the test. |
conf.int |
a confidence interval for the difference or relative change appropriate to the specified alternative hypothesis. |
estimate |
the estimated means of the two groups, and the difference or relative change. |
null.value |
the hypothesized value of the difference or relative change in means under the null hypothesis. |
stderr |
the standard error of the difference or relative change. |
alternative |
a character string describing the alternative hypothesis. |
method |
a character string describing the method used. |
data.name |
the name of the data. |
References
Deng, A., Knoblich, U., & Lu, J. (2018). Applying the Delta Method in Metric Analytics: A Practical Guide with Novel Ideas. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. doi:10.1145/3219819.3219919
Examples
library(dplyr)
library(deltatest)
n_user <- 2000
set.seed(314)
df <- deltatest::generate_dummy_data(n_user) |>
group_by(user_id, group) |>
summarise(click = sum(metric), pageview = n(), .groups = "drop")
deltatest(df, click / pageview, by = group)