Analysis of Categorical Data

Introduction to categorical data

Many experiments result in measurements that are qualitative (categorical) rather than quantitative (numbers).

Example

• $n$ manufactured items are sampled and categorized into "acceptable", "seconds", or "rejects"
• $n$ employees are surveyed and classified into one of five income brackets.

We can represent each outcome $X_i$ as a length-$k$ vector such that $X_{ij}=1$ if $X_i$ belongs to the $j$th category and $0$ otherwise for $j=1,\dots,k$.

Note the sample $(X_1,\dots,X_n)$ can be summarized by providing the counts of the number of measurements that fall into each of the categories, i.e., by the random vector $N = (N_1,\dots,N_k)$ such that $N_j = \sum_{i=1}^n X_{ij}$.

Definition (multinomial distribution) Consider a random experiment such that

• the experiment consists of $n$ independent and identical trials, and
• the outcome of each trial falls into exactly one of $k$ distinct categories.

Then the counts for each category ($N_1,\dots,N_{k}$) follow a multinomial distribution with the number of trials $n$ and cell probabilities ($p_1,\dots,p_k$), i.e. ${\rm Mult}(n,p_1,\dots,p_k)$.

The pmf for ${\rm Mult}(n,p_1,\dots,p_k)$ is

where the cell probabilities sum up to $1$, i.e., $p_1+\dots+p_k=1$

Remark 1: a Binomial distribution is a special case of the multinomial distribution when $k=2$, i.e., ${\rm Bin}(n,p)={\rm Mult}(n,p_1,1-p_1)$.

Remark 2: Suppose $X_1,\dots,X_n$ are i.i.d., such that $X_{i}\sim {\rm Mult}(1,p_1,p_2,\dots,p_k)$. Then $N = \sum_{i=1}^n X_i \sim {\rm Mult}(n,p_1,\dots,p_k)$.

Using observed counts $(n_1,\dots,n_k)$ (i.e., a realization of $N \sim {\rm Mult}(n,p_1,\dots,p_k)$), we would like to make inferences about the category probabilities $p_1, p_2,..., p_k$ .

Chi-Square Tests

Setting

We have a random sample $(X_1,\dots,X_n)$ where each $X_i \sim {\rm Mult}(1,p_1,\dots,p_k)$. Equivalently, we have a random variable $N = (N_{1},\dots,N_{k})$ such that $N \sim {\rm Mult}(n, p_1,\dots,p_k)$.

1. Null and alternative hypothesis:

• $H_0: p_1 = p_{10},\dots,p_k = p_{k0}$

• $H_1:$ at least one $p_i$ differs from the hypothesized value $p_{i0}$, $i=1\dots,k$,

  where $p_{10},\dots,p_{k0}$ are pre-specified probabilities such that $\sum_{i=1}^k p_{i0} = 1$.

2. Test statistic:

3. Null distribution:

   $$Q \underset{H_0}{\dot\sim} \chi^2(k-1)$$

4. Given a significance level $\alpha$, the rejection region for the observed test statistic $q_{obs}$: $\{q_{obs}; q_{obs} \ge \chi^2_{\alpha}(k-1)\}$, where the observed test statistic $q_{obs} =\sum_{i=1}^k \frac{(n_i - n p_{i0})^2}{n p_{i0}}$.

   The test of "rejecting $H_0$ if the observed test statistic $q_{obs}$ exceeds $\chi^2_\alpha(k-1)$ " is an approximate sig. level $\alpha$ test.

Remark 1: this is an asymptotic test. The sample size $n$ needs to be large to approximate the null distribution with the $\chi^2$ distribution. A rule of thumb is to check whether $n p_{i}\ge 5$ for $i=1,\dots,k$. In practice, we check whether each observed count exceeds $5$ or not.

Remark 2: this test has a connection with the asymptotic LRT. In fact, it can be shown that $-2\log \lambda\approx Q$ when $n$ is large. In particular, the degrees of freedom of the $\chi^2$ distribution is the difference of the number of free parameters in the null and full parameter spaces.

• Number of free parameters in the null parameter space = $0$
• Number of free parameters in the full parameter space = $k-1$.

Example: A group of rats, one by one, proceed down a ramp to one of three doors. We wish to test the hypothesis that the rats have no preference concerning the choice of a door. Suppose that the rats were sent down the ramp $n = 90$ times and that the three observed cell frequencies were $n_1 = 23, n_2 = 36$, and $n_3 = 31$.

Chi-square Goodness of Fit (GoF) test

The $\chi^2$ test above can be used to test whether sample data $(x_1,\dots,x_n)$ are from a given distribution $F_0$ or not.

Idea: partition the range of $X$ into $k$ distinct regions ($A_1,\dots,A_k$) and compare observed and expected counts for each region.

• $H_0$: $X\sim F_0$ vs. $H_1: X \sim F\ne F_0$ $\rightarrow$
• $H_0: (p_1,\dots,p_k) = (p_{10},\dots,p_{k0})$ vs. $H_1$: at least one $p_i$ differs from the hypothesized probability $p_{i0}$, where $p_{i0} = P_{X\sim F_0}(X \in A_i )$.

Example Let $X$ denote the number of heads that occur when four identical coins are tossed at random. Under the assumption that the four coins are independent and the probability of heads on each coin is $1/2$, $X$ is ${\rm Bin}(4, 1/2)$. One hundred repetitions of this experiment resulted in $0, 1, 2, 3$, and $4$ heads being observed on $7, 18, 40, 31$, and $4$ trials, respectively. Do these results support the assumptions? Make a conclusion at $\alpha=.05$.

Often, we are interested in testing whether the random variable of interest $X$ is from a certain family of distributions or not. Note, in such case, the distribution of $X$ is not completely specified under $H_0$.

1. Null and alternative hypotheses

• $H_0$: $X\sim F_\theta$ vs. $H_1: X \sim F\ne F_\theta$ $\rightarrow$
• $H_0: (p_1,\dots,p_k) = (p_{10}(\theta),\dots,p_{k0}(\theta))$ vs. $H_1$: at least one $p_i$ differs from the hypothesized probability $p_{i0}(\theta)$, where $p_{i0}(\theta) = P_{X\sim F_{\theta}}(X \in A_i )$.

2. Test statistic:

where $\hat{\theta}$ is a ML estimator of $\theta$ under $H_0$ (i.e., $X \sim F_\theta$).

3. Null distribution:

   $$Q \underset{H_0}{\dot\sim} \chi^2(k-1-s)$$

   where $s$ is the length of $\theta$ (i.e., the number of free parameters under $H_0$).

4. Given a significance level $\alpha$, the rejection region for the observed test statistic $q_{obs}$: $\{q_{obs}; q_{obs} \ge \chi^2_{\alpha}(k-1-s)\}$, where the observed test statistic $q_{obs} =\sum_{i=1}^k \frac{(n_i - n p_{i0}(\hat{\theta}))^2}{n p_{i0}(\hat{\theta})}$.

   The test of "rejecting $H_0$ if the observed test statistic $q_{obs}$ exceeds $\chi^2_\alpha(k-1-s)$" is an approximate sig. level $\alpha$ test.

    

Example The number of accidents $X$ per week at an intersection was checked for $n = 50$ weeks; Out of $50$ weeks, $32$ weeks had no accidents, $12$ weeks had one accident, and $6$ weeks had $2$ accidents. Test the hypothesis that the random variable $X$ has a Poisson distribution, assuming the observations to be independent. Use $α = .05$.

Chi-square tests of independence

Two-way contingency table (a table for two categorical variables)

Example:

A statistics 415 instructor wants to know if there is a relationship between favorite color (red or yellow) and the preferred condiment on a corn dog. The following table summarizes the results.

A general example of our contingency table with two classifying factors can be displayed as follows.

• $X_{ij}$ denotes the count in the $(i,j)$ cell, i.e., the count in row $i$ and column $j$. We use $X_{i.}=X_{i1}+X_{i2}+...+X_{ic}$ to denote the marginal total count for the $i$th row, and $X_{.j}=X_{1j}+X_{2j}+...+X_{cj}$ the marginal total count for the jth column, and $X_{..}$ the overall total count, which is equal to $n$, the sample size.
• When we talk about an $r\times c$ contingency table, we mean a table with $r$ categories for the row variable and $c$ categories for the column variable

In some problems, the $rc$ counts $X_{ij}$, $i=1,...,r$, $j=1,...,c$ can be modeled using a multinomial distribution

• Assume each of the $n$ observations results in an outcome that can be classified by two attributes

  • e.g, "Treatment/Control" on the rows, "Disease/No disease" on the columns

• The first attribute is randomly assigned to one and only one of $r$ mutually exclusive and exhaustive events $A_1,...,A_{r}$ and the second attribute falls into one and only one the $c$ mutually exclusive events $B_1,...,B_c$

• Define $p_{ij}=P(A_i\cap B_j)$

Chi-square tests of independence

Chi-square tests of independence aim to answer the question: for a single observation, is the row assignment statistically independent of the column assignment?

1. In terms of hypotheses, we will test

$H_0:P(A_i\cap B_j)=P(A_i)P(B_j)$, $i=1,...,r$, $j=1,...,c$

vs.

$H_1: P(A_{i}\cap B_j)\neq P(A_i)P(B_j)$ for at least 1 $i,j$ pair.

Defining

$p_{i.}=P(A_i),\, p_{.j}=P(B_j)$

we can rewrite $H_0$ and $H_1$ as

$H_0:p_{ij}=p_{i.}p_{.j}$, $i=1,...,r$, $j=1,...,c$

$H_1:p_{ij}\neq p_{i.}p_{.j}$ for at least 1 $i,j$ pair

2. Test statistic:

$Q=\sum_{i=1}^{r}\sum_{j=1}^{c}\frac{\{X_{ij}-n(X_{i.}/n)(X_{.j}/n)\}^2}{n(X_{i.}/n)(X_{.j}/n)}=\sum_{i=1}^{r}\sum_{j=1}^{c}\frac{(O_{ij}-E_{ij})^2}{E_{ij}}$

where

• the observed value at cell $(i,j)$ : $O_{ij}=X_{ij}$

  • the expected value at cell $(i,j)$: $E_{ij}=n(X_{i.}/n)(X_{.j}/n)$

3. Under $H_0:p_{ij}=p_{i.}p_{.j}$, $Q$ follows a $\chi^2$ distribution with $(r-1)(c-1)$ degrees of freedom-that is, $Q\sim \chi^2((r-1)(c-1))$.

• the number of free parameters in the null parameter space: $(r-1)+(c- 1)$
• the number of free parameters in the full parameter space: $rc-1$
• the difference in the number of free parameters: $(rc-1)-[(r-1)+(c-1)]=(r-1)(c-1)$

4. Given a significance level $\alpha$, the rejection region for the observed test statistic $q_{obs}$: $\{q_{obs}; q_{obs} \ge \chi^2_{\alpha}((r-1)(c-1))\}$.

• Large discrepancy between observed and expected counts = favors the alternative hypothesis

Example:

Test whether there is a relationship between favorite color (red or yellow) and the preferred condiment on a corn dog at $\alpha=10\%$.