small sample size hypothesis tests

Upper-tailed, Lower-tailed, Two-tailed Tests

The research or alternative hypothesis can take one of three forms. An investigator might believe that the parameter has increased, decreased or changed. For example, an investigator might hypothesize:  

The exact form of the research hypothesis depends on the investigator's belief about the parameter of interest and whether it has possibly increased, decreased or is different from the null value. The research hypothesis is set up by the investigator before any data are collected.

Rejection Region for Upper-Tailed Z Test (H : μ > μ ) with α=0.05

The decision rule is: Reject H if Z 1.645.

Rejection Region for Lower-Tailed Z Test (H 1 : μ < μ 0 ) with α =0.05

The decision rule is: Reject H 0 if Z < 1.645.

Rejection Region for Two-Tailed Z Test (H 1 : μ ≠ μ 0 ) with α =0.05

The decision rule is: Reject H 0 if Z < -1.960 or if Z > 1.960.

The complete table of critical values of Z for upper, lower and two-tailed tests can be found in the table of Z values to the right in "Other Resources."

Critical values of t for upper, lower and two-tailed tests can be found in the table of t values in "Other Resources."

• Step 4. Compute the test statistic.  

Here we compute the test statistic by substituting the observed sample data into the test statistic identified in Step 2.

• Step 5. Conclusion.  

The final conclusion is made by comparing the test statistic (which is a summary of the information observed in the sample) to the decision rule. The final conclusion will be either to reject the null hypothesis (because the sample data are very unlikely if the null hypothesis is true) or not to reject the null hypothesis (because the sample data are not very unlikely).  

If the null hypothesis is rejected, then an exact significance level is computed to describe the likelihood of observing the sample data assuming that the null hypothesis is true. The exact level of significance is called the p-value and it will be less than the chosen level of significance if we reject H 0 .

Statistical computing packages provide exact p-values as part of their standard output for hypothesis tests. In fact, when using a statistical computing package, the steps outlined about can be abbreviated. The hypotheses (step 1) should always be set up in advance of any analysis and the significance criterion should also be determined (e.g., α =0.05). Statistical computing packages will produce the test statistic (usually reporting the test statistic as t) and a p-value. The investigator can then determine statistical significance using the following: If p < α then reject H 0 .  

• Step 1. Set up hypotheses and determine level of significance

H 0 : μ = 191 H 1 : μ > 191                 α =0.05

The research hypothesis is that weights have increased, and therefore an upper tailed test is used.

• Step 2. Select the appropriate test statistic.

Because the sample size is large (n > 30) the appropriate test statistic is

• Step 3. Set up decision rule.  

In this example, we are performing an upper tailed test (H 1 : μ> 191), with a Z test statistic and selected α =0.05.   Reject H 0 if Z > 1.645.

We now substitute the sample data into the formula for the test statistic identified in Step 2.  

We reject H 0 because 2.38 > 1.645. We have statistically significant evidence at a =0.05, to show that the mean weight in men in 2006 is more than 191 pounds. Because we rejected the null hypothesis, we now approximate the p-value which is the likelihood of observing the sample data if the null hypothesis is true. An alternative definition of the p-value is the smallest level of significance where we can still reject H 0 . In this example, we observed Z=2.38 and for α=0.05, the critical value was 1.645. Because 2.38 exceeded 1.645 we rejected H 0 . In our conclusion we reported a statistically significant increase in mean weight at a 5% level of significance. Using the table of critical values for upper tailed tests, we can approximate the p-value. If we select α=0.025, the critical value is 1.96, and we still reject H 0 because 2.38 > 1.960. If we select α=0.010 the critical value is 2.326, and we still reject H 0 because 2.38 > 2.326. However, if we select α=0.005, the critical value is 2.576, and we cannot reject H 0 because 2.38 < 2.576. Therefore, the smallest α where we still reject H 0 is 0.010. This is the p-value. A statistical computing package would produce a more precise p-value which would be in between 0.005 and 0.010. Here we are approximating the p-value and would report p < 0.010.                  

In all tests of hypothesis, there are two types of errors that can be committed. The first is called a Type I error and refers to the situation where we incorrectly reject H 0 when in fact it is true. This is also called a false positive result (as we incorrectly conclude that the research hypothesis is true when in fact it is not). When we run a test of hypothesis and decide to reject H 0 (e.g., because the test statistic exceeds the critical value in an upper tailed test) then either we make a correct decision because the research hypothesis is true or we commit a Type I error. The different conclusions are summarized in the table below. Note that we will never know whether the null hypothesis is really true or false (i.e., we will never know which row of the following table reflects reality).

Table - Conclusions in Test of Hypothesis

In the first step of the hypothesis test, we select a level of significance, α, and α= P(Type I error). Because we purposely select a small value for α, we control the probability of committing a Type I error. For example, if we select α=0.05, and our test tells us to reject H 0 , then there is a 5% probability that we commit a Type I error. Most investigators are very comfortable with this and are confident when rejecting H 0 that the research hypothesis is true (as it is the more likely scenario when we reject H 0 ).

When we run a test of hypothesis and decide not to reject H 0 (e.g., because the test statistic is below the critical value in an upper tailed test) then either we make a correct decision because the null hypothesis is true or we commit a Type II error. Beta (β) represents the probability of a Type II error and is defined as follows: β=P(Type II error) = P(Do not Reject H 0 | H 0 is false). Unfortunately, we cannot choose β to be small (e.g., 0.05) to control the probability of committing a Type II error because β depends on several factors including the sample size, α, and the research hypothesis. When we do not reject H 0 , it may be very likely that we are committing a Type II error (i.e., failing to reject H 0 when in fact it is false). Therefore, when tests are run and the null hypothesis is not rejected we often make a weak concluding statement allowing for the possibility that we might be committing a Type II error. If we do not reject H 0 , we conclude that we do not have significant evidence to show that H 1 is true. We do not conclude that H 0 is true.

 The most common reason for a Type II error is a small sample size.

return to top | previous page | next page

Content ©2017. All Rights Reserved. Date last modified: November 6, 2017. Wayne W. LaMorte, MD, PhD, MPH

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Indian J Psychol Med
• v.42(1); Jan-Feb 2020

Sample Size and its Importance in Research

Chittaranjan andrade.

Clinical Psychopharmacology Unit, Department of Clinical Psychopharmacology and Neurotoxicology, National Institute of Mental Health and Neurosciences, Bengaluru, Karnataka, India

The sample size for a study needs to be estimated at the time the study is proposed; too large a sample is unnecessary and unethical, and too small a sample is unscientific and also unethical. The necessary sample size can be calculated, using statistical software, based on certain assumptions. If no assumptions can be made, then an arbitrary sample size is set for a pilot study. This article discusses sample size and how it relates to matters such as ethics, statistical power, the primary and secondary hypotheses in a study, and findings from larger vs. smaller samples.

Studies are conducted on samples because it is usually impossible to study the entire population. Conclusions drawn from samples are intended to be generalized to the population, and sometimes to the future as well. The sample must therefore be representative of the population. This is best ensured by the use of proper methods of sampling. The sample must also be adequate in size – in fact, no more and no less.

SAMPLE SIZE AND ETHICS

A sample that is larger than necessary will be better representative of the population and will hence provide more accurate results. However, beyond a certain point, the increase in accuracy will be small and hence not worth the effort and expense involved in recruiting the extra patients. Furthermore, an overly large sample would inconvenience more patients than might be necessary for the study objectives; this is unethical. In contrast, a sample that is smaller than necessary would have insufficient statistical power to answer the primary research question, and a statistically nonsignificant result could merely be because of inadequate sample size (Type 2 or false negative error). Thus, a small sample could result in the patients in the study being inconvenienced with no benefit to future patients or to science. This is also unethical.

In this regard, inconvenience to patients refers to the time that they spend in clinical assessments and to the psychological and physical discomfort that they experience in assessments such as interviews, blood sampling, and other procedures.

ESTIMATING SAMPLE SIZE

So how large should a sample be? In hypothesis testing studies, this is mathematically calculated, conventionally, as the sample size necessary to be 80% certain of identifying a statistically significant outcome should the hypothesis be true for the population, with P for statistical significance set at 0.05. Some investigators power their studies for 90% instead of 80%, and some set the threshold for significance at 0.01 rather than 0.05. Both choices are uncommon because the necessary sample size becomes large, and the study becomes more expensive and more difficult to conduct. Many investigators increase the sample size by 10%, or by whatever proportion they can justify, to compensate for expected dropout, incomplete records, biological specimens that do not meet laboratory requirements for testing, and other study-related problems.

Sample size calculations require assumptions about expected means and standard deviations, or event risks, in different groups; or, upon expected effect sizes. For example, a study may be powered to detect an effect size of 0.5; or a response rate of 60% with drug vs. 40% with placebo.[ 1 ] When no guesstimates or expectations are possible, pilot studies are conducted on a sample that is arbitrary in size but what might be considered reasonable for the field.

The sample size may need to be larger in multicenter studies because of statistical noise (due to variations in patient characteristics, nonspecific treatment characteristics, rating practices, environments, etc. between study centers).[ 2 ] Sample size calculations can be performed manually or using statistical software; online calculators that provide free service can easily be identified by search engines. G*Power is an example of a free, downloadable program for sample size estimation. The manual and tutorial for G*Power can also be downloaded.

PRIMARY AND SECONDARY ANALYSES

The sample size is calculated for the primary hypothesis of the study. What is the difference between the primary hypothesis, primary outcome and primary outcome measure? As an example, the primary outcome may be a reduction in the severity of depression, the primary outcome measure may be the Montgomery-Asberg Depression Rating Scale (MADRS) and the primary hypothesis may be that reduction in MADRS scores is greater with the drug than with placebo. The primary hypothesis is tested in the primary analysis.

Studies almost always have many hypotheses; for example, that the study drug will outperform placebo on measures of depression, suicidality, anxiety, disability and quality of life. The sample size necessary for adequate statistical power to test each of these hypotheses will be different. Because a study can have only one sample size, it can be powered for only one outcome, the primary outcome. Therefore, the study would be either overpowered or underpowered for the other outcomes. These outcomes are therefore called secondary outcomes, and are associated with secondary hypotheses, and are tested in secondary analyses. Secondary analyses are generally considered exploratory because when many hypotheses in a study are each tested at a P < 0.05 level for significance, some may emerge statistically significant by chance (Type 1 or false positive errors).[ 3 ]

INTERPRETING RESULTS

Here is an interesting question. A test of the primary hypothesis yielded a P value of 0.07. Might we conclude that our sample was underpowered for the study and that, had our sample been larger, we would have identified a significant result? No! The reason is that larger samples will more accurately represent the population value, whereas smaller samples could be off the mark in either direction – towards or away from the population value. In this context, readers should also note that no matter how small the P value for an estimate is, the population value of that estimate remains the same.[ 4 ]

On a parting note, it is unlikely that population values will be null. That is, for example, that the response rate to the drug will be exactly the same as that to placebo, or that the correlation between height and age at onset of schizophrenia will be zero. If the sample size is large enough, even such small differences between groups, or trivial correlations, would be detected as being statistically significant. This does not mean that the findings are clinically significant.

Financial support and sponsorship

Conflicts of interest.

There are no conflicts of interest.

User Preferences

Content preview.

Arcu felis bibendum ut tristique et egestas quis:

• Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris
• Duis aute irure dolor in reprehenderit in voluptate
• Excepteur sint occaecat cupidatat non proident

Keyboard Shortcuts

25.3 - calculating sample size.

Before we learn how to calculate the sample size that is necessary to achieve a hypothesis test with a certain power, it might behoove us to understand the effect that sample size has on power. Let's investigate by returning to our IQ example.

Example 25-3 Section  

Let \(X\) denote the IQ of a randomly selected adult American. Assume, a bit unrealistically again, that \(X\) is normally distributed with unknown mean \(\mu\) and (a strangely known) standard deviation of 16. This time, instead of taking a random sample of \(n=16\) students, let's increase the sample size to \(n=64\). And, while setting the probability of committing a Type I error to \(\alpha=0.05\), test the null hypothesis \(H_0:\mu=100\) against the alternative hypothesis that \(H_A:\mu>100\).

What is the power of the hypothesis test when \(\mu=108\), \(\mu=112\), and \(\mu=116\)?

Setting \(\alpha\), the probability of committing a Type I error, to 0.05, implies that we should reject the null hypothesis when the test statistic \(Z\ge 1.645\), or equivalently, when the observed sample mean is 103.29 or greater:

\( \bar{x} = \mu + z \left(\dfrac{\sigma}{\sqrt{n}} \right) = 100 +1.645\left(\dfrac{16}{\sqrt{64}} \right) = 103.29\)

Therefore, the power function \K(\mu)\), when \(\mu>100\) is the true value, is:

\( K(\mu) = P(\bar{X} \ge 103.29 | \mu) = P \left(Z \ge \dfrac{103.29 - \mu}{16 / \sqrt{64}} \right) = 1 - \Phi \left(\dfrac{103.29 - \mu}{2} \right)\)

Therefore, the probability of rejecting the null hypothesis at the \(\alpha=0.05\) level when \(\mu=108\) is 0.9907, as calculated here:

\(K(108) = 1 - \Phi \left( \dfrac{103.29-108}{2} \right) = 1- \Phi(-2.355) = 0.9907 \)

And, the probability of rejecting the null hypothesis at the \(\alpha=0.05\) level when \(\mu=112\) is greater than 0.9999, as calculated here:

\( K(112) = 1 - \Phi \left( \dfrac{103.29-112}{2} \right) = 1- \Phi(-4.355) = 0.9999\ldots \)

And, the probability of rejecting the null hypothesis at the \(\alpha=0.05\) level when \(\mu=116\) is greater than 0.999999, as calculated here:

\( K(116) = 1 - \Phi \left( \dfrac{103.29-116}{2} \right) = 1- \Phi(-6.355) = 0.999999... \)

In summary, in the various examples throughout this lesson, we have calculated the power of testing \(H_0:\mu=100\) against \(H_A:\mu>100\) for two sample sizes ( \(n=16\) and \(n=64\)) and for three possible values of the mean ( \(\mu=108\), \(\mu=112\), and \(\mu=116\)). Here's a summary of our power calculations:

As you can see, our work suggests that for a given value of the mean \(\mu\) under the alternative hypothesis, the larger the sample size \(n\), the greater the power \(K(\mu)\) . Perhaps there is no better way to see this than graphically by plotting the two power functions simultaneously, one when \(n=16\) and the other when \(n=64\):

As this plot suggests, if we are interested in increasing our chance of rejecting the null hypothesis when the alternative hypothesis is true, we can do so by increasing our sample size \(n\). This benefit is perhaps even greatest for values of the mean that are close to the value of the mean assumed under the null hypothesis. Let's take a look at two examples that illustrate the kind of sample size calculation we can make to ensure our hypothesis test has sufficient power.

Example 25-4 Section  

Let \(X\) denote the crop yield of corn measured in the number of bushels per acre. Assume (unrealistically) that \(X\) is normally distributed with unknown mean \(\mu\) and standard deviation \(\sigma=6\). An agricultural researcher is working to increase the current average yield from 40 bushels per acre. Therefore, he is interested in testing, at the \(\alpha=0.05\) level, the null hypothesis \(H_0:\mu=40\) against the alternative hypothesis that \(H_A:\mu>40\). Find the sample size \(n\) that is necessary to achieve 0.90 power at the alternative \(\mu=45\).

As is always the case, we need to start by finding a threshold value \(c\), such that if the sample mean is larger than \(c\), we'll reject the null hypothesis:

That is, in order for our hypothesis test to be conducted at the \(\alpha=0.05\) level, the following statement must hold (using our typical \(Z\) transformation):

\(c = 40 + 1.645 \left( \dfrac{6}{\sqrt{n}} \right) \) (**)

But, that's not the only condition that \(c\) must meet, because \(c\) also needs to be defined to ensure that our power is 0.90 or, alternatively, that the probability of a Type II error is 0.10. That would happen if there was a 10% chance that our test statistic fell short of \(c\) when \(\mu=45\), as the following drawing illustrates in blue:

This illustration suggests that in order for our hypothesis test to have 0.90 power, the following statement must hold (using our usual \(Z\) transformation):

\(c = 45 - 1.28 \left( \dfrac{6}{\sqrt{n}} \right) \) (**)

Aha! We have two (asterisked (**)) equations and two unknowns! All we need to do is equate the equations, and solve for \(n\). Doing so, we get:

\(40+1.645\left(\frac{6}{\sqrt{n}}\right)=45-1.28\left(\frac{6}{\sqrt{n}}\right)\) \(\Rightarrow 5=(1.645+1.28)\left(\frac{6}{\sqrt{n}}\right), \qquad \Rightarrow 5=\frac{17.55}{\sqrt{n}}, \qquad n=(3.51)^2=12.3201\approx 13\)

Now that we know we will set \(n=13\), we can solve for our threshold value c :

\( c = 40 + 1.645 \left( \dfrac{6}{\sqrt{13}} \right)=42.737 \)

So, in summary, if the agricultural researcher collects data on \(n=13\) corn plots, and rejects his null hypothesis \(H_0:\mu=40\) if the average crop yield of the 13 plots is greater than 42.737 bushels per acre, he will have a 5% chance of committing a Type I error and a 10% chance of committing a Type II error if the population mean \(\mu\) were actually 45 bushels per acre.

Example 25-5 Section  

Consider \(p\), the true proportion of voters who favor a particular political candidate. A pollster is interested in testing at the \(\alpha=0.01\) level, the null hypothesis \(H_0:9=0.5\) against the alternative hypothesis that \(H_A:p>0.5\). Find the sample size \(n\) that is necessary to achieve 0.80 power at the alternative \(p=0.55\).

In this case, because we are interested in performing a hypothesis test about a population proportion \(p\), we use the \(Z\)-statistic:

\(Z = \dfrac{\hat{p}-p_0}{\sqrt{\frac{p_0(1-p_0)}{n}}} \)

Again, we start by finding a threshold value \(c\), such that if the observed sample proportion is larger than \(c\), we'll reject the null hypothesis:

That is, in order for our hypothesis test to be conducted at the \(\alpha=0.01\) level, the following statement must hold:

\(c = 0.5 + 2.326 \sqrt{ \dfrac{(0.5)(0.5)}{n}} \) (**)

But, again, that's not the only condition that c must meet, because \(c\) also needs to be defined to ensure that our power is 0.80 or, alternatively, that the probability of a Type II error is 0.20. That would happen if there was a 20% chance that our test statistic fell short of \(c\) when \(p=0.55\), as the following drawing illustrates in blue:

This illustration suggests that in order for our hypothesis test to have 0.80 power, the following statement must hold:

\(c = 0.55 - 0.842 \sqrt{ \dfrac{(0.55)(0.45)}{n}} \) (**)

Again, we have two (asterisked (**)) equations and two unknowns! All we need to do is equate the equations, and solve for \(n\). Doing so, we get:

\(0.5+2.326\sqrt{\dfrac{0.5(0.5)}{n}}=0.55-0.842\sqrt{\dfrac{0.55(0.45)}{n}} \\ 2.326\dfrac{\sqrt{0.25}}{\sqrt{n}}+0.842\dfrac{\sqrt{0.2475}}{\sqrt{n}}=0.55-0.5 \\ \dfrac{1}{\sqrt{n}}(1.5818897)=0.05 \qquad \Rightarrow n\approx \left(\dfrac{1.5818897}{0.05}\right)^2 = 1000.95 \approx 1001 \)

Now that we know we will set \(n=1001\), we can solve for our threshold value \(c\):

\(c = 0.5 + 2.326 \sqrt{\dfrac{(0.5)(0.5)}{1001}}= 0.5367 \)

So, in summary, if the pollster collects data on \(n=1001\) voters, and rejects his null hypothesis \(H_0:p=0.5\) if the proportion of sampled voters who favor the political candidate is greater than 0.5367, he will have a 1% chance of committing a Type I error and a 20% chance of committing a Type II error if the population proportion \(p\) were actually 0.55.

Incidentally, we can always check our work! Conducting the survey and subsequent hypothesis test as described above, the probability of committing a Type I error is:

\(\alpha= P(\hat{p} >0.5367 \text { if } p = 0.50) = P(Z > 2.3257) = 0.01 \)

and the probability of committing a Type II error is:

\(\beta = P(\hat{p} <0.5367 \text { if } p = 0.55) = P(Z < -0.846) = 0.199 \)

just as the pollster had desired.

We've illustrated several sample size calculations. Now, let's summarize the information that goes into a sample size calculation. In order to determine a sample size for a given hypothesis test, you need to specify:

The desired \(\alpha\) level, that is, your willingness to commit a Type I error.

The desired power or, equivalently, the desired \(\beta\) level, that is, your willingness to commit a Type II error.

A meaningful difference from the value of the parameter that is specified in the null hypothesis.

The standard deviation of the sample statistic or, at least, an estimate of the standard deviation (the "standard error") of the sample statistic.

Explore the Scientific R&D Platform

Scientific intelligence platform for AI-powered data management and workflow automation

Statistical analysis and graphing software for scientists

Bioinformatics, cloning, and antibody discovery software

Plan, visualize, & document core molecular biology procedures

Proteomics software for analysis of mass spec data

Electronic Lab Notebook to organize, search and share data

Modern cytometry analysis platform

Analysis, statistics, graphing and reporting of flow cytometry data

Intelligent panel design & inventory management for flow cytometry

Software to optimize designs of clinical trials

Computational fluid dynamics (CFD) software for engineers and researchers

Genetic analysis software for research, forensics & healthcare applications

Everything to Know About Sample Size Determination

A step-by-step interactive guide including common pitfalls

Nothing showing? Click here to accept marketing cookies

Download and explore the data yourself. Data files include:

• Blinded Sample Size Re-estimation.nqt   
• Blinded SiteAndSubject.nqt
• Center Covariate Reducing Sample Size.nqt
• Cluster Randomized Extension.nqt
• External Pilot Study Sample Size Example.nqt
• Log-Rank Test Everolimus.nqt
• Log-Rank Test with Dropout.nqt
• MaxCombo Model Selection with Delayed Effect.nqt
• Responder Analysis Higher Sample Size Chi-Squared.nqt
• Two Means Group Sequential Replication.nqt
• Two Proportions Inequality Difference Scale.nqt
• Two Proportions Inequality Ratio Scale.nqt
• Two Proportions Non-inferiority Difference Scale.nqt
• Two Proportions Non-inferiority Ratio Scale.nqt
• Two Sample t-test Simvastin.nqt
• Win Ratio for Composite Endpoint.nqt

Designing a trial involves considering and balancing a wide variety of clinical, logistical and statistical factors.

One decision out of many is how large a study needs to be to have a reasonable chance of success. Sample size determination is the process by which trialists can find the ideal number of participants to balance the statistical and practical aspects that inform study design. 

In this interactive webinar, we provided a comprehensive overview of sample size determination, the key steps to successfully finding the appropriate sample size and cover several common pitfalls researchers fall into when finding the sample size for their study.

In this free webinar we will cover

• What is sample size determination?
• A step-by-step guide to sample size determination
• Common sample size pitfalls and solutions

+ Q&A about your sample size issues!

In most clinical trials, sample size determination is found by reaching a predefined statistical power - typically defined as the Type II error or how likely a significant p-value is under a given treatment effect.

Power calculations require pre-study knowledge about the study design, statistical error rates, nuisance parameters (such as the variance) and effect size with each of these adding additional complexity. 

Sample size determination has a number of common pitfalls which can lead to inappropriately small or large sample sizes with issues ranging from poor design decisions, misspecifying nuisance parameters or choosing the effect size inappropriately.

In this interactive webinar, we explore these and some solutions to avoid these mistakes and help maximise the efficiency of your clinical trial.  

We provide a comprehensive overview of sample size determination, the key steps to successfully finding the appropriate sample size and cover several common pitfalls researchers fall into when finding the sample size for their study.

Looking for more trial design and sample size resources? Check out webinars to improve clinical trial designs & practical examples of sample size determination

Browse our Webinars

Guide to Sample Size

Designing Robust Group Sequential Trials | Free nQuery Training

Group Sequential Design Theory and Practice

Get started with nQuery today

Try for free and upgrade as your team grows

Stack Exchange Network

Stack Exchange network consists of 183 Q&A communities including Stack Overflow , the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

Test difference between samples with very small sample size

Suppose I have observed 3 realizations of two non-negative, integer random variables $X$ and $Y$. Nothing is known about their underlying distribution. The results were $x = \{4,~8,~2\}$, and $y = \{22,~11,~8\}$. Hence, $\overline{x} = 14/3$ and $\overline{y} = 41/3$, for a sample mean difference of $9$.

My question is: is there any way to conduct a meaningful statistical hypothesis test in order to decide whether $H_0: E(X) = E(Y)$, can be rejected?

My first idea was to use a two-sample t-test or Welch-test, but I guess the sample size is way too low for that. Is there anything one can reasonably test?

• hypothesis-testing
• small-sample

• 4 $\begingroup$ You may find the following thread informative: is-there-a-minimum-sample-size-required-for-the-t-test-to-be-valid . $\endgroup$ –  gung - Reinstate Monica Commented Jun 25, 2013 at 14:52
• $\begingroup$ What makes you say the sample size is too low? The big problem is that you can't check the assumptions, but a two sample t-test can actually be performed with two observations in each sample - or indeed, if you're prepared to assume equality of variance, with even as little as 3 observations total! $\endgroup$ –  Glen_b Commented Jun 26, 2013 at 2:28

2 Answers 2

There are potentially a number of ways of testing if these two samples differ, but all will probably have low power. You could use a t-test, but its validity will depend on whether the underlying populations are normally distributed and have equal variances. With so few data, you really can't check that very well so you have to rely entirely on prior knowledge (of which you say you have none) and the assumptions you are willing to make. Given that your variances are $9.3$ and $54.3$, I would not want to make the assumption of equal variances (although, again, with so few data they actually could be), so the Satterthwaite-Welch correction seems appropriate. If you weren't willing to assume the populations were exactly normal (since the central limit theorem cannot cover you with samples this small), you could use the Mann-Whitney U test . As it happens, that test gives a lower p-value ($.12$) than the corrected t-test ($.16$). The question then, is what you want to conclude from these results. My opinion is that using a rigid $.05$ cutoff is typically not appropriate (see my answer here: When to use Fisher and Neyman-Pearson framework? ); so I would say this result is somewhat ambiguous, but you might find it does provide some evidence against the null, depending on how plausible the null is a-priori.

• 1 $\begingroup$ how did you compute your Mann-Whitney p-value? did you use the normal approximation? adjust for ties? etc.? The exact permutation test gives a p-value of 0.10 and since the MW test is a permutation test it should give a p-value of 0.10 as well (but I could see a small difference coming from an approximation rather than an exact test). $\endgroup$ –  Greg Snow Commented Jun 25, 2013 at 19:55
• $\begingroup$ @GregSnow, I just typed the data into R & ran wilcox.test(x, y) . $\endgroup$ –  gung - Reinstate Monica Commented Jun 25, 2013 at 19:58
• $\begingroup$ @Greg The M-W is not a permutation test of the mean, so it should be no surprise that it gives a (slightly) different p-value. With some of the permutations, such as $((2,4,22), (8,8,11))$, the order of the means ($28/3, 27/3$) differs from the order of the medians ($4, 8$) or the rank sums ($9, 12$). How you adjust for ties can make a substantial difference, too, depending on what test statistic is used in the permutation test. Gung: talking about "power" may be a little dicey when no alternative hypothesis is clearly formulated. $\endgroup$ –  whuber ♦ Commented Jun 25, 2013 at 20:16
• $\begingroup$ @whuber, There are only 20 possible combinations for splitting 6 observations into 2 groups of 3 each, so any p-value from an exact permutation test will be a multiple of 0.05. Since the data has 2 8's, 1 in each group that means that any permutation that switches the 2 8's will be equivalent giving a p-value of 0.10 (though it should be doubled to 0.2 for a 2-sided test) for any statistic that measures location, whether it be mean, median, minimum, sum of ranks, 42nd percentile, etc. A permutation test based on ratio of variances could give a 0.05 for a 1-sided test. $\endgroup$ –  Greg Snow Commented Jun 25, 2013 at 20:45
• 1 $\begingroup$ @gung I think the big problem with the Mann-Whitney on this tiny a sample size (3,3) is that - no matter how vastly separated the samples are, and even without any ties - you simply can't get a p-value below 0.1 (two tailed). At least a suitable parametric test (if its assumptions could be argued to be reasonable) has the possibility of reaching an interesting p-value. One needs to bring some assumptions (not necessarily normality) to this small a sample, and try to justify them. $\endgroup$ –  Glen_b Commented Jun 26, 2013 at 2:33

You say that nothing is known about the distribution, but you also say that the values are non-negative integers, so that tells you something, can you learn more about the theory?

An exact permutation test on your data gives a p-value of 0.10 and does not require assumptions about the distribution (just testing that it is identical for the 2 groups).

If your data represent counts (non-negative integers) then a Poisson model may be appropriate (which gives a p-value less than 0.001), but make sure that the assumptions are reasonable.

As @gung mentions, there are many ways to look at this and which is best is going to depend on the source of the data and the science that underlies it. learning about the data, talking to the person that gave you the data, talking to other experts, etc. is going to be of the most benefit.

If you really cannot learn anything more about the underlying distribution or approximations to it then you may have to resort to SnowsCorrectlySizedButOtherwiseUselessTestOfAnything (a function in the TeachingDemos package for R) which will give a p-value without requiring any assumptions about your data. But note that that function is considered less useful than its documentation.

Your Answer

Sign up or log in, post as a guest.

Required, but never shown

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy .

Not the answer you're looking for? Browse other questions tagged hypothesis-testing t-test small-sample or ask your own question .

• Featured on Meta
• Site maintenance - Mon, Sept 16 2024, 21:00 UTC to Tue, Sept 17 2024, 2:00...
• User activation: Learnings and opportunities
• Join Stack Overflow’s CEO and me for the first Stack IRL Community Event in...

Hot Network Questions

• What is the oldest open math problem outside of number theory?
• Tensor product of intersections in an abelian rigid monoidal category
• Rocky Mountains Elevation Cutout
• Offset+Length vs 2 Offsets
• Definition of annuity
• What was it that Wittgenstein found funny here?
• The consequence of a good letter of recommendation when things do not work out
• 1950s comic book about bowling ball looking creatures that inhabit the underground of Earth
• Is it true that before European modernity, there were no "nations"?
• How did people know that the war against the mimics was over?
• Trying to find air crash for a case study
• Drill perpendicular hole through thick lumber using handheld drill
• Swapping front Shimano 105 R7000 34x50t 11sp Chainset with Shimano Deore FC-M5100 chainset; 11-speed 26x36t
• Looking for a short story on chess, maybe published in Playboy decades ago?
• Longtable goes beyond the right margin and footnote does not fit under the table
• Why was Esther included in the canon?
• On the history of algae classification
• how does the US justice system combat rights violations that happen when bad practices are given a new name to avoid old rulings?
• Is it a correct rendering of Acts 1,24 when the New World Translation puts in „Jehovah“ instead of Lord?
• Need help for translating old signs
• In Photoshop, when saving as PNG, why is the size of my output file bigger when I have more invisible layers in the original file?
• Copyright Fair Use: Is using the phrase "Courtesy of" legally acceptable when no permission has been given?
• How many engineers/scientists believed that human flight was imminent as of the late 19th/early 20th century?
• Why was Panama Railroad in poor condition when US decided to build Panama Canal in 1904?