Sampling and Experimentation
As we mentioned previously, the first thing we should do before conducting a survey is to identify the population that we want to study. In this lesson, we will show you examples of how to identify the population in a study, and determine whether or not the study actually represents the intended population. We will discuss different techniques for random sampling that are intended to ensure a population is well represented in a sample. We will also identify the difference between an observational study and an experiment, and ways experiments can be conducted. By the end of this lesson, we hope that you will also be confident in identifying when an experiment may have been affected by confounding or the placebo effect, and the methods that are employed to avoid them.Learning Objectives
- Identify methods for obtaining a random sample of the intended population of a study
- Identify ineffective ways of obtaining a random sample from a population
- Identify types of sample bias
- Identify the differences between observational study and an experiment
- Identify the treatment in an experiment
- Determine whether an experiment may have been influenced by confounding
Sampling Methods and Bias
Selecting a Population
Suppose we are hired by a politician to determine the amount of support he has among the electorate should he decide to run for another term. What population should we study? Every person in the district? Not every person is eligible to vote, and regardless of how strongly someone likes or dislikes the candidate, they don't have much to do with him being re-elected if they are not able to vote. What about eligible voters in the district? That might be better, but if someone is eligible to vote but does not register by the deadline, they won't have any say in the election either. What about registered voters? Many people are registered but choose not to vote. What about "likely voters?" This is the criteria used in much political polling, but it is sometimes difficult to define a "likely voter." Is it someone who voted in the last election? In the last general election? In the last presidential election? Should we consider someone who just turned 18 a "likely voter?" They weren't eligible to vote in the past, so how do we judge the likelihood that they will vote in the next election? In November 1998, former professional wrestler Jesse "The Body" Ventura was elected governor of Minnesota. Up until right before the election, most polls showed he had little chance of winning. There were several contributing factors to the polls not reflecting the actual intent of the electorate:- Ventura was running on a third-party ticket and most polling methods are better suited to a two-candidate race.
- Many respondents to polls may have been embarrassed to tell pollsters that they were planning to vote for a professional wrestler.
- The mere fact that the polls showed Ventura had little chance of winning might have prompted some people to vote for him in protest to send a message to the major-party candidates.
Sampling bias
A sampling method is biased if every member of the population doesn’t have equal likelihood of being in the sample.
Simple random sample
A random sample is one in which each member of the population has an equal probability of being chosen. A simple random sample is one in which every member of the population and any group of members has an equal probability of being chosen.
example
If we could somehow identify all likely voters in the state, put each of their names on a piece of paper, toss the slips into a (very large) hat and draw 1000 slips out of the hat, we would have a simple random sample.
In practice, computers are better suited for this sort of endeavor than millions of slips of paper and extremely large headgear.
Sampling variability
The natural variation of samples is called sampling variability.
This is unavoidable and expected in random sampling, and in most cases is not an issue.
Stratified sampling
In stratified sampling, a population is divided into a number of subgroups (or strata). Random samples are then taken from each subgroup with sample sizes proportional to the size of the subgroup in the population.
example
Suppose in a particular state that previous data indicated that the electorate was comprised of 39% Democrats, 37% Republicans and 24% independents. In a sample of 1000 people, they would then expect to get about 390 Democrats, 370 Republicans and 240 independents. To accomplish this, they could randomly select 390 people from among those voters known to be Democrats, 370 from those known to be Republicans, and 240 from those with no party affiliation.
Quota sampling
Quota sampling is a variation on stratified sampling, wherein samples are collected in each subgroup until the desired quota is met.
example
Suppose the pollsters call people at random, but once they have met their quota of 390 Democrats, they only gather people who do not identify themselves as a Democrat.
You may have had the experience of being called by a telephone pollster who started by asking you your age, income, etc. and then thanked you for your time and hung up before asking any "real" questions. Most likely, they already had contacted enough people in your demographic group and were looking for people who were older or younger, richer or poorer, etc. Quota sampling is usually a bit easier than stratified sampling, but also does not ensure the same level of randomness.
Another sampling method is cluster sampling, in which the population is divided into groups, and one or more groups are randomly selected to be in the sample.
Cluster sampling
In cluster sampling, the population is divided into subgroups (clusters), and a set of subgroups are selected to be in the sample.
example
If the college wanted to survey students, since students are already divided into classes, they could randomly select 10 classes and give the survey to all the students in those classes. This would be cluster sampling.Systematic sampling
In systematic sampling, every nth member of the population is selected to be in the sample.
example
To select a sample using systematic sampling, a pollster calls every 100th name in the phone book. Systematic sampling is not as random as a simple random sample (if your name is Albert Aardvark and your sister Alexis Aardvark is right after you in the phone book, there is no way you could both end up in the sample) but it can yield acceptable samples.The Worst Way to Sample
Perhaps the worst types of sampling methods are convenience samples and voluntary response samples.Convenience sampling and voluntary response sampling
Convenience sampling is the practice of samples chosen by selecting whoever is convenient.
Voluntary response sampling is allowing the sample to volunteer.
example
A pollster stands on a street corner and interviews the first 100 people who agree to speak to him. Which sampling method is represented by this scenario?Answer: This is a convenience sample.
A website has a survey asking readers to give their opinion on a tax proposal. Which sampling method is represented?
Answer:
This is a self-selected sample, or voluntary response sample, in which respondents volunteer to participate.
Usually voluntary response samples are skewed towards people who have a particularly strong opinion about the subject of the survey or who just have way too much time on their hands and enjoy taking surveys.
Try It Now
In each case, indicate what sampling method was used a. Every 4th person in the class was selected b. A sample was selected to contain 25 men and 35 women c. Viewers of a new show are asked to vote on the show’s website d. A website randomly selects 50 of their customers to send a satisfaction survey to e. To survey voters in a town, a polling company randomly selects 10 city blocks, and interviews everyone who lives on those blocks.Experiments
Observing vs. Acting
So far, we have primarily discussed observational studies – studies in which conclusions would be drawn from observations of a sample or the population. In some cases these observations might be unsolicited, such as studying the percentage of cars that turn right at a red light even when there is a “no turn on red” sign. In other cases the observations are solicited, like in a survey or a poll. In contrast, it is common to use experiments when exploring how subjects react to an outside influence. In an experiment, some kind of treatment is applied to the subjects and the results are measured and recorded.Observational studies and experiments
- An observational study is a study based on observations or measurements
- An experiment is a study in which the effects of a treatment are measured
Examples
Here are some examples of experiments: A pharmaceutical company tests a new medicine for treating Alzheimer’s disease by administering the drug to 50 elderly patients with recent diagnoses. The treatment here is the new drug.A gym tests out a new weight loss program by enlisting 30 volunteers to try out the program. The treatment here is the new program.
You test a new kitchen cleaner by buying a bottle and cleaning your kitchen. The new cleaner is the treatment.
A psychology researcher explores the effect of music on temperament by measuring people’s temperament while listening to different types of music. The music is the treatment.
Try It Now
Is each scenario describing an observational study or an experiment? a. The weights of 30 randomly selected people are measured b. Subjects are asked to do 20 jumping jacks, and then their heart rates are measured c. Twenty coffee drinkers and twenty tea drinkers are given a concentration testAnswer: a. Observational study
b. Experiment; the treatment is the jumping jacks
c. Experiment; the treatments are coffee and tea
There are a number of measures that can be introduced to help reduce the likelihood of confounding. The primary measure is to use a control group.
Control group
When using a control group, the participants are divided into two or more groups, typically a control group and a treatment group. The treatment group receives the treatment being tested; the control group does not receive the treatment.examples
To determine if a two day prep course would help high school students improve their scores on the SAT test, a group of students was randomly divided into two subgroups. The first group, the treatment group, was given a two day prep course. The second group, the control group, was not given the prep course. Afterwards, both groups were given the SAT.A company testing a new plant food grows two crops of plants in adjacent fields, the treatment group receiving the new plant food and the control group not. The crop yield would then be compared. By growing them at the same time in adjacent fields, they are controlling for weather and other confounding factors.
Placebo effect
The placebo effect is when the effectiveness of a treatment is influenced by the patient’s perception of how effective they think the treatment will be, so a result might be seen even if the treatment is ineffectual.
example
A study found that when doing painful dental tooth extractions, patients told they were receiving a strong painkiller while actually receiving a saltwater injection found as much pain relief as patients receiving a dose of morphine.[footnote]Levine JD, Gordon NC, Smith R, Fields HL. (1981) Analgesic responses to morphine and placebo in individuals with postoperative pain. Pain. 10:379-89.[/footnote]Placebo and Placebo controlled experiments
- A placebo is a dummy treatment given to control for the placebo effect.
- An experiment that gives the control group a placebo is called a placebo controlled experiment.
examples
In a study for a new medicine that is dispensed in a pill form, a sugar pill could be used as a placebo.In a study on the effect of alcohol on memory, a non-alcoholic beer might be given to the control group as a placebo.
In a study of a frozen meal diet plan, the treatment group would receive the diet food, and the control could be given standard frozen meals stripped of their original packaging. The following video walks through the controlled experiment scenarios, including the ones using placebos. https://youtu.be/UkCHUeqMb5Y
Blind studies
- A blind study is one in which the participant does not know whether or not they are receiving the treatment or a placebo.
- A double-blind study is one in which those interacting with the participants don’t know who is in the treatment group and who is in the control group.
examples
In a study about anti-depression medicine, you would not want the psychological evaluator to know whether the patient is in the treatment or control group either, as it might influence their evaluation, so the experiment should be conducted as a double-blind study.
It should be noted that not every experiment needs a control group. If a researcher is testing whether a new fabric can withstand fire, she simply needs to torch multiple samples of the fabric – there is no need for a control group. These examples are demonstrated in the following video. https://youtu.be/7BFZVGCxeYc
Try It now
To test a new lie detector, two groups of subjects are given the new test. One group is asked to answer all the questions truthfully, and the second group is asked to lie on one set of questions. The person administering the lie detector test does not know what group each subject is in. Does this experiment have a control group? Is it blind, double-blind, or neither?Answer:
The truth-telling group could be considered the control group, but really both groups are treatment groups here, since it is important for the lie detector to be able to correctly identify lies, and also not identify truth telling as lying. This study is blind, since the person running the test does not know what group each subject is in.
Licenses & Attributions
CC licensed content, Original
- Introduction and Learning Objectives. Provided by: Lumen Learning License: CC BY: Attribution.
- Revision and Adaptation. Provided by: Lumen Learning License: CC BY: Attribution.
CC licensed content, Shared previously
- Math in Society. Authored by: David Lippman. Located at: http://www.opentextbookstore.com/mathinsociety/. License: CC BY-SA: Attribution-ShareAlike.
- crowd-lego-staff-choice-selector. Authored by: mwewering. License: CC0: No Rights Reserved.
- Sampling methods. Authored by: OCLPhase2's channel. License: CC BY: Attribution.
- Question ID 6733, 6727. Authored by: Lippman, David. License: CC BY: Attribution. License terms: IMathAS Community License CC-BY +GPL.
- lab-research-chemistry-test. Authored by: PublicDomainPictures. License: CC0: No Rights Reserved.
- Experiments. Authored by: OCLPhase2's channel. License: CC BY: Attribution.
- Confounding. Authored by: OCLPhase2's channel. License: CC BY: Attribution.
- Controlled Experiments. Authored by: OCLPhase2's channel. License: CC BY: Attribution.
- Blind Experiments. Provided by: https://youtu.be/7BFZVGCxeYc Authored by: OCLPhase2's channel. License: CC BY: Attribution.
- Question ID 6736, 6728, 6914. Authored by: Lippman, David. License: CC BY: Attribution. License terms: IMathAS Community License CC-BY + GPL.