How Many of My Students Should I Survey?
“How large a sample do I need?” is the question we are most frequently asked by client institutions following their decision to conduct a survey. Clients are often very surprised with our answer, the most common of which is, “we recommend a census, not a sample.” That is, except perhaps in some cases of very large institutions, say of 40-50,000 students, we recommend that all students be surveyed. Why, because of the efficiency of online surveying.
It typically requires less labor and cost to survey everyone than to invest in an elaborate sampling plan. The marginal increase in labor and costs declines rapidly after the first few thousand respondents, while the cost of devising and enforcing a statistical sampling plan can be prohibitive.
Still, in some cases, a sample may make more sense than a census – e.g., a large institution needs a pilot study conducted quickly to evaluate how many students will support some proposed emergency measure. In these cases, it is necessary to consider sample size and, since both cost and precision increase with larger sample sizes, the choice of sample size is a tradeoff between the informational value of additional data judged against the cost of gathering them. The precision of the sample information is best illustrated by what most people know as the “margin of error,” which is expressed as a plus and minus error range. So, if a survey shows that 60 percent, +/- 5 percent, of the student body will support a university policy, it means that most likely somewhere between 55 and 65 percent of the students support the policy. Specifically, it means that there is a 95 percent probability that the actual or “true” percentage of students who support the policy is between 55 and 65 percent, and therefore, necessarily, a 5 percent probability that it falls out side this range. So you see, the uncertainty in never resolved completely . . . except with a census.
Now suppose this estimate was not precise enough for your decision-making needs. For example, maybe you had decided that you would implement the policy if more than 56 percent of the students support it. With the current result it is uncertain if this criteria is met. Suppose you wanted to know, or at least be very confident, what the “true” percentage is to within plus or minus 2.5 percent. A general rule of thumb is that to decrease the margin of error by half you must quadruple the sample size. So, in our example above, assume our estimate of 60 percent, +/- 5 percent, was calculated on a sample of 1,000. If we want our margin of error to be reduced by one-half — viz., to 2.5 percent – we will need a sample of 4,000. You can see that the marginal increase in precision with increasing sample sizes diminishes rapidly. Now if you wanted to reduce again, the margin or error by one-half – i.e., from 2.5 percent to 1.25 percent, you would need a sample of 16,000 – i.e., 4,000 x 4.
Now let’s consider costs. Assume for the moment that the survey costs $1 per student. By increasing the sample size from 1,000 to 4,000 your total survey cost increases by $3,000 and your margin of error decreases by 2.5 percentage points. This represents an average cost of $1,200 per percent of precision gain – i.e., $3,000/2.5 = $1,200. Similarly, by increasing the sample size from 4,000 to 16,000 your total survey cost increases by an additional $12,000 and your margin of error decreases by 1.25 percentage points. This represents an average cost of $9,600 per percent of precision gain – i.e., $12,000/1.25 = $9,600. So you see, achieving increasing degrees of precision requires a disproportionate increase in the sample size and therefore becomes increasingly costly . . . unless of course the per-student cost of surveying is negligible, which it is in many online surveys, and this is essentially why we advocate conducting a census instead of sampling.
But, if you must sample, how should your sample be selected? It is a widely held belief that, to be valid, a sample must be random. That is, every member of the student population must have an equal chance of being selected into the sample. This belief is not correct, because, in many situations, nonrandom samples can give more reliable results than random ones. It depends on how much information you have about the population beforehand. In the case of your student population, this is typically a lot.
The goal in sampling is to have the sample be representative of our population. When we do not have enough information about the population to choose a sample that we can know, a priori, is representative, we resort to random sampling, hoping that the laws of chance will produce a sample that more or less mirrors our population. One caveat is in order here, and that is that some inferential statistical procedures require that the computed estimates come from a random sample in order for the conclusions to be valid. Typically, however, the managerial decision making of student service organizations, which use the survey results of the type being described here, are gathered for far more practical purposes.
In practice, at some stage of a sampling protocol, we often resort to random selection. For example, if we know we have 70 percent undergraduate students, half of whom are male and half of whom are female, and 30 percent graduate students, 40 percent of whom are male and 60 percent of whom are female, we would sample these subgroups in the same proportion that they are known to exist in the population, but the selections from each subgroup would be random. This concept is easily generalized to many more (and therefore smaller) subgroups, so that selecting, even in a non random manner, a few students from each of the multiple subgroups would, with near certainty, produce a representative sample. The point is: it is more important to have a representative sample than to have a random one, and it is ideal to have both.
Ah, but even with a census there is a problem, or at least there could be, that might threaten the validity of the results: you may give the entire student body the opportunity to respond to a survey, but not all will. Invariably, some students will not respond to the survey, no matter how much you may coax them. So the data, it could be argued, constitute a sample, albeit one without a premeditated sampling plan. After all it is a subset, not the entirety, of the population. There is an unwarranted tendency to be less concerned about the reliability and the margin of error with the results of a census than with sample results. While this is generally inconsequential unless the non-response rate is high, say 20 to 30 percent, it requires attention. Fortunately, in the case of student surveys it is not a problem for the following reason: the demographics of the entire student population will be well known from university data sources, and demographics are routinely included in the survey questionnaire. Thus, after the data have been collected, it is possible to know how close the demographics of the sample are to the population. The sample results can then be weighted to achieve, in the sample, a proportionate representation that corresponds to that known to exist in our population. For example, suppose it is known that in the student population half the students are male and half female, but that it is observed that in the sample 25 percent of the respondents are male and 75 percent female. Unweighted, the responses of the males are only half of what they should be (25% vs. 50%) so males’ responses should be weighted by a factor of 2.0. On the other hand, unweighted, the responses of the females are half again what they should be (75% vs. 50%) so they should be weighted by a factor of 2/3. With the weightings applied to the responses, the results will more closely match what they would have been expected to be, had all students – i.e., 50% males and 50% females –responded to the survey. Weighting is advised when it is either known or suspected that survey responses systematically with the demographic subgroups. If they do not, then there is no need to apply weightings.
A final misconception about sample size requirements needs to be dispelled, and that is the idea that when sampling from a larger population you need to take a larger sample. This is not true. A sample of 1,000 drawn from your student body of 10,000 will have more or less the same precision as a sample of 1,000 drawn from a student body of 50,000, or for that matter a sample of 1,000 drawn from a student population of 1,000,000. This fact is admittedly counterintuitive, but true nevertheless. It is beyond the scope of this “Blue Paper” to show why this is true, but if you are interested you can email us and we will reply with the reason.
In summary, sampling really is not a critical issue in surveying student populations, since we typically have efficient on-line access to the entire student population and can therefore efficiently conduct a census. It is also because we are in the position of having substantial information about the demographics of the student population, which allows for post data collection weighting of the results in the case of disproportionate response rates among different demographic subgroups.
But does a good sample, or a near 100 percent response rate to a census, guarantee a good survey result? No, not necessarily! In our subsequent “Blue Papers,” we will address other requirements for ensuring good survey results, as well as many strategies and methods for making practical use of survey results to foster data-driven managerial decision making.
We welcome discussion that flows from our BluePapers, and hope it will help student services managers better understand their own needs as they research the needs of their students. Please note that the discussions will be moderated to prevent spam and flame wars, but we welcome and will publish all sides of the conversations.