The surveyor, in this case UDS, will also be very concerned, not only that privacy, confidentiality, and security are achieved, but also in convincing students, deans, and directors that it will be achieved, because a failure to do either will threaten the validity of the survey results. In short, it is in the best interest of all parties to ensure privacy of the respondents and the confidentiality and security of their responses. There is little that is ambiguous about this issue: confidentiality and privacy of survey responses are respondents’ right and UDS’s responsibility. One we take very seriously.

It is important, therefore, that we not only achieve respondents’ privacy and the confidentiality and security of their responses, but also that we effectively convince the students, deans, and directors that our claims of privacy, confidentiality, and security are credible. This can be difficult because, while much of the credibility rests with the competence and diligence of the surveyors, characteristics that students, deans, and directors all understand very well, some of the credibility rests with the design and deployment of the technology which many of them may not understand. So how can you be sure? Here, we try to lay out the basic issues surrounding privacy, confidentiality, and security in on-line surveying; how they are dealt with generally; and how we deal with them specifically at UDS. The intent is to allay these concerns. thereby allowing deans and directors, as well as students, to focus on the benefits that competent surveying can generate.

The protection of respondents’ confidentiality and privacy rights can not be achieved with technology alone, no more than it can be achieved with competent IT personnel guided by clear policy alone. It requires both. There are ten industry principles that guide our approach to privacy and confidentiality in on-line surveys. Here we provide a summary of these ten principles, a detailed explanation of which can be viewed in our privacy statement.

Principle 1: Accountability – There is no “passing-the-buck” on compliance. UDS is solely responsible for personal information under its control designates an individual or individuals who are accountable for the organization’s compliance with confidentiality and security principles and policies. Accountability for compliance rests with the designated individual(s) and the identity of these individual(s) shall be made known to any client organization or survey respondent upon request.

Principle 2: Identifying Purposes – Respondents have a right to know why survey data are being collected. Four precepts serve as the foundation for this principle: (i) openness as to purpose, (ii) individual access to one’s personal data, (iii) limited collection of data, (iv) limited use, disclosure, and retention of data. The purposes for which personal information is being collected is identified before its collection and persons collecting personal information must be able to explain to individuals those purposes. Further, only those data necessary to serve the stated purposes are collected, and the data may not be used for ad hoc purposes unrelated to the original stated purposes.

Principle 3: Consent: The knowledge and consent of individuals are required for the collection, use, or disclosure of personal information. UDS shall make a reasonable effort to ensure that the individual is advised of the purposes for which the information will be used and to communicate the purpose in a manner that will make an individual’s consent informed and meaningful. Consent is required for the collection of personal information and the subsequent use or disclosure of this information. Sometimes, simply for practical reasons, all necessary consents may be sought just prior to data collection, and if so individuals are always given the opportunity of opting out without prejudice. In certain circumstances, consent with respect to use or disclosure may be sought after the information has been collected but before its use (for example, when a UDS client realizes after the fact that it would like to service a previously unanticipated information research objective)

Principle 4: Limiting Collection – Collect only what you say you are going to collect. The collection of personal information shall be limited to that which is necessary for the purposes identified by UDS and disclosed to the respondents, and all information shall be collected by fair and lawful means. Both the amount and the type of information collected shall be limited to that which is necessary to fulfill the disclosed purposes of the data collection. UDS shall specify the type of information collected as part of its information-handling policies and practices, in accordance with the “openness principle,” which is closely linked to the “identifying purposes principle” as well as the “consent principle.”

Principle 5: Limiting Use, Disclosure, and Retention - Personal information shall not be used or disclosed for purposes other than those for which it was collected, except with the consent of the individual (or, in extremely rare instances, as required by law). Personal information shall be retained only as long as necessary for the fulfillment of those purposes. UDS has developed guidelines and implemented procedures with respect to the retention of personal information. These guidelines include minimum and maximum retention periods to protect the confidentiality rights of survey participants. Personal information that has been collected from an individual shall be retained long enough to allow the individual an opportunity to access the information and verify its authenticity.

Principle 6: Accuracy - Personal information shall be as accurate, complete, and up-to-date as is necessary for the purposes for which it was originally or subsequently intended. The extent to which personal information shall be accurate, complete, and up-to-date will depend upon the use of the information, taking into account the interests of the individual. Information shall be sufficiently accurate, complete, and up-to-date to minimize the possibility that inappropriate information may be used to make decisions or recommendations that might not reflect the individual’s attitudes at the time of the decision or recommendation.

Principle 7: Safeguards – It is not what you say; it is what you do! Personal information is protected by security safeguards appropriate to the sensitivity of the information. The security safeguards shall protect personal information against loss or theft, as well as unauthorized access, disclosure, copying, use, or modification. UDS protects personal information regardless of the format in which it is held. The nature of the safeguards will vary depending on the sensitivity of the information that has been collected, the amount, distribution, and format of the information, and the method of storage. UDS makes its employees aware of the importance of maintaining the confidentiality of personal information, and care is always taken in the disposal or destruction of personal information, to prevent unauthorized parties from gaining access to the information.

Principle 8: Openness – Let the sunshine in! Transparency is the best guardian. UDS is open about its policies and practices with respect to the management of personal information. Individuals are able to acquire information about UDS’s policies and practices without unreasonable effort, and the information shall be made available in a form that is generally understandable.

Principle 9: Individual Access – If it is about you, then you have a right to access. Upon request, UDS shall inform any individual of the existence, use, or disclosure of his or her personal information and the individual shall be given access to that information. An individual is able to challenge the accuracy and completeness of the information and have it amended as appropriate. When an individual successfully demonstrates the inaccuracy or incompleteness of personal information, UDS shall amend the information as required. Depending upon the nature of the information challenged, amendment involves the correction, deletion, or addition of information.

Principle 10: Challenging Compliance – Respondents have a voice, and it will be heard. Individuals are able to address a challenge concerning compliance with the above principles to the designated individual or individuals accountable for the UDS’s compliance, and there will be no “passing the buck.” The person responsible for confidentiality and security will address the challenge. UDS investigates all complaints, and, if a complaint is found to be justified, UDS will take appropriate measures including, if necessary, amending its policies and practices.

These ten principles, which you can view in more detail in our privacy statement, serve as our guide for respecting and protecting survey participants’ rights of privacy and confidentiality. State-of-the-art security technology is necessary but not sufficient to protect these rights. It also requires a thoughtful and enforceable set of policy guidelines and a designated competent data custodian and policy policeman to do the enforcing. Any provider of survey services that does not put confidentiality and privacy as a top priority will be cutting off their own legs, because client institutions will understandably balk at using their services if they feel confidentiality and privacy may be compromised. Client institutions should be equally demanding in privacy and confidentiality assurances for a more basic reason: survey responses cannot be assumed to be valid when rendered by respondents who harbor doubts about the privacy and confidentiality of their responses.

The concept is simple: confidentially and privacy are the rights of respondents and essential to good survey results. There must be no compromises, period, and the earned trust of the surveyor is the best way to be sure there will be none.

]]> http://universitydecisionsupport.com/?feed=rss2&p=94 0 http://universitydecisionsupport.com/?p=57 http://universitydecisionsupport.com/?p=57#comments Thu, 30 Oct 2008 13:04:40 +0000 Don http://universitydecisionsupport.com/?p=57 In a previous BluePaper we addressed the question of how many of your students you need to survey to get a “good” survey result-viz., the question of sample size-and we ended our answer to that question with another rhetorical question: Will a “good” sample ensure a “good” survey result? Leaving aside for the moment what exactly is meant by a “good” survey result, it will be obvious to anyone that, while a “bad” sample will ensure a “bad result,” it may not be as obvious that a “good” sample will not ensure a “good” survey result. It is the old “necessary but not sufficient” condition. Taking up the question of what exactly makes for a “good” survey result leads to a line of inquiry that, like the question of sample size, yields some surprises and provides some not-so-obvious answers.

What is meant by a “good survey result”?

Well, in the most general of terms, a good survey result is one that tells you what you need to know, that is, what you set out to learn in the first place. But peel this onion back a few layers and you will see that the threats to a “good” survey result, so defined, are numerous. Ironically, the most common threat to a “good” survey result is the one that is most controllable-viz., lack of clarity about what you seek to know. Poorly formulated information objectives account for as many poor survey results as do technical methodological defects. There are other threats to be sure, but all these are manageable, and there are also specific methodological practices that can help you to avoid them. In one way or another, most threats to a “good” survey result, other than those attributable to a lack of clarity of information objectives, boil down to threats to the validity or reliability of the results. Of these two, validity, for reasons that will be described shortly, is by far the more critical.

But first, allow a short digression here to avoid the risk of some later terminology-induced confusion. In question here is the common misuse of the term “validity,” The term is often incorrectly used in reference to samples, but it really applies only to survey results. If we clarify the use of the term first, our discussion of “good” survey results will be made easier. We are frequently asked “how large must the sample be to be valid?” This question cannot really be answered, because validity is not a property of a sample. Validity is a property of the responses recorded from the survey respondents, i.e. the survey results. If you ask a student how much time s/he spends studying, and s/he exaggerates by reporting double what they actually spend, then the response is not valid. That is, it does reflect the facts as they are. If students are going to be untruthful about the time spent studying, there is nothing to gain by asking the question to more and more of them. In fact, asking more and more respondents will only make matters worse; it will lead to an invalid result in which you have high but unwarranted confidence.

The concern of those who incorrectly ask about the validity of a sample originates with an intuitive, if vague, notion that you cannot know the attitudes or habits of a large group of people by asking only a few. This is a very reasonable concern and an understandable, and generally correct intuition, but it has nothing to do with the construct of validity as it is applied in survey research. As applied in survey research, validity refers to a property of the responses, i.e. the survey results, but you cannot control the validity of your results by increasing your sample size. Two examples will make this obvious. First example: You want to know what percentage of your students are liars, so you survey them with the question “do you tell lies?” What can you know from a sample of one? What can you know from a sample of one million? In both cases the answer is little or nothing. Second example: why does a chef stir the soup before tasting it? Because if she stirs it, the soup becomes homogeneous. Hence, it does not matter from where she takes her sample spoonful. She needs only one spoonful to know how the entire pot tastes, to know the fact she seeks, viz. “is the soup done?” If everyone in a population is the same, then a sample of one tells all. Your intuition is probably telling you that the more heterogeneous the population, the larger your sample size needs to be, and, if it is, our compliments to your intuition, because it is correct.

So, the question we should ask about the sample is “is it representative?” not “is it valid?” Representativeness is a property of the sample; whereas validity a property of the survey responses, i.e. of the survey results. (Please see our BluePaper entitled “How many of My Students do I Need to Survey?” for a discussion of sample size.)

Now that we are not at risk of misusing or misunderstanding the term, validity, let’s return the focus of our discussion to the question of what makes a “good” survey result. As we have already said, a good survey result is one that delivers what you need to know. To reach this objective you first need clarity about your objective, and then you need a survey process that can produce results that have both the basic attributes of validity and reliability. The importance of clarifying, in advance, not only what you need to know, but also why you need to know it cannot be understated. Ask yourself “what managerial objective or action am I trying to inform with the survey results, and what managerial action will I take if I discover A, and what action will I take if I discover B? Another important point to make here is that the greater the clarity you achieve concerning your information objectives beforehand, the easier it is to achieve validity and reliability in the results, and the more likely it is that you will. I will make this connection clearer later, but first, let’s discuss what is meant by the attributes, validity, and reliability.

Validity and reliability: what are they?

Validity and reliability are very simple, but very specific, concepts. Validity simply refers to the degree to which a response reflects (or measures) what we intend for it to reflect. For example, it will be understandably difficult for anyone in current times to believe that the circumference of one’s skull can tell you anything about their intelligence, but there was a time in the history of psychology in which this was considered one way to measure a person’s intelligence. Now, of course, we know that there is no relationship between the two, and that skull circumference is not a valid measure of intelligence. Similarly, in our earlier example of wanting to know what percentage of our students are liars, their answer to the question “do you tell lies?” will not reveal if they are or are not liars. Someone who says no and is lying cannot be distinguished from an honest “no.” So, unless a survey question, evokes an accurate and truthful response, it will lack validity. Invalid responses are useless. There are no statistical analyses nor any other methodological tool that can restore validity to invalid responses. Validity has to be built into the survey from the outset.

On the other hand, reliability refers simply to the stability, or reproducibility, of a response. Skull circumference as a measure of intelligence is not valid, but it is extremely reliable. If you take ten, a hundred, or a thousand measures of someone’s skull circumference, you will get pretty much the same result on all measurements-i.e., the measure is highly reproducible. The measurements however, will not tell you a single thing about intelligence. So it is a reliable but not a valid measure of intelligence (though it is a valid measure of how large the person’s head is, if that is what you sought). Survey responses can have some, but incomplete, reliability. For example, if you want to know how satisfied students are with the variety of foods served in the dining hall, they may be quite satisfied overall, but, depending on what was on the menu the day you ask them, they may report being a little more or a little less satisfied. Even attitudes that are more or less stable will have some small degree of day-to-day variability. The point here is that reliability is not an all or nothing attribute, rather there are degrees of reliability.

We have used some extreme examples here to illuminate the concepts of validity and reliability, but, in most practical cases of institutions surveying their students, the issues of validity and reliability are not as clear cut. For example, suppose you want to measure the how effective your process for matching roommates is, and, as a measure of your effectiveness, you decide to ask the student residents “do you like your roommate?” Will this question produce valid responses for answering your question about the effectiveness of your roommate matching process? It may tell you something about the matching effectiveness, but it assumes roommates who like each other are good matches. Maybe the roommates like each other because they are good drinking buddies, which leads both to never study; or, maybe a student likes his or her roommate as a person, but simply does not prefer to room with them. Perhaps a student who does not like his or her roommate fears word may get back to them, so they simply report liking the roommate more than is the case. So, a response may have some, but not complete validity. The point here is that validity, like reliability, is not an all or nothing attribute, rather there are degrees of validity. Perhaps a different question would have more validity for the information objective-e.g., “If given the opportunity to choose your roommate, would you choose your current roommate?”

Now let’s say something about why validity is more critical than reliability. Think about it, if a response is valid, it is a true measure of what it was intended to measure. And, if it is a true measure of what it was intended to measure and you make repeated measurements of the characteristic, you will get the same result every time. Hence the measure is reliable. Therefore, if a measure is valid, it will also necessary be reliable. The inverse, however, is not true: a reliable response does not guarantee its validity. The moral of this story? Concentrate on validity, and reliability will take care of itself.

Clarity of information objectives.

This is a difficult issue. An exchange between Alice and the Cheshire Cat in Lewis Carroll’s Alice in Wonderland comes to mind:

Alice: Would you tell me, please, which way I ought to go from here?
The Cat: That depends a good deal on where you want to get to
Alice: I don’t much care where.
The Cat: Then it doesn’t much matter which way you go.
Alice: …so long as I get somewhere.
The Cat: Oh, you’re sure to do that, if only you walk long enough.

This is not exactly a good strategy for designing surveys. As simple as it sounds, it is essential in surveying that you achieve great clarity about what you seek, before you start designing a survey. This includes great clarity in your terminology. For example, suppose you are interested in knowing if your off-campus students feel integrated into campus life. Asking the question “do you feel integrated into campus life” may get you some information, but it will not be clear exactly what information. First, different students will have different levels of need for being integrated, so they will have different thresholds for what qualifies as integrated. Second, students will have different interpretations of what the term “integrated” means. Maybe to some it is a matter of how you feel about the campus experience, while to others maybe it is a matter of the participation in on-campus activities. You, the surveyor, must decide what information you seek, how you want “integrated” to be defined, and then ask the question in a way that makes your definition clear to the student. One way to help achieve clarity of the constructs you are asking about is to “operationalize” them. For example, maybe you would operationalize the construct of “integration into campus life” as the number of non academic on-campus activities in which the student participates. The more operationalized a concept is, the easier it is to achieve validity and reliability in its measurement. The risk, of course, is that one can, if not careful, operationalize the meaningfulness out of a response by over operationalizing it. A careful balance must be achieved between clarity of a response to a question and its meaningfulness.

A technical word here about information is in order. Above, we have used the term information, but what exactly does it mean? In information theory, as well as in scientific surveying, the term “information” has a very specific meaning. Information refers to uncertainty, and a response that reduces uncertainty conveys, or “transmits,” a lot of information. A response that reduces little uncertainty transmits little information. So, in our earlier example where we asked a student if he or she is a liar, the response, whether it be yes or no, will reduce the uncertainty by no practically useful amount. We will be just as “in the dark” about whether they lie after their response as we will have been before it. In the example of asking a student if they feel integrated into campus life, an answer of yes or no reduces uncertainty somewhat, but maybe not as much as you need it reduced-e.g., maybe they answer yes because they have no need or desire to be integrated. In the example of asking a student if they are a member of one or more student organizations, an answer of yes or no reduces practically all uncertainty of the matter (assuming, of course, that they are not lying). But notice that, in each of these sample questions, there is an incrementally greater level of specificity about what we are asking: “do you feel integrated into campus life?” can mean different things to different people. But, asking “are you a member of one or more student organizations?” will be interpreted more or less the same by everyone, especially if the question also clarifies what qualifies as a student organization. The qualification criteria could be easily handled by simply listing all student organizations and asking the student to indicate those of which they are a member. In designing a questionnaire, it is a useful exercise to ask yourself, for each question, what uncertainty will the response to each resolve? and to what degree will it be resolved? If the answer to these two questions is either “it is not clear” or “not much,” the question needs revision. Achieving clarity with your information objectives will go a long way toward ensuring you compose well structured questions; ones that will produce valid and reliable responses, at the same time as reducing the amount of uncertainty. In the final analysis, that is why we conduct surveys.

So, we have talked about response validity, response reliability, and clarity of information objective(s) We looked at how each of these properties drive whether we get a “good” survey result. We have also cited some examples of the threats to getting a “good” survey result.

To really appreciate the importance, as well as the difficulty, of achieving a “good” survey result, we suggest the following exercise: try to write a survey question (or set of questions) that will resolve practically all uncertainty concerning the following question: How many of your off-campus students would elect to live on campus if the housing fees were lower? Then, subject your formulation of the question to an evaluation of how you would control for the following threats to validity and reliability:

Common threats to response validity and some effective measures to overcome them:

1.     Question not clear to respondent (e.g., “Do you want more campus activities?” Meaning what, more to choose from? more opportunity to participate in existing ones? more time made available for participation?)-Be clear and precise as to what information you seek, operationalize the construct.

2.     Too many dimensions of a single construct included or implied in wording of question (e.g., “Are you satisfied with your residential life?”)-Decompose large construct into smaller operationalized constructs.

3.     Response scale too vague or not appropriate to question being asked (e.g., “How often do you take your meals off campus?” [  ] never, [  ] when I don’t like on-campus menu, [  ] when my family comes to visit, [  ] often, [  ] as often as I can )-Operationalize the construct being scaled, apply proper measurement scale (nominal, ordinal, interval, ratio), and properly and clearly label the response options. Seek competent technical survey development expertise.

4.     Student feels a risk to confidentiality and that his or her response might become known by others (e.g., “Do you get along well with your resident assistant?”)-Communicate that responses are confidential, use an external vendor with high security to conduct survey.

5. Student has tendency to give popular or politically correct responses instead of expressing their honest feelings (e.g., “Would you be willing to room with student of a different race than yourself?”)-Include other “concordance-checking” questions, e.g., “Would you support a race-blind policy of roommate assignments?” elsewhere in the survey, and check for consistency during data analysis.

6.     Student believes no follow up action will be taken on survey results, so does not invest in giving thoughtful answers (e.g., because the survey of students’ satisfaction with room assignment is being conducted in the face of years of continuing complaints from students; complaints that have, to date, resulted in no change to the assignment policy or process)-At the time of the survey, announce that survey results will be posted, for student viewing, on the department’s web site or published in the campus newspaper, and that the posting will include the response by student services management.

7.     Student does not feel s/he is a stakeholder in the object of the question, therefore no thought is given to the response (e.g., “Do you think the compensation of student service directors is sufficient to attract talented managers?”)-Avoid such questions, since the issue is not likely to be important to students and their responses are not relevant to managers of the issue. Or, make the connection to their interests explicit in the question.

There are other threats, but these are the ones you are most likely to encounter in surveying students on student service issues. If you are able to avoid the threats to the validity and reliability, you have properly clarified the information you seek and why you seek it, and you have found the proper balance between the meaningfulness of the questions asked and how they have been operationalized, then you will most probably obtain a “good” survey result. At least you will have “good” survey data. Raw survey data, however, are not information, rather they are potential information that get processed into actual information, by means of proper analysis. The engine for performing that processing is statistical analysis.

In a later BluePaper we will take up the topic of statistical analysis of survey data. It is simpler than you think. Meanwhile, we invite you to take one or more of the surveys we have developed to see how we have achieved validity and reliability in our surveys. Data-driven decision making requires more than just data. It requires valid and reliable data that have been properly processed into managerially relevant information with the proper statistical analysis. More on these ideas in a future BluePaper.

It typically requires less labor and cost to survey everyone than to invest in an elaborate sampling plan. The marginal increase in labor and costs declines rapidly after the first few thousand respondents, while the cost of devising and enforcing a statistical sampling plan can be prohibitive.

Still, in some cases, a sample may make more sense than a census – e.g., a large institution needs a pilot study conducted quickly to evaluate how many students will support some proposed emergency measure. In these cases, it is necessary to consider sample size and, since both cost and precision increase with larger sample sizes, the choice of sample size is a tradeoff between the informational value of additional data judged against the cost of gathering them. The precision of the sample information is best illustrated by what most people know as the “margin of error,” which is expressed as a plus and minus error range. So, if a survey shows that 60 percent, +/- 5 percent, of the student body will support a university policy, it means that most likely somewhere between 55 and 65 percent of the students support the policy. Specifically, it means that there is a 95 percent probability that the actual or “true” percentage of students who support the policy is between 55 and 65 percent, and therefore, necessarily, a 5 percent probability that it falls out side this range. So you see, the uncertainty in never resolved completely . . . except with a census.

Now suppose this estimate was not precise enough for your decision-making needs. For example, maybe you had decided that you would implement the policy if more than 56 percent of the students support it. With the current result it is uncertain if this criteria is met. Suppose you wanted to know, or at least be very confident, what the “true” percentage is to within plus or minus 2.5 percent. A general rule of thumb is that to decrease the margin of error by half you must quadruple the sample size. So, in our example above, assume our estimate of 60 percent, +/- 5 percent, was calculated on a sample of 1,000. If we want our margin of error to be reduced by one-half — viz., to 2.5 percent – we will need a sample of 4,000. You can see that the marginal increase in precision with increasing sample sizes diminishes rapidly. Now if you wanted to reduce again, the margin or error by one-half – i.e., from 2.5 percent to 1.25 percent, you would need a sample of 16,000 – i.e., 4,000 x 4.

Now let’s consider costs. Assume for the moment that the survey costs $1 per student. By increasing the sample size from 1,000 to 4,000 your total survey cost increases by $3,000 and your margin of error decreases by 2.5 percentage points. This represents an average cost of $1,200 per percent of precision gain – i.e., $3,000/2.5 = $1,200. Similarly, by increasing the sample size from 4,000 to 16,000 your total survey cost increases by an additional $12,000 and your margin of error decreases by 1.25 percentage points. This represents an average cost of $9,600 per percent of precision gain – i.e., $12,000/1.25 = $9,600. So you see, achieving increasing degrees of precision requires a disproportionate increase in the sample size and therefore becomes increasingly costly . . . unless of course the per-student cost of surveying is negligible, which it is in many online surveys, and this is essentially why we advocate conducting a census instead of sampling.

But, if you must sample, how should your sample be selected? It is a widely held belief that, to be valid, a sample must be random. That is, every member of the student population must have an equal chance of being selected into the sample. This belief is not correct, because, in many situations, nonrandom samples can give more reliable results than random ones. It depends on how much information you have about the population beforehand. In the case of your student population, this is typically a lot.

The goal in sampling is to have the sample be representative of our population. When we do not have enough information about the population to choose a sample that we can know, a priori, is representative, we resort to random sampling, hoping that the laws of chance will produce a sample that more or less mirrors our population. One caveat is in order here, and that is that some inferential statistical procedures require that the computed estimates come from a random sample in order for the conclusions to be valid. Typically, however, the managerial decision making of student service organizations, which use the survey results of the type being described here, are gathered for far more practical purposes.

In practice, at some stage of a sampling protocol, we often resort to random selection. For example, if we know we have 70 percent undergraduate students, half of whom are male and half of whom are female, and 30 percent graduate students, 40 percent of whom are male and 60 percent of whom are female, we would sample these subgroups in the same proportion that they are known to exist in the population, but the selections from each subgroup would be random. This concept is easily generalized to many more (and therefore smaller) subgroups, so that selecting, even in a non random manner, a few students from each of the multiple subgroups would, with near certainty, produce a representative sample. The point is: it is more important to have a representative sample than to have a random one, and it is ideal to have both.

Ah, but even with a census there is a problem, or at least there could be, that might threaten the validity of the results: you may give the entire student body the opportunity to respond to a survey, but not all will. Invariably, some students will not respond to the survey, no matter how much you may coax them. So the data, it could be argued, constitute a sample, albeit one without a premeditated sampling plan. After all it is a subset, not the entirety, of the population. There is an unwarranted tendency to be less concerned about the reliability and the margin of error with the results of a census than with sample results. While this is generally inconsequential unless the non-response rate is high, say 20 to 30 percent, it requires attention. Fortunately, in the case of student surveys it is not a problem for the following reason: the demographics of the entire student population will be well known from university data sources, and demographics are routinely included in the survey questionnaire. Thus, after the data have been collected, it is possible to know how close the demographics of the sample are to the population. The sample results can then be weighted to achieve, in the sample, a proportionate representation that corresponds to that known to exist in our population. For example, suppose it is known that in the student population half the students are male and half female, but that it is observed that in the sample 25 percent of the respondents are male and 75 percent female. Unweighted, the responses of the males are only half of what they should be (25% vs. 50%) so males’ responses should be weighted by a factor of 2.0. On the other hand, unweighted, the responses of the females are half again what they should be (75% vs. 50%) so they should be weighted by a factor of 2/3. With the weightings applied to the responses, the results will more closely match what they would have been expected to be, had all students – i.e., 50% males and 50% females –responded to the survey. Weighting is advised when it is either known or suspected that survey responses systematically with the demographic subgroups. If they do not, then there is no need to apply weightings.

A final misconception about sample size requirements needs to be dispelled, and that is the idea that when sampling from a larger population you need to take a larger sample. This is not true. A sample of 1,000 drawn from your student body of 10,000 will have more or less the same precision as a sample of 1,000 drawn from a student body of 50,000, or for that matter a sample of 1,000 drawn from a student population of 1,000,000. This fact is admittedly counterintuitive, but true nevertheless. It is beyond the scope of this “Blue Paper” to show why this is true, but if you are interested you can email us and we will reply with the reason.

In summary, sampling really is not a critical issue in surveying student populations, since we typically have efficient on-line access to the entire student population and can therefore efficiently conduct a census. It is also because we are in the position of having substantial information about the demographics of the student population, which allows for post data collection weighting of the results in the case of disproportionate response rates among different demographic subgroups.

But does a good sample, or a near 100 percent response rate to a census, guarantee a good survey result? No, not necessarily! In our subsequent “Blue Papers,” we will address other requirements for ensuring good survey results, as well as many strategies and methods for making practical use of survey results to foster data-driven managerial decision making.

We welcome discussion that flows from our BluePapers, and hope it will help student services managers better understand their own needs as they research the needs of their students. Please note that the discussions will be moderated to prevent spam and flame wars, but we welcome and will publish all sides of the conversations.

Please leave your comment below, and we’ll receive it immediately. 

These BluePapers will also serve as a forum for discussion about the tasks related to surveying university faculty and students on matters related to housing, dining, health, international students, and similar topics. 

If you’d like to suggest a topic for a future BluePaper, please do so using our BluePapers suggestions page. We will be e-mailed immediately with your suggestion. 

To arrange a meeting with us, please contact Don Pardew, who will arrange a time convenient to you.