Survey Planning and Reflection

Survey policies and procedures (separate page)

• Good Scenarios for Surveys
  • Most Important - There’s a strong need to collect relatively shallow information from a large number of people
  • There is a lack of existing relevant data for your purpose or research question  
    • If you are curious about data that has already been collected about your topic of interest, we highly encourage you to reach out to us in Institutional Research at [email protected].
  • Your questions for participants are very simple to understand and interpret and do not need to be customized per participant across multiple dimensions 
  • Your survey participants have clear motivation to actually participate
  • You feel confident in the possible range of responses you will receive for each question and thus can design a survey that accurately reflects the experiences of your participants 
  • You can obtain valuable and actionable insights using descriptive statistics and charts, alongside few to no open-ended questions
  • You have a clear plan for conducting the quantitative analysis - or qualitative analysis, if you are asking complex open-ended questions - and you and your collaborators have the time, resources, and skills to do an appropriate amount of analysis
  • You have discussed the purpose and goals of the survey with relevant decision or policy makers, and they have bought-into the project; you and your department will be well-positioned to ultimately take action based on the results and analysis of the survey
  • Typical, but not required - You are asking sensitive questions where anonymity or confidentiality is important
  • Optional - You want to be able to compare results directly to some kind of earlier result or finding

• Non-Survey Options to Consider

  Other Research Methods

  • Structured interviews
  • Focus groups (please reach out to Institutional Research at [email protected] for support)
  • Observational research
  • Quantitative analysis of existing data 

  Non-Research Methods for Gathering Input

  • Anonymous feedback form
  • Open hours for discussion or input 
  • Intercepts
  • New committee
  • Campus stakeholders interviews
  • Expert interviews
  • Roundtables 
  • Consult with established groups or committees
  • Actively recruit partners (students, faculty, or staff) to assist in decision making or policy development process

• Reflection Questions for a Survey Project
  • If you or your department has administered a similar survey in the past, what lessons were learned from prior administrations? How were the results of the previous survey administrations utilized?
    • A common problem with campus surveys at many colleges is the underutilization of the results. The most important stage of the survey process is the dissemination and discussion of the results. It’s worth thinking about the reasons that a similar survey in the past may or may not have led to new insights or policy changes.
  • What is the purpose of your survey?
  • Have you investigated whether relevant data already exists?
    • If you are curious about data that has already been collected about your topic of interest, we highly encourage you to reach out to us in Institutional Research at [email protected].
  • Do you have a research question in mind?
  • What population do you want to survey? How will you contact them?
    • Please consider that only a limited number of surveys will meet the criteria to be allowed in Daily Messages and only a select number of staff members have permission to email all students. In certain cases, Institutional Research ([email protected]) will provide lists of student emails as requested by those administering a survey.
  • How are you approaching the anonymity and/or confidentiality of respondents?
    • We recommend that you specify within the survey whether the survey responses will be anonymous (no names associated with responses at any point) or confidential (names will be associated with responses, but only a limited number of people will see this information). If the survey is confidential, it’s best to clarify within the text of your survey which staff or faculty members will have access to survey responses tied to names. 

• Analysis and Dissemination Planning

  We recommend that you create a draft of an analysis and dissemination plan prior to even writing your survey. To reiterate, a common problem with campus surveys at many colleges is the underutilization of the results. The most important stage of the survey process is the dissemination and discussion of the results. 

  Here are some questions to consider as part of your plan: 

  • Who will be responsible for analyzing this survey? What technical or tools will be utilized for this analysis? 
  • What are the goals for the analysis? What levels of breadth and depth of analysis will best serve the purpose of this survey?
    • While it sounds counterintuitive, aiming for a high quality but relatively brief analysis may best serve the purpose of the survey. 
  • Who is the audience for the analysis? How will you design the analysis for this audience?
  • How will you share the results of the survey? How will you communicate with the respondents about how their responses were used?
    • We highly recommend that you “close the loop” with all surveys, meaning that you communicate with your survey participants about the outcomes of the survey, i.e., what you learned or what changes were made at Williams as a result. This practice shows respect for your participants and their time, builds trust in your office or unit, and helps to build a better survey culture at Williams. 
  • How will you create a forum for discussing the survey results and considering next steps? 
    • Discussing and thinking through survey results takes time, particularly when this discussion is paired with consideration of policy/procedure changes. 

• Survey Concerns

  Survey Research Weaknesses

  • Response bias: Surveys rely on self-reported data, and respondents may not always provide honest or accurate answers. Respondents may be unwilling to disclose sensitive information or may be influenced by social desirability bias.
  • Nonresponse bias: Surveys may have a high rate of nonresponse, which can lead to a biased sample. This can occur when certain groups of people are less likely to respond to the survey, such as those with low income or low education levels.
  • Limited range of response: Surveys often rely on closed-ended questions that provide a limited range of response options. This can limit the ability to capture the full range of respondents' experiences and perspectives.
  • Limited flexibility: Surveys often rely on a fixed set of questions, which may not be flexible enough to capture the nuances of a complex research question. This can be a limitation when studying sensitive or complex topics.
  • Limited generalizability: Surveys are often conducted on a sample of the population, and the results may not be generalizable to the entire population.
  • Limited ability to observe behavior: Surveys rely on self-reported data, and may not be able to observe behavior or actions. This can be a limitation when studying behavior change or the effectiveness of interventions.
  • Cost: Surveys can be time-consuming and costly to conduct, especially if they are administered to a large sample of people.

  Over Surveying

  Issuing too many surveys, or surveys that are poorly executed can have negative consequences for the researcher and the population:

  • Reduced response rates: If a population is surveyed too frequently, respondents may become less willing to participate in surveys, leading to a decrease in response rates.
  • Increased response bias: When a population is surveyed too often, respondents may become more selective in their answers or give socially desirable responses, leading to an increase in response bias.
  • Decreased data quality: Over-surveying can result in respondents becoming fatigued and not putting as much thought into their responses, leading to a decrease in data quality.
  • Resource waste: Conducting too many surveys can be costly and time-consuming for both the surveyor and the respondents. It can also divert resources from other important projects.
  • Negative impact on the relationship with the population: Over-surveying can lead to frustration and annoyance among the population, which can harm the relationship between the surveyor and the population.

  It can also cause harm to the population

  • Privacy concerns: Respondents may feel that their privacy is being invaded if they are constantly asked to participate in surveys, which could lead to a lack of trust in the surveyor.
  • Psychological harm: Repeatedly asking individuals to participate in surveys can lead to survey fatigue and frustration, which can negatively impact their mental well-being.
  • Disruption to daily life: Over-surveying can disrupt people's daily lives, as they may feel burdened by constantly being asked to participate in surveys. This can lead to resentment towards the surveyor and a negative perception of the survey process.
  • Stigmatization: If a specific population is repeatedly surveyed, it may create a perception that they are being targeted or singled out for scrutiny, which could lead to stigmatization.
  • Distortion of data: If the same population is surveyed too frequently, it can distort the data collected and lead to inaccurate conclusions and decisions.

  Research Findings

  • People tend to predict their future behavior with 50-60% accuracy.
    • Oettingen and colleagues (2001) found that people's predictions of their future behavior were only accurate about 50% of the time. 
    • Roese and Summerville (2008) found that people's predictions of their future behavior were accurate about 60% of the time.
  • People’s previous behaviors tend to predict their future behavior with 50-70% accuracy. This is a wide generalization; specific predictions vary depending on the specific behavior and the context surrounding it. 
    • Laibson (1997) found that people's past savings behavior accounted for about 50% of the variance in their future savings behavior. A study by Shiffman and colleagues (1996) found that people's previous smoking behavior accounted for about 60-70% of the variance in their future smoking behavior.