Do You Need a Survey?

Surveys are great for collecting information and are particularly useful for assessing that which cannot be "seen" (i.e., thoughts, opinions, beliefs, learning). With tools such as SNAP, SurveyMonkey, and StudentVoice, conducting surveys is easier than ever before. As a result, the number of surveys being conducted is on the rise. Perhaps relatedly, survey response rates continue to fall.

Before you conduct a survey at Case, please consider the following:

1. Is a survey the right choice for your research? open

 

Though conducting surveys is easier than ever, there are biases inherent to survey research that almost always make collecting objective data a preferable choice, when possible. By objective data, we mean data that can be observed without bias or distortion. Two examples of objective data would be the number of students at an event or the number of classes a student takes in a given semester.

For example, there are two ways to assess whether participation in events sponsored by your group has risen: conducting a survey or collecting attendance over time.

The first, conducting a survey, will produce biased results for several reasons. Students have to think back over a span of time and may forget to list events they attended. Some of the students who attended the event may choose not to respond to the survey, and some participants may have left the university.

The second way to answer this question, taking attendance at events and seeing how participation changes over time, is a much more reliable technique. Answering a research question using this approach requires some forethought, and data will need to be collected over time. That said, the results will be easier to interpret and the results will be more accurate.

This, of course, is only one example of a type of research question that is best answered using a non-survey method. Other research questions may be better answered using, for example, focus groups, case studies, or face-to-face interviews.

2. Is a new survey necessary? open

 

If you have decided that a survey really is the best technique to answer your questions, please consider contacting the Office of Institutional Research to see if your questions have already been asked. The Office of Institutional Research surveys students on a regular basis (for a list of previous surveys, as well as our survey schedule for the next several years, click here). It is possible the questions you want to ask have already been asked. In fact, in some instances, Institutional Research has survey data going back several years.

Much of the data Institutional Research collects is IRB protected. As such, we cannot and do not release student data without an approved IRB protocol application. However, we can and do work with campus groups to formulate researchable questions, conduct the necessary analyses, and share the results that are of particular interest. We may also be able to provide richer information or answer more complex questions by linking data you've collected with institutional data.

3. Have you given yourself enough time to define questions and design a questionnaire? open

 

Designing a questionnaire that provides reliable answers to all of your research questions is not a simple endeavor. In fact, this process may take several months. Writing a good questionnaire starts by definining questions and thinking about what you want to be able to say with your results. For example, if you want to know about student engagement, you will want to carefully think through all types of engagement. Do you want to study classroom engagement only? Engagement with the community? Engagement with peers? Faculty? Administrative personnel? If you want to study student satisfaction, how do you plan to measure "satisfaction?" Similarly, if you want to be able to look at group differences (i.e., men versus women), you will need be sure you have a way of determining group membership. Though this may seem obvious, it is important to think of possible group differences you would like to explore before writing the questionnaire.

Once questions have been defined, the questionnaire itself must be written. Click here for a .pdf of a presentation discussing how to write good survey questions and construct research-quality questionnaires.

 

4. Who should receive the survey? open

 

When conducting survey research, it is also important to consider which students you intend to sample. If questions ask about a specific event or activity, it is best to survey just those who attended or participated. It is also important to realize that, regardless of how well it is written, many students may still choose not to answer your survey. In many cases, it is better to invite all participants to answer the survey than to invite only a sample.

5. Does your research protocol need to be approved by Case's Institutional Review Board (IRB)? open

 

If you intend to use the results of your survey for anything but program improvement, or if you intend to share results outside of the university (i.e., presentations, publications) you may need to have your research approved by Case's Institutional Review Board. If your research requires IRB approval, you cannot begin data collection until your protocol is approved. Depending on your research, the IRB approval process may take a month or more.

6. Have you thought about how to market your survey? open

 

How do you plan to tell students about your survey? Email messages are a good start; however, these are frequently overlooked or deleted by students, so follow-ups and reminders are usually a good idea. Other options include posters, announcements in the Case Daily, or presentations/announcements to the sample you are targeting. Incentives are also one way to motivate students to complete your survey. The website of the National Survey of Student Engagement (NSSE) has a section (click here) that provides detailed tips on marketing surveys and boosting response rates.

7. Have you thought about data security? open

 

There are three types of surveys: Anonymous, confidential, and non-confidential.

Anonymous surveys are those that do not collect any identifying information. Identifying information is anything that could be used to identify a student. Names, student ID numbers, and email addresses are all considered identifying information. Similarly, combinations of data that could be used together to identify a student can be considered identifying information. For example, an anonymous survey could ask for a student's gender, home state/country, major, or ethnicity; however, if a survey asks for all four, and the four items could be used together to identify a participant, the survey is no longer anonymous.

Confidential surveys do ask identifying information. These types of surveys are extremely useful, since the identifier can be used to link information from your survey to other identified data (i.e., demographics, previous surveys). This type of survey is also useful for before-and-after studies. By having an identifier, you can explore the ways in which the individuals in your study changed over time. Because confidential surveys contain identifying information, considerable thought needs to be put into a plan to securely store your data in order to keep the responses confidential.

The final type of study, non-confidential, is one that collects identifying information with no guarantee of confidentiality. This type of survey is not recommended, as students are much less likely to respond if they do not feel that their data will be kept in confidence.

Regardless of the type of survey you choose, you will need to explain to your participants whether their data will be anonymous, confidential or non-confidential. If you choose to conduct a confidential survey, you will need to further explain how you plan to secure your data and how long you plan to keep your data.

8. Have you thought about how you will analyze and report your results? open

 

Before finalizing your survey instrument, you should think carefully about the types of analysis you hope to conduct and the types of reports you intend to make available. Do you plan to examine individual change over time? Do you want to examine the success of a program by assessing student abilities or attitudes before and after attending? Do you want to determine if there are gender differences in certain student attitudes or behaviors?

The questions you want to answer affect not only the types of questions you ask, but also the analyses you need to conduct. Simple frequencies and distributions can be conducted with small groups of participants. Significance testing and regression analyses require larger sample sizes to be effective.

You sould also consider, ahead of time, how you want to share your data and with whom. Sharing results is an important part of the process. Students who see how the results of surveys they answered are being used across the university are more likely to participate in future studies. Similarly, sharing findings across campus may help to inform other interested parties about the beliefs, attitudes and behaviors of our students, thus reducing the number of surveys we need to conduct.

Remember, if you plan to share data outside of the university, you will likely need to have your research IRB approved.

The Center for Institutional Research at Case would like to thank the Cornell University Division of Planning and Budget for providing inspiration for this site.