Jelke Bethlehem - Handbook of Web Surveys

In Designing the questionnaire, it is important considering that the questionnaire will be answered using different devices (desktop, laptop, tablet, smartphone) (see details in Chapter 5). The question with error message of Figure 3.5shows how it looks like on a smartphone. If a mobile device accesses a website without a mobile version, the user will still be able to navigate the page. However, differences in screen size will usually require the user to perform gestures or scrolling in order to browse the content in its entirety. Therefore, if a smartphone accesses a nonmobile version of an online survey, then it is likely that the respondent will see only a portion of the content/question or question completion may require zooming first in order to select desired response. Some question types (e.g., multiple choice, lists) will not generally take advantage of the smartphone's native features. Peytchev and Hill (2010) reported the results for a series of experiments comparing various aspects of questionnaire design and layout, including horizontal scrolling, number of questions per screen, direction of response options, impact of embedded images, and the use of open‐ended options, among others, using a smartphone. Couper and Mavletova (2014) explore the effect of scrolling versus paging design on the break‐off rate, item nonresponse, and completion time in mobile web surveys. The scrolling design leads to significantly faster completion times, lower (though not statistically significant) break‐off rates, fewer technical problems, and higher subjective ratings of the questionnaire. In general, web/mobile web survey design display should be accurately adapted on mobile phones, netbooks, and tablets as they do on desktop and laptop computers. Let's say a mixed devices approach should be adopted. When the questionnaire design is completed and the paradata decided, the software or the selection of the programming language to implement the web questionnaire is undergone, and the digital version of the questionnaire (including paradata and metadata) is created.

Figure 3.5 Message for asking answer to a compulsory question: smartphone device

Following the implementation of the web questionnaire and the sample selection, the fourth main step of the survey process is the Collecting data. Conducting data collection implies sending the web questionnaire link to the sampled units along with an invitation letter for survey participation (see Chapter 7) and monitoring the data collection process, for example, by sending solicitations and, eventually, applying the responsive design (see Chapter 8) to better finalize the sampled units' participation.

Data processing takes place from the fifth step forward. Processing data— step 1is on database creation , where many error risks need identification and their corrections adopted. For instance, item nonresponse must be considered, the reasons for them evaluated, and, when necessary, imputation methods applied (sub‐step Data imputation ). Coding should also be considered a sub‐step (sub‐step Code open questions ). Erroneous coding could cause misinterpretation of the survey results. Compilation of the questionnaire automatically generates the database and avoids the significant risk of errors connected with data transcription. It is a great advantage of web surveys; data errors are only due to incoherence or respondent little attention. Therefore, a data quality check is still necessary in web surveys, even if the risks for errors are smaller than in traditional paper‐and‐pencil interviews.

Once the checked database has been created, the second phase of data processing takes place ( Processing data — step 2 ). Note that several computations and activities carried out in the second phase of data processing are undertaken according to decisions done at the Designing the web survey step. Application of the estimation and weighting techniques allows for the extension of the survey results to the target population and to present the survey results (sub‐steps: Estimation technique choice, Calculation of weights and estimators, and Process data and produce tables ; see Chapters 12and 13). Estimation procedures are like to those applied in the traditional survey modes. However, several aspects encompass many advantages that are specific to the web. In many cases, it could be possible to link the survey microdata to another database providing individual data for auxiliary variables. A key variable useful for connecting the two databases at the individual level must be available though. Through database integration, modeling survey results, understanding participation behavior, and better profiling the respondents are possible. Integration with administrative databases, paradata databases, or other survey databases could be undergone.

Thus, in selecting the estimation techniques, the researcher needs to integrate the survey database (sub‐step: Integrate datasets ) with another database (e.g., an administrative database or a paradata database). This is useful when the researcher has planned to use auxiliary variables in the data processing, to compute weights, or to improve estimates. For instance, propensity scores technique ( Chapter 13) is one approach that uses auxiliary variables for estimation purposes.

After deciding possible integration and applying the appropriate estimation and weighting technique, the data processing ends up; tables and figures are then produced to synthesize the survey results. Presenting the survey results includes offering quality indicators as well. AAPOR (2016) provides a detailed description of rates and indicators and emphasizes that all survey researchers should adopt the standardized final disposition for the outcome of the survey. For example, response rate calculation is in many different ways. Not to simply state “the response rate is X ” is important. The researcher should exactly name the rate he is talking. In particular, there are two types of response rate:

1 The response rate type 1 (RR1), i.e., the number of complete interviews divided by the number of interviews plus the number of non‐interviews (i.e. refusal, break‐off plus non‐contacts and others, plus all cases of unknown eligibility);

2 The response rate type 2 (RR2), where also partial interviews are considered at the numerator of the rate.

In web surveys, as discussed in Chapter 5, in regard to possible errors that might occur throughout the process, quality also uses specific indicators that take advantage of the paradata collected by the survey system during the survey process. For example, it is possible to compute the time spent in the completion of the questionnaire.

The final step, the End of the survey process , involves writing the conclusions. Usually, a final reporting is written. A clear and consistent reporting of the methods complements comments about substantive research results. This is a key component for a methodologically reliable survey. AAPOR (2016) is stressing this idea. A set of standardization of the outcome details and outcome rates has been proposed, and they should be available and part of a survey documentation.

In the construction of the termed PAADEL (Agro‐Food and Demographic Panel in Lombardy), the steps described in the flowchart ( Figures 3.1and 3.2) were followed. This section examines some problems that arise at the different steps and the interrelationship between errors. In particular, focus is on the step of choice of the mode and on the use of adaptive design during the step of the data collection.