Jelke Bethlehem - Handbook of Web Surveys

The message in this chapter is that both TSE and total quality survey perspective embody the need to consider different error sources and that errors occur at every step of the survey process and, sometimes, they are interrelated.

In this chapter, a framework of the web survey process is proposed. The framework's main purpose is to provide a clear overview of the necessary decisions when organizing a mobile web survey or web‐only survey and to create a shared overview of the survey process steps. Practitioners, as well as researchers, can refer to the flowchart and obtain a clear understanding of the procedure to follow, the choices to do, and the type of errors that might occur. Thus, there is a more complete, integrated perspective in studying and interpreting errors. Understanding them becomes more easily.

The mobile web survey or the web‐only survey process differs in some respects from a survey based on traditional modes (paper and pencil, telephone, and fax). Errors arising in mobile web surveys and only mobile surveys have some specific characteristics due mainly to the coverage aspects of the target population and to various other possibilities (the availability of auxiliary variables, the time required to deliver the questionnaire and to receive the completed questionnaire, technical skills, the equipment needed, and many other situations). Therefore, a well‐defined outline of the survey research steps and of the errors that might occur at each step is vital for the surveyor. Furthermore, in many cases, the errors at different steps have some relationships, even if the relationship is not clearly defined and formalized. Thus, decisions pertaining to the reduction of an error at a specific step could increase the errors occurring at other steps of the survey process. For instance, increasing response rate by stressing the interviewee with a high number of solicitations might decrease the quality of the data (more item nonresponse, less accuracy, and so forth). Therefore, the overall quality of the survey process could be either improved or deteriorated depending on the decision made.

The flowchart mentioned in the previous section is in Figure 3.1. It shows the main steps and sub‐steps for a web‐only or mobile web survey based on probability sample.

Figure 3.1 Flowchart of a probability‐based web‐only or mobile web survey

The six main steps (gray highlighted) and their related sub‐steps (hell gray and white highlighted) are listed in the previous section. Hereunder, detailed comments of the flowchart steps are reported.

The first step is Determine the survey objectives, and it is a preliminary point of every survey, independent from the mode. Objectives need to be clearly stated and the questions to be identified. Since this flowchart considers web mode (either web only or mobile web), it should be critically evaluated if the objectives and the structure of the survey are adequate for that mode. Web mode implies a self‐completed questionnaire; thus, for example, self‐completed surveys are better in capturing sensitive topics. If the questionnaire is long self‐completion, especially using web mode, it is not in favor of high participation rates and smaller measurement error. This becomes especially critical in mobile web surveys, where the participation situation has a high probability to be disturbed from other activities or tasks the interviewee is doing.

The second step is Metadata definition. Recent literature and empirical research brought significant attention to metadata, showing a set of information describing the survey (that is, elements for evaluating the data quality, like the list and description of the variables, the target population, the sample units, and so forth). The metadata refer to variables and activities occurring in every step of the process. The International Organization for Standardization (ISO) defines metadata as “data describing and defining other data in a certain context.” In such a case, the contest is the condition under which the data collection and processing takes place. The definition of metadata is an important step, recommended for all surveys, independently from the mode. Details to consider in mobile web surveys (and in web‐only surveys) differ from traditional surveys due to the characteristics of various steps and sub‐steps. Metadata are mainly semantic based; a bad or imprecise metadata description affects the quality of the questionnaire and could cause errors in the answers (questions not correctly understood, questions that each interviewed could interpret in a different way). Metadata are useful for the selection of appropriate sampling techniques and for guiding and evaluating the procedures of the survey process. The so‐called metadata database contains metadata, which complement the data database. See Example 3.1. The data user should look at the metadata to be conscious of the quality and meaning of the data he or she is going to use. On the other hand, a surveyor should take care to write down the metadata precisely because accuracy is crucial for undertaking the survey process in a correct and valuable way. For example, if a surveyor is drawing his probability‐based sample from a largely incomplete sampling list, it will not be possible to obtain highly accurate results from the inference process. Thus, information about the sampling frames characteristics (coverage, reference time of the frame, i.e. is it updated or an old one?) is an important quality. Another example is a bad or insufficient description of one variable; in this case, each interviewed could assign to the variable a different meaning, sorting out in scarcely meaningful results. From the user point of view, this implies both the use of low‐quality data and the impossibility to understand what is the effective meaning of the variable and how to interpret the data.

The Eurostat website is offering a metadata database that includes Euro‐SDMX Metadata Structure (ESMS) (a set of international standards for the exchange of statistical information between organizations), classifications, legislation and methodology, concepts and definitions (CODED, Eurostat's Concepts and Definitions Database, and other online glossaries relating to survey statistics), glossary, national methodologies, and standard code lists.

A section reports the description of the variables in different sources. Variable descriptions are detailed explanations of the researcher's intended meaning of the variable in the questionnaire, and it is one example of basic metadata. For example, for the purposes of the Labour Force Survey, the following definition is used: “Employees are defined as persons who work for a public or private employer and who receive compensation in the form of wages, salaries, fees, gratuities, payment by results or payment in kind; non‐conscripted members of the armed forces are also included.”

In structural business statistics, employees are defined as “those persons who work for an employer and who have an employment contract and receive compensation in the form of wages, salaries, fees, gratuities, piecework pay or remuneration in kind.”

Furthermore, a worker is a wage or salary earner of a particular unit if he or she receives a wage or salary from the unit, regardless where he or she works (in or outside the production unit). A worker from a temporary employment agency is considered to be an employee of the temporary employment agency not of the unit (customer) in which they work. Metadata states that “employees include part‐time workers, seasonal workers, person on strike or on short‐term leave, but excludes those persons on long‐term leave. Employees does not include voluntary worker.”