Quantitative evaluation of data collection methods

Data Collection Methodology (DCM) also investigates various aspects of survey data collection from a quantitative perspective.

  • Measurement and reduction of non-response error enables DCM to see how:
    • non-respondents differ from respondents in important characteristics
    • non-response can lead to bias
    • bias can affect the accuracy of survey estimates
  • New ways of collecting data, such as web-based data collection
  • Mode effects: differences in survey results obtained using different modes of data collection (such as by telephone, through face-to-face interviewing or self-completion; paper or computer-assisted)
  • Question validation: Validating the accuracy of responses by:
    • comparing similar information from other surveys
    • comparing information provided in person and by proxy
  • Provision of methodological support for coding and social classifications

Examples of the quantitative evaluation methods DCM can use

Split sample designs

This involves splitting a sample into two (or more) random parts to compare how alternative design features affect response such as different wording of questions. For example, when testing proposed questions, half of a survey sample was asked an existing banded income question and the other half was asked to estimate an exact value for their income. The main aim was to investigate whether both questions produced similar estimates of income.

Comparison of results from similar questions between surveys

When trying to improve questions on educational attainment, data collected from surveys using a simple educational attainment question and those using a detailed question were compared in order to identify any discrepancies between the outputs for the different measures. Using this approach, it was possible to test the acceptability and accuracy of the simple educational attainment question and the feasibility of using such a question in the development of a new social survey.

Comparison of results between established survey questions and modified versions

New survey questions can be tested to observe if they have improved the rate and accuracy of response significantly from the original version of the question.

Comparison of data collected from different modes

Data collected from the same survey question, but from different modes of data collection (such as telephone and face to face interviews) can be compared to assess potential mode effects.