Developing a survey is a multistage, iterative process. Before creating a survey instrument, be sure to review the literature carefully. It’s easier to adapt an existing survey than to create your own. If you plan to develop a survey, you should use a systematic process to improve your end-results. This article discusses one systematic process you can use to develop survey scales.
Phase 1: Item Development
In Phase 1, generate an item pool, which fully covers the construct(s) of interest. First, use concept diagrams and definitions to map out the content domain. Next, generate a set of questions (i.e. item generation) which covers the entire construct. This item pool should contain two to three times as many items as will appear on the final instrument. Ask content experts and members of your target population to review the items as you revise them to ensure they are clearly written, relevant and representative of the construct (i.e., content validity).
Phase 2: Scale Development
In Phase 2, identify the best set of items from the item pool. First, pre-test the item pool with a small sample from your target population and ask for feedback. Revise the items and repeat pre-tests as needed until you are satisfied with item clarity and quality. Next, pilot-test the items with a larger sample (i.e. minimum of 10 subjects per item). Analyze pilot-test data using descriptive statistics, reliability statistics, and classical and modern test theories to identify and remove poor performing items (i.e. item reduction). Use exploratory factor analysis to examine the factor structure of the pilot-test data (i.e. extraction of factors). These activities help you identify the items for your final scale.
Phase 3: Scale Evaluation
In Phase 3, establish reliability and validity evidence before you use the scale scores in subsequent statistical analyses. If you have done a thorough job in Phases 1 and 2, you are more likely to be successful in Phase 3! Begin by administering your scale to a new sample from the target population. Use confirmatory factor analysis to determine if the factor structure of your data is unchanged (i.e., tests of dimensionality). Calculate reliability coefficients to examine the consistency (e.g., Cronbach’s alpha) of the scores (i.e., test of reliability). Conduct a variety of tests (i.e., tests of validity) to provide evidence of the validity of the scale scores (e.g. predictive, convergent and/or discriminant validity). Pre-planning for these tests can help ensure you have adequate data to strengthen your validity evidence.
Done right, the process of survey scale development can be time consuming; however, being thoughtful and thorough when developing your scales will improve the quality of your research.
References
- Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar- Quiñonez, H. R., & Young, S. L. (2018). Best practices for developing and validating scales for health, social, and behavioral research: A primer. Frontiers in Public Health, 6(149), 1-17.
- DeVellis, R. F. (2016). Scale development: Theory and applications (Vol. 26). Thousand Oaks, CA: Sage Publishing.
- McCoach, D. B., Gable, R. K., & Madura, J. P. (2013). Instrument development in the affective domain. New York, NY: Springer.
