Pilot Surveys: The Survey Before the Survey

How Pilot Surveys Can Help Government Agencies Maximize Research Efficiencies & Outcomes

One way to ensure that the survey or data collection process is as closely aligned to desired outcomes as possible is to conduct a pilot survey or exploratory study before developing the main initiative. It is important to note that pilot surveys or data collection efforts may have to go through the OMB and agency approval processes. The Paperwork Reduction Act, as administered by OMB, serves as an independent review of government survey and data collection engagements and is a key step in ensuring ultimate success.

Many federal agencies are required to collect and retain very rich data about the constituencies they serve and industries in which they are operating. Because of the scale of agency operations, the input required from diverse and influential groups of stakeholders can pose a significant challenge. A related challenge for government agencies is that the data they collect—whether transactional, demographic, or obtained through surveys or other means—is a valuable resource for citizens and other public constituencies. As a consequence, this data must be collected and handled with appropriate care.

Therefore, agencies must ensure that any survey research or data collection initiatives are methodologically sound and appropriate to the uses for which the data is intended. This caution applies not only to new efforts but also to ongoing initiatives that may be affected by changes in laws or regulations, citizen expectations, or the ability to compile data across agencies.

As an example of how conducting a pilot can help maximize survey outcomes, imagine that an agency wants to develop a new survey to augment industry metrics it is already collecting. However, potential sensitivities may exist among industry stakeholders who will be asked to participate—sensitivities that could affect stakeholder willingness to provide the needed information. The upshot is that the agency is not sure what the data to be collected might look like. In this case, the agency should consider launching a pilot survey to explore stakeholder interest in the initiative, test reactions to the survey questionnaire and related communication techniques, and evaluate data resulting from the pilot to ensure that the volume and quality will support the goal of augmenting existing metrics.

Not only does this pilot approach embody sound risk management, but it is likely to improve the quality of the ultimate engagement. There are also favorable cost considerations related to the pilot approach. Rather than invest the significant amount of capital required for a full-scale survey, the agency can make a much smaller investment to assess potential outcomes and the value behind the approach. Also, while an agency may not be able to obtain budgetary approval for the full engagement, it might be able to get approval for a pilot that would show proof of concept and break the initiative up into more manageable chunks.

In addition to validating the agency’s approach to the study, a pilot survey can help the organization formulate the most effective survey-related communication techniques. These range from inviting individuals to participate in the survey to rolling out the results so they reach the right audiences and are positioned to be interpreted correctly.

When all is said and done, a “survey before the survey” helps determine if there is benefit to the data collection idea. Conducting pilot surveys gives everyone a chance to feel good about the way things are going to work while also identifying and addressing any unexpected survey-related problems early in the game rather than stopping the effort to make expensive and time-consuming mid-course corrections.

From a risk management standpoint, it’s definitely the way to get a government agency survey headed in the right direction.

If you haven’t planned or conducted pilot surveys or data collection efforts before, it’s a good idea to check in with a trusted resource who can help you avoid any dead ends or potholes. An experienced researcher can help you plan your route so you arrive at your destination having learned as much as possible. This will also ensure that the much longer journey you will take with a full-scale deployment will end on a high note.