The Survey Production Pipeline: From Approved Questionnaire to Delivered Data

Most market research software discussions focus on the commercial workflow: bids, suppliers, fieldwork, and invoices.

But there is another workflow that can create just as many problems.

Between an approved questionnaire and the final dataset, the survey needs to be versioned, programmed, tested, translated, soft-launched, and converted into properly documented data.

The research management software used between questionnaire authoring and analysis needs to handle these handoffs reliably.

This guide looks at the features needed across that process.

The commercial side is covered separately in OnGraph’s guide to market research project management tools. Respondent fraud and panel-quality controls are another part of the workflow.

Handoff 1: Questionnaire Versioning and Approval

The first handoff is between the questionnaire author and the approver.

Most platforms already handle version history reasonably well.

Qualtrics automatically creates a version whenever a survey is published. It also saves a version when changes exist and more than an hour has passed since the previous save.

Forsta Surveys, from the former Decipher product line, records previous versions along with the survey XML, date, user, and number of changes.

Retention varies between platforms, so teams should check how far back they can restore a questionnaire.

Alchemer does not document full survey version history in the same way. Instead, it provides an undo buffer for the ten most recently deleted questions and five most recently deleted pages. User access logs are retained for 90 days.

Qualtrics also documents a built-in approval feature called Project Approval.

An administrator can define which users need approval before publishing. The survey owner selects an approver and can include notes. The approver can approve or reject the request and add feedback.

Equivalent approval functionality was not found in the Forsta or Alchemer documentation reviewed for this article.

In practice, client approval often happens outside the survey platform.

A client may approve the questionnaire document before programming and later approve the test link by email.

Because of that, the most useful feature is not simply an approval button. The system should be able to connect the approval record to a specific questionnaire version.

That makes it possible to answer a simple but important question later:

Which version did the client actually approve?

Two platform behaviors are also important when making changes during live fieldwork.

Qualtrics states that publishing changes only affects new respondents. People already completing the survey continue on the earlier version.

It also warns that restoring and publishing an earlier version can recover data affected by later question changes, but may also invalidate data collected after those changes.

Restoring a version alone does not publish it. That gives teams an additional review step before the live survey changes.

Forsta Surveys handles live editing differently.

Editing an active survey creates a temporary copy. Once the changes are ready, they are merged back into the live survey without breaking respondent sessions or changing the live link.

Handoff 2: Scripting and Survey Programming

The second handoff is from the approved questionnaire to the survey programmer.

Despite growing automation, survey programming still requires a significant amount of manual work.

Many vendors promote Word-to-survey conversion.

The actual documentation shows more limited capabilities.

Forsta Surveys’ Word Importer accepts .docx files and automatically identifies five question types:

• single select
• multi select
• text
• essay
• number

Other content is imported as unidentified content and must be classified manually.

More complex survey features such as grids, loops, randomization, quotas, and piping are not part of the automatically detected set.

This means Word import can reduce work on simpler questionnaires, but complex instruments still need programming.

Published estimates for survey programming time are limited.

Questra, which sells automated survey scripting software, estimates a manual survey build at 8–16 hours and around $70 per hour. It compares this with its own price of no more than $95 per survey.

Because these figures come from the vendor selling the automation product, they should be treated as vendor estimates rather than an independent benchmark.

The bigger factor affecting programming time is reuse.

At this stage, research teams should look for three capabilities.

First, the platform should have a reusable question library.

Tracking studies often reuse the same questions across waves. Without copy-forward functionality, programmers repeatedly rebuild the same modules.

Second, the platform should support code-level access alongside the visual builder.

Forsta, for example, provides Python access for advanced survey logic.

This matters because complex conjoint studies, MaxDiff surveys, and advanced routing can go beyond what a drag-and-drop editor handles comfortably.

Third, the platform should record which questionnaire version the programmed survey represents.

That connection becomes important when questions change during review or later fieldwork.

Handoff 3: Instrument QA

The third handoff is from programming to quality assurance.

This is one of the areas where good survey software can save a significant amount of time.

Several activities are often grouped under the word “testing,” but they are different.

Link testing checks the programmed survey against the approved questionnaire. Internal teams and clients normally test different paths and routing conditions.

Pretesting evaluates the questionnaire itself, often through cognitive interviews with people similar to the target respondents.

Pilot testing checks the overall research process, including recruitment, administration, and data cleaning.

Soft launch tests the live fieldwork assumptions using a limited number of respondents before the full launch.

AAPOR makes a clear distinction between pretesting and pilot testing. It also states that quality checks should take place throughout the survey lifecycle to make sure the questionnaire has been programmed correctly.

One detailed specification for automated questionnaire testing comes from academic research.

SurveyMan, developed by Tosch and Berger in 2014, treats a questionnaire like a program and runs static checks before launch.

It can verify whether survey branching forms a directed acyclic graph so that respondents always move forward through the instrument.

It also checks whether branch targets are valid top-level locations.

Another useful measure is minimum path length.

This shows the fewest questions a respondent can answer and still complete the survey. It helps identify routing that could accidentally allow respondents to skip large parts of the questionnaire.

Commercial survey platforms now provide some similar testing features.

Forsta Surveys offers a test environment with:

• a task log for open QA issues
• a recorder that tracks the current test path
• in-place text editing
• displayable QA codes

Forsta Professional Authoring, a separate product from the former Confirmit product line, can also generate an external test link for clients who do not have a platform login.

That test link expires after 30 minutes, which may be limiting for clients reviewing the survey across different time zones or schedules.

Qualtrics and Forsta can also generate simulated responses to test survey paths and populate reports.

However, generated test data has limits.

Alchemer states that generated test responses follow question, page, skip, and piping logic.

They do not follow questions hidden with CSS, do not trigger JavaScript actions, and do not run random number generation.

Qualtrics documents different limitations.

Generated test responses are not available for surveys using an authenticator element and are not generated for some question types such as constant sum, star slider, graphic slider, and profile matrix.

Custom validation is also not applied.

There is another issue to consider.

Qualtrics states that test responses can count toward real quotas and trigger real workflows unless those responses are cleaned up. Deleted responses can be configured to reduce quota counts again.

The practical takeaway is straightforward.

The more custom scripting a survey uses, the less automated testing can cover everything.

Automation can reduce repetitive QA work, but human link testing is still necessary.

Handoff 4: Translation and Localization

Translation should normally begin after the source-language survey has been programmed, tested, and approved.

If the base questionnaire changes after translation starts, every translated version may need to be reviewed again.

That is why translation comes later in the production workflow.

Cross-national research commonly uses the TRAPD process:

Translation, Review, Adjudication, Pretest, Documentation.

Variants of this approach are used by organizations such as the European Social Survey and European Values Study.

In a TRAPD workflow, two translators independently translate the source material. Their versions are reviewed together, differences are discussed, and an adjudicator approves the final version.

Walde and Völlm, writing in Frontiers in Psychiatry in 2023, explain some of the weaknesses of relying only on back-translation.

Back-translation can introduce its own errors.

It can also keep the translated version too close to the original wording and make it harder to adapt the question properly for the local culture.

ISO 17100:2015 provides another quality standard.

It requires a translation to be reviewed by a second qualified person who was not the original translator. It also defines minimum qualification requirements for translators.

TRAPD and ISO 17100 serve different purposes.

TRAPD is a research translation methodology.

ISO 17100 is a certifiable translation-services standard.

That difference is useful when clients ask which standard or process is being followed.

Survey-platform translation features are usually more practical and file-based.

Qualtrics supports translation import and export using XML, Excel, CSV, and TSV.

It also provides machine translation through Amazon or Google.

Qualtrics warns that machine translations may contain errors and recommends using them as a draft that should be professionally reviewed.

Forsta Surveys creates an Excel XLATE file for each language. That file can be sent to a translation vendor and imported again through Language Manager.

The file-based process creates three common risks.

First, if the survey changes after the translation files have been sent, the imported translations may no longer match the latest question identifiers.

Second, piping variables and HTML tags can be damaged during spreadsheet editing.

Third, translated text may be longer than the English version and no longer fit correctly in the survey interface.

That last problem only becomes visible when someone reviews the translated survey as it will actually appear to respondents.

Qualtrics provides a language selector in preview mode.

Forsta Professional Authoring includes a dedicated translator module.

What is still less common is a complete in-context translation workflow where translators can see each translated string inside the actual rendered question with its routing and piping.

Handoff 5: Soft Launch

Soft launch is the point where the survey moves from QA into real fieldwork.

It is both a quality check and a commercial check.

Cint defines soft launch as:

“a period of fielding where only a subset of the desired total quotas are filled before the full launch of the target group.”

There is no universal rule for how large the soft launch should be.

aytm documents a common range of 5–10% and can automatically pause fieldwork when that threshold is reached.

Drive Research notes that some email and list-based studies may use 1–2% of the sample pool, while others start with 100 randomly selected contacts.

The exact percentage matters less than what the team checks before opening the survey to the full sample.

Important checks include:

• actual incidence rate versus quoted incidence
• actual length of interview versus quoted LOI
• dropout
• straightlining

There is another important check that no dashboard can fully replace.

Someone should download and inspect the raw data.

This can reveal routing problems that were missed during link testing.

It can also expose missing or truncated variable labels.

A data-processing lead can then run a preliminary table to confirm that the data structure matches the questionnaire.

Two areas are especially useful to define with clear thresholds.

Length of Interview

The ESOMAR/GRBN Guideline for Online Sample Quality from 2015 notes research showing higher satisficing and breakoff after approximately 18–20 minutes.

That makes measured LOI useful for both survey quality and project costing.

Speeder Detection

Speeding rules should also be defined clearly.

Different thresholds can flag very different numbers of respondents.

Greszki, Meyer, and Schoen’s 2014 analysis of German Longitudinal Election Study panel data found that:

• 30% faster than the median flagged around 22% of respondents
• 40% faster flagged around 15%
• 50% faster flagged around 8%

The number of respondents flagged changed almost threefold depending on the rule.

The same study found that removing speeders did not materially change the substantive results.

The important point is that the chosen rule strongly affects how many respondents are challenged or removed.

The commonly referenced one-third-of-median rule is often associated with ESOMAR, but that specific figure does not appear in the ESOMAR/GRBN guideline text.

It should therefore be treated as an industry convention rather than a formal standard.

Dropout Analysis

Overall completion rate does not explain where respondents are leaving the survey.

SurveyMan separates two types of breakoff:

• breakoff at a survey position
• breakoff at a particular question

In one labour-economics survey run on Mechanical Turk, the researchers found that 72.5% of breakoffs occurred within the first six positions.

They linked this to the $0.10 incentive.

They also found that 52% of breakoff instances were concentrated around five questions, all of which were demographic questions shown early in the survey.

These numbers apply to that specific study.

The broader lesson is more useful: a single overall dropout rate cannot tell the team whether respondents are leaving because of where a question appears or because of the question itself.

Handoff 6: Data Map, Codebook, and Deliverables

The final handoff is from fieldwork to data processing.

This is one area where general-purpose survey tools can require more manual work.

A data map and codebook are closely related, but they are not exactly the same thing.

A data map normally describes:

• variables
• types
• positions
• value labels

A codebook usually provides the human-readable question wording and coding information.

Many research teams need both.

There is also a third deliverable: the tab plan or banner specification.

This defines crossbreaks, nets, and other requirements used during tabulation.

Without it, the data may be ready while the tabulation team still cannot start.

Forsta Surveys provides a data map API endpoint.

It can return a question-based view connected to the survey XML and a variable-based view connected to the data structure.

This provides a direct link between the questionnaire and the delivered dataset.

Forsta also documents several research-specific download formats, including:

• SPSS
• SAS script
• Quantum script
• Uncle tables
• Wincross
• Triple-S XML

These formats are designed for research data processing and tabulation rather than general spreadsheet use.

General-purpose survey platforms can still support the same workflow, but sometimes with more manual preparation.

Qualtrics exports data in:

• CSV
• TSV
• Excel
• XML
• SPSS
• JSON
• NDJSON

The SPSS export includes variable and value labels.

That means teams can create a usable data dictionary from the output.

However, it may need to be assembled using the SPSS export or a Word survey printout rather than being produced automatically as a separate named deliverable.

For a research team sending datasets to a data-processing department every week, this extra work adds up.

Triple-S is also worth considering as an interchange format.

Version 3.0, released in 2017 under CC BY 4.0, uses a metadata file together with a data file.

The 3.0 release added:

• UTF-8 support
• rich text
• multiple keys for hierarchical data
• a defined default language for multilingual surveys

These additions help translated questionnaires move more reliably into data-processing and tabulation systems.

Research documentation is also part of the final delivery.

The ESOMAR/GRBN guideline lists information that should accompany a study, including:

• questionnaire and data-collection documents
• average or median survey length
• fieldwork dates
• quotas used
• de-duplication methods and counts
• engagement measures
• participation rates
• completion rate

Connecting these outputs to other systems is a separate integration requirement.

OnGraph’s guide to market research software integrations covers that area in more detail.

What to Ask a Vendor

1. Is there an approval process, or only version history?
These are different capabilities. An approval workflow should show which version was actually approved.

2. What exactly does the Word importer recognize?
Ask which question types are automatically detected and which still require manual programming.

3. Is there a reusable question library with copy-forward between waves?
Also check whether programmers can access code as well as the visual builder.

4. What does generated test data not test?
Ask specifically about CSS-hidden questions, JavaScript, custom validation, and randomization.

5. Can external reviewers test the survey without buying a platform license?
Also ask how long the test link remains available.

6. Can translators review questions in context?
Check whether they can see the translated string inside the rendered survey rather than only inside a spreadsheet.

7. Can soft launch pause automatically at a defined threshold?
Also check whether the platform compares actual IR and LOI with the original project assumptions.

8. Which research data formats are generated directly?
Ask about Triple-S, SPSS, Quantum, and Uncle, and whether the system produces a codebook automatically.

Where OnGraph Fits

OnGraph builds this survey-production workflow into the research platforms it develops.

Its survey creation tool supports multilingual surveys, survey logic, and respondent-quality controls such as digital fingerprinting, speeder checks, and straightlining detection, along with SPSS export.

The broader workflow connecting survey production with fieldwork, suppliers, and reporting is covered through OnGraph’s market research workflow automation capabilities.

Custom and white-label development are available through its market research software development services.

The Bottom Line

Survey platforms are often compared mainly on their survey builders.

For research teams, a lot of the operational work happens around the builder.

The questionnaire needs to move through approval, programming, testing, translation, soft launch, fieldwork, and data delivery without losing information between steps.

That is why the handoffs matter as much as the survey editor itself.

A good platform should make those transitions easier, more traceable, and less dependent on manual work.