Survey Pilot
AI Engineer Internship · King AbdulAziz University, Data Managing Center · 07/2026 – 09/2026
Piloting a survey normally means recruiting people, measuring reliability, fixing weak items and repeating after every revision. Survey Pilot shortens that loop in two ways: it tells researchers what reliability is normal for surveys like theirs, using instruments mined from real theses, and it lets AI respondents pilot the draft first.
The pipeline
Harvest → Filter → Classify → Store → Pilot → Calibrate.
- Harvest: query OpenAlex for open-access English doctoral and master's dissertations from 2010 onward that mention Cronbach's alpha, download them in parallel, and resume automatically after an interruption.
- Filter: keep only theses with a genuine instrument (rules below).
- Classify and store: group instruments by topic and keep their items and reported reliability in PostgreSQL with pgvector, so a new survey can be matched to published ones.
- Pilot: persona-conditioned LLM respondents answer the new survey, and reliability is computed from their response matrix.
- Calibrate: run the respondents on instruments with known human alphas and tune them before trusting any estimate.
Finding real instruments
Every harvested thesis is recorded with 32 metadata fields (topic hierarchy, institutions, citations, licence and more). A PDF is accepted only if it has all of: a real Cronbach's alpha value (a number, not just the word), a questionnaire appendix with actual numbered or Likert items, a statement that it is a doctoral or master's thesis, at least 40 pages, and English text confirmed from the PDF. Each rejection says why.
| Filter setting | Theses accepted (of 5,517) |
|---|---|
| All rules required (used) | 916 |
| Without the degree statement | 1,066 |
| Without the alpha value | 1,903 |
| Items appendix and English only (loosest useful) | 2,244 |
Piloting with synthetic respondents
- Respondents are conditioned on demographic backgrounds and five response styles (balanced, acquiescent, extreme, midpoint and careless), so the simulated sample includes the biases a real one has.
- For each scale it reports Cronbach's alpha with bootstrap confidence intervals and item-total correlations, and flags double-barreled, leading and ambiguous items.
- Benchmarked against human baseline data, the synthetic pilots matched real-survey reliability within 0.05 Cronbach's alpha.
- A FastAPI backend runs survey pilots on local Ollama models and large batch jobs on vLLM on Modal.
Scope
Survey Pilot is a pre-validation aid, not a replacement for piloting with people. It checks internal consistency and how well a draft covers its topic, not construct validity.