حريص (Haris): Semantic Census Data Validation Engine
GASTAT Data Innovation Hackathon (General Authority for Statistics & SDAIA) · Team Optimizers
Haris (حريص, “the one who watches over”) checks survey answers while they are being entered and flags combinations that cannot all be true, before a bad record reaches the national dataset. Built by Yasser Albogami, Omar Bin Talib and Hassan Algamdi for the “Semantic Guardian” challenge, it placed 3rd of 130+ teams.
The challenge
GASTAT asked for an assistant that plugs into survey platforms and catches semantic and logical errors in real time, going beyond rigid rule-based checks, with a confidence score for every response and a live demo. Each field can look valid on its own while the combination is impossible:
- a respondent aged 14 whose job title is “Senior Accountant”;
- no formal education, but a profession of “Neurosurgeon”;
- 22 years of work experience at the age of 20;
- a work start year of 1995 with a birth year of 2000.
Why the first design was dropped
The proposal had two stages: a “context gate”, then a LangChain tools agent with Wikipedia, a calculator and custom rule checkers. Testing showed the external tools added up to 60 seconds of latency and many tokens without a matching gain in accuracy, and if-condition checkers wasted what LLMs are good at: reasoning over messy input. The final system is a single structured chain at temperature 0, which is what brought live validation down to about 5 seconds.
Taming 400+ columns
- Readable fields: cryptic column names (such as q_314) are mapped to their Arabic questions from a metadata dictionary.
- Noise removal: regular expressions strip internal GASTAT tags, enumerator instructions and stray punctuation.
- No empty fields: blank and NaN values are dropped before the record is sent, so the model cannot invent relationships across missing data.
From 160 rules to 15 constraints
The 160+ official if-condition rules were generalised into 15 high-level semantic constraints, such as generational gaps, education matching profession, and salary floors, written into the system prompt. Two directives keep the output clean: return an empty list when nothing is wrong, and treat statistical outliers (working 8 hours a week) differently from logical impossibilities.
Scoring
Every issue comes back with the fields involved, a severity, a confidence, and an Arabic description and suggestion, in one of four categories. The report's confidence model subtracts a penalty per issue: Final Confidence = max(0, 1 − Σ penalties), and Total Risk = 1 − Final Confidence.
| Category | Meaning | Penalty |
|---|---|---|
| Direct contradiction | Two claims cannot both be true | 0.50 |
| Context mismatch | A required relationship between fields is missing | 0.25 |
| Temporal inconsistency | Dates, ages or order of events conflict | 0.15 |
| Improbable but possible | Unusual, yet still plausible | 0.05 |
The service
- POST /validate is called as the respondent fills the form; with fewer than two fields it returns immediately without calling the model.
- POST /batch-validate takes a CSV or XLSX file and validates every row, with configurable concurrency to throttle model calls.
- It is stateless: the caller sends the full set of fields every time, so the service scales by simply adding instances.
- A complex batch row takes 20 to 30 seconds on its own, but rows run concurrently with asyncio, so a whole file returns in about 30 seconds.
Choosing the model
| Model | Finding |
|---|---|
| DeepSeek | Cheapest, but weak schema adherence: it wrote its reasoning into the output fields, which inflated risk scores |
| Claude | Average latency and reasoning, and too costly at 400+ columns per record |
| Gemini | A solid baseline without a distinct edge for this schema-bound task |
| OpenAI (used) | Best value: strongest grasp of the rules, strict JSON, and 2 to 4 seconds faster in batch mode |
Result and next steps
3rd place out of 130+ teams at the GASTAT Data Innovation Hackathon, run by the General Authority for Statistics with SDAIA. Proposed next steps: drop-in SDKs for field apps, a small open model fine-tuned on GASTAT rules for offline validation, and a dashboard for editing rules in plain language.