Transformer encoder
A small transformer encoder reading the 21-step per-question session as a sequence. In theory the most flexible architecture on the board. In practice it loses to both the BiGRU and the LSTM on gender - probably because 6,710 sessions is below the regime where transformers start to pay off.
Scores
How it works
Each session becomes a sequence of 21 steps - 16 round-1 elimination questions, 4 round-2 questions, and 1 final pick. Each step is a 40-dim vector containing the four offered colors (RGB + HSL), the chosen color, a one-hot position indicator, the per-step time delta, and a step-type indicator. A learned [CLS] token is prepended; the encoder output at that position becomes the session summary. 14 side features (total time, hour of day, mean question time, etc.) are concatenated to the [CLS] output before the three heads.
Trained jointly with a multi-task loss (binary cross-entropy on gender,
scaled MSE on age and mood). Best snapshot picked on validation gender BCE.
Reads raw save.ligma and re-parses it directly so it does not
depend on the engineered feature vector.
Configuration
- d_model64
- Encoder layers3
- Attention heads4
- FFN size128
- Dropout0.2
- Session tokenlearned [CLS]
- Side features14, fused after the encoder
- OptimiserAdamW, warmup + cosine decay
- CV5-fold StratifiedKFold on gender, seed 42