Stacked GBM ensemble
Five gradient-boosting families - LightGBM, XGBoost, CatBoost, HistGradientBoosting, and ExtraTrees - each given its own random search, then the top-K out-of-fold predictions from every family are blended by a meta-learner. Best score on every column of the leaderboard.
Scores
How it works
Each base family is given a random search over a large hyperparameter space (learning rate, tree depth, regularisation, column subsampling, and so on). For every trial the script does a full 5-fold CV on the 441-feature engineered vector and stores the out-of-fold predictions in a CSV. After all trials finish, the top-K configurations per family are refit (so we have an OOF prediction array per top-K model) and stacked with a meta-learner.
The meta-learner is the part that quietly does a lot of work. For gender it is a logistic regression with internal CV over the regularisation strength. For the regression heads it is a non-negative blend that minimises L1 loss. Originally this was a RidgeCV, which silently inflated age MAE from 6.61 to 7.43: the base LightGBM and XGBoost models were trained with an MAE objective, so their OOF predictions are approximately conditional medians, and a least-squares fit pulled the blend toward the conditional mean. Switching to an L1 meta-learner recovered the missing 0.8 MAE.
Configuration
- Feature vector441 engineered features per session (full breakdown)
- Base familiesLightGBM, XGBoost, CatBoost, HistGradientBoosting, ExtraTrees
- Trials (gender)578 LightGBM + 250 XGBoost + 28 CatBoost trials
- Trials (age)40 LightGBM + 30 XGBoost + 8 HistGradientBoosting trials
- Trials (mood)30 LightGBM + 20 XGBoost trials
- Top-K per family5 LGB, 4 XGB, 3 CAT, 3 HGB, 2 ET
- Meta-learnerLogisticRegressionCV (gender) · non-negative L1 minimisation (age, mood)
- CV5-fold StratifiedKFold for gender, 5-fold KFold for age and mood, seed 42