My experience with tabular data textbooks is that they introduce many different techniques, in a relatively un-opinionated way, which doesn't help give you intuitions on what your strategy for real-world problems should be. In practice, almost all tabular data problems I've encountered boil down to:
1. Make sure you don't have any impossible nonsense or data leakage in your features or target
2. Split your data in an intelligent way (temporally for time-series, group-wise for hierarchical data)
3. Try a really simple linear regression / logistic regression model to get a "dumb" baseline for your accuracy/error metric and make sure it is reasonable
4. If you need interpretability, consider a GAM, else, throw it into XGBoost and you'll get state of the art results
I am writing a tabular data textbook for O'Reilly
- on building ML systems with a feature store.
I try to be opinionated about modelling - XGBoost is all you really need, but the challenges are more like you say - how to prevent data leakage (ASOF LEFT JOIN (or use a feature store)), separating model-independent data transformations from model-specific data transformations, APIs for things like time-series splits, logging, monitoring, building and operating the pipelines. All pretty standard software engineering in Python nowadays.
Personally I am very interested on building big data pipeline for machine learning with initially batch and then real-time data of ECG and seismic, for CVDs screening/early detection and earthquakes early detection/prediction respectively. Any idea when the completed book will be available?
Just wondering what is the main difference between your book and this book, Architecting Data and Machine Learning Platforms also from O'Reilly:
For 4, I would always start with RF. You have fewer knobs to turn (aka degrees of freedom with which to hang yourself), but still get within spitting distance of XGBoost.
Its far from a principled statistical approach to what is supposed to be at least an observational study in every case —- and have all the caveats and preparations thereof —- but it is very easy to explain to devs, and one cannot stop the tide unfortunately.
I'd replace XGBoost with either LightGBM or CatBoost. Though all three are capable of producing excellent results, and XGBoost has the advantage of not being maintained by an organization you might disagree with.
What do you think about the mechanics of machine learning book by Jeremy Howard? It seems to have a systematic approach to fit models and improve them.
At risk of being excessively cheeky, I think you should throw it into xgboost and not worry about it.
I love Jeremy Howard, but if you're doing applied ML on tabular data, tweaking specifics of model construction probably won't get you anything beyond stock xgboost. The concerns highlighted by your sibling commenter jamesblonde end up eating most of your time in deployed ML products, not model construction.
If you're looking to learn about ML model construction I'm sure it's a great book.
Like others say, usually just go for XGBoost, which is increasingly even shown (proven?) in literature, eg:
https://arxiv.org/abs/2106.03253
You can also start with scikit-learn's `LogisticRegressionCV`, if a linear model is more palatable to your audience.
The bigger challenge is to reliably estimate how good your model is.
It's not about getting the best performance – it's about getting the "real" performance. How will your model _really_ do on future unseen datasets?
The answer to this challenge is cross-validation. But what are the questions?
There are 2 very different questions for which the answer is cross-validation.
One is, which hyper parameters to use with your model?
The second is, what is the generalization performance of the model?
This requires 2 separate applications (loops) of cross-validation. The authors of this book talk about this in terms of having a "validation set" and a "test set" (Sometimes, these terms are switched around, and there is also "holdout set". It's critical to know how you, and the rest of your team, are using these terms in your modeling)
A robust way to implement these 2 CV's is with nested cross-validation, readily available in many packages – and also, should be "fast enough" using modern computers.
One exercise that remains is: with nested CV, which model do you pick as your "production" model?
> One exercise that remains is: with nested CV, which model do you pick as your "production" model?
Why even pick one, just make an ensemble out of all of them. Unless you need your model to remain blind to one test or validation set for some reason, but then you should probably choose at random anyways to avoid bias.
Of course this being HN somebody is going to mention some case where they only have the CPU or memory budget for one model. But in general making ensembles of multiple XGBoost models is a great way to get better model performance.
I appreciate the section on independent component analysis (ICA). It's not well known and very underused. In my experience, it usually works great on heterogeneous tabular data - which PCA usually does poorly on.
1. Make sure you don't have any impossible nonsense or data leakage in your features or target
2. Split your data in an intelligent way (temporally for time-series, group-wise for hierarchical data)
3. Try a really simple linear regression / logistic regression model to get a "dumb" baseline for your accuracy/error metric and make sure it is reasonable
4. If you need interpretability, consider a GAM, else, throw it into XGBoost and you'll get state of the art results
reply