Use mmap for external memory.

[trivialfis]

On a system with 128GB memory, I am able to train a model with hist on a dataset of about 290GB.

*Windows support is removed.

• Remove the temporary page in GPU hist, eliminating unnecessary batch iteration.
• Improve documentation.

Performance is bounded by IO, and it's unlikely to change for the foreseeable future. I ran the test on a PICE-4 NVME drive. Observing the run from htop/iotop, the disk read is relatively consistent. For fetching gradient index, it's about 3G/s throughout the training.

There are still many limiting factors in scaling with storage since we only batch the predictor. But this PR will make the external memory a bit more practical.

I will do some more experiments in coming days. At the moment, using mmap can help (not in this branch yet) reusing the Linux cache. Allocating large chunks of memory is extremely expensive operation when the memory is already under pressure.

The next step after this PR is to make the DMatrix pages' size immutable after construction. This way, we can reuse the pointer from mmap in structures like SparsePage. The end goal is to make sure XGBoost can use the Linux cache efficiently.

[trivialfis]

This initiates an iteration on the sparse DMatrix but doesn't finish it. As a result, we run sketching twice before the PR.

[trivialfis]

There are a couple of places where we can eliminate batch fetching, but I will leave that as future optimization:

xgboost/src/gbm/gbtree.cc

Line 638 in e70810b

auto is_ellpack = f_dmat && f_dmat->PageExists<EllpackPage>() &&

xgboost/src/data/sparse_page_dmatrix.cc

Line 170 in e70810b

this->InitializeSparsePage(ctx);

[trivialfis]

@rongou Would you like to take a look when you are available? I have some experiments in mind for external memory; this PR is the first step. I will follow up with some of them after 2.0:

• Eliminate all unnecessary calls to DMatrix::GetBatches, which initiates a fetch sequence.
• Try to support GPUDirect. Not sure if this is necessary though. The Linux kernel usually does an excellent job of caching data on the host memory.
• Eliminate some page allocations by swapping in mapped ptr.
• Look into scaling it with distributed training.

Maybe there are other types of memory reduction algorithms, and I would also like to learn more about them.

[RAMitchell]

Very nice docs as usual.