Avoid calling set device during destruction.

[trivialfis]

Close #7153 .

[codecov-commenter]

Codecov Report

Merging #7154 (5bec53f) into master (7a1d67f) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #7154   +/-   ##
=======================================
  Coverage   82.67%   82.67%           
=======================================
  Files          13       13           
  Lines        4017     4017           
=======================================
  Hits         3321     3321           
  Misses        696      696           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7a1d67f...5bec53f. Read the comment docs.