Refactor tensor data

[laggui]

Checklist

• Confirmed that run-checks all script has been executed.
• Made sure the book is up to date with changes in this PR.
• Confirmed that examples are up to date / can be executed.

Changes

This is a big refactor to move away from the Data<E, D> and DataSerialize<E> structures, which are marked as deprecated.

It introduces the new data format TensorData, which stores the elements as bytes to preserve the flexibility to switch between data types.

Breaking ⚠️

The new serialization format is not compatible with the previous format, but we can easily load records saved in previous versions with the record-backward-compat feature flag (except for binary formats, but this is expected). So any other self-describing format that we support can be loaded and saved into the new format easily.

Important Bits

This PR impacts a lot of the codebase for minor stuff, but make sure you take a look at the most important bits.

New TensorData type & implementation
burn-tensor/src/tensor/data.rs
burn-tensor/src/tensor/element/base.rs

De/serialization with backward compat
burn-core/src/record/tensor.rs

De/serialization of NestedValue for new u8 type
burn-core/src/record/serde/data.rs
burn-core/src/record/serde/de.rs
burn-core/src/record/serde/ser.rs

from_data implementation
burn-candle/src/tensor.rs
burn-jit/src/ops/base.rs
burn-ndarray/src/tensor.rs
burn-tch/src/tensor.rs

Deprecations
burn-tensor/src/libs.rs (allow deprecated usage internally)
burn-tensor/src/tensor/data.rs (Data and DataSerialize deprecation message)

Testing

All tests passed.

[codecov]

Codecov Report

Attention: Patch coverage is 95.43513% with 171 lines in your changes missing coverage. Please review.

Project coverage is 84.72%. Comparing base (2fbc462) to head (21de95e).
Report is 1 commits behind head on main.

Files	Patch %	Lines
crates/burn-tensor/src/tensor/data.rs	74.52%	81 Missing ⚠️
crates/burn-tensor/src/tests/ops/aggregation.rs	80.00%	18 Missing ⚠️
crates/burn-core/src/record/serde/data.rs	38.46%	8 Missing ⚠️
crates/burn-import/src/onnx/to_burn.rs	72.41%	8 Missing ⚠️
crates/burn-tensor/src/tests/ops/iter_dim.rs	80.95%	8 Missing ⚠️
crates/burn-core/src/record/tensor.rs	65.00%	7 Missing ⚠️
crates/burn-tensor/src/tests/ops/chunk.rs	70.83%	7 Missing ⚠️
crates/burn-core/src/record/serde/de.rs	87.17%	5 Missing ⚠️
crates/burn-tensor/src/tensor/api/check.rs	58.33%	5 Missing ⚠️
crates/burn-tensor/src/tensor/element/base.rs	0.00%	3 Missing ⚠️
... and 16 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1916      +/-   ##
==========================================
- Coverage   85.02%   84.72%   -0.31%     
==========================================
  Files         790      791       +1     
  Lines       93275    93591     +316     
==========================================
- Hits        79310    79291      -19     
- Misses      13965    14300     +335     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[laggui]

One of the drawbacks of this change is that creating a tensor from data would previously give a compile error for something like the following:

let tensor = Tensor::<B, 1>::from_floats([[1.0], [2.0]], &device)

error[E0277]: the trait bound `Data<f32, 1>: From<[[{float}; 1]; 2]>` is not satisfied

But now it gives a runtime error instead since we removed the const generic D from TensorData.

thread 'main' panicked at /home/laggui/workspace/burn/crates/burn-tensor/src/tensor/shape.rs:52:13:
index out of bounds: the len is 1 but the index is 1
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Also, this

let tensor = Tensor::<B, 2>::from_floats([1.0, 2.0], &device)

wouldn't work previously (similar compilation error) but now the data is coerced into shape [2, 1]. Bug or feature? 😅

We could discuss how we want to handle this.

[nathanielsimard]

thread 'main' panicked at /home/laggui/workspace/burn/crates/burn-tensor/src/tensor/shape.rs:52:13:
index out of bounds: the len is 1 but the index is 1
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

We could discuss how we want to handle this.

I think we should create a custom parnic message that indicates a rank mismatch.

[nathanielsimard]

I have some general comments, but thanks for the huge refactor!

[laggui]

I think we should create a custom parnic message that indicates a rank mismatch.

With the latest changes it would look like this:

thread 'main' panicked at /home/laggui/workspace/burn/crates/burn-tensor/src/tensor/api/base.rs:708:9:
=== Tensor Operation Error ===
  Operation: 'From Data'
  Reason:
    1. Given dimensions differ from the tensor rank. Tensor rank: '1', given dimension: '[2, 1]'.

note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

[laggui]

@nathanielsimard one thing we missed is documentation. Where should we document these?

1. Deprecation of Data in favor of the new TensorData
2. Backward compatible record loading with the record-backward-compat feature flag

In the book? And also README? Just so it's visible to users and they know what to do for the transition.