Custom dataset

[zerzerzerz]

Thank you for your wonderful work and I would like to know whether I can use this model or train a model from scratch to imputate my time series?

[WenjieDu]

Hi there,

Thank you so much for your attention to SAITS! If you find SAITS is helpful to your work, please star⭐️ this repository. Your star is your recognition, which can let others notice SAITS. It matters and is definitely a kind of contribution.

I have received your message and will respond ASAP. Thank you again for your patience! 😃

Best,
Wenjie

[WenjieDu]

Hi @zerzerzerz , your recognition is much appreciated!

Yes, absolutely, you can train SAITS on your own dataset to impute its missing values. SAITS is a general time-series imputation model and does not impose specific assumptions on the data itself. So give it a try, please. If you have any questions or need any help, please feel free to contact me. I always try my best to help.

[zerzerzerz]

Hi @zerzerzerz , your recognition is much appreciated!

Yes, absolutely, you can train SAITS on your own dataset to impute its missing values. SAITS is a general time-series imputation model and does not impose specific assumptions on the data itself. So give it a try, please. If you have any questions or need any help, please feel free to contact me. I always try my best to help.

I have some time series whose shape is [T, D], where T is the length of each series and D is the number of features and D = 1. I would like to know is this format is OK? I guess I should modify codes you provided in An example of training SAITS for imputing dataset PhysioNet-2012 is shown below. Is that right? Besides, is there any other requirement of the time series? Thank you for your reply~

[WenjieDu]

I think your data is fine, but considering it has only one feature, I would like to know if your dataset is big enough to train a NN for the imputation task? If its size is small, you can try some simple and intuitive imputation method, like last observation carried forward (LOCF) or linear interpolation, which may also work well.

For your question about applying SAITS with PyPOTS, the answer is yes. With PyPOTS, you only need to care about the preprocessing of your own data. Code after your modification should be like the below:

# Install PyPOTS first: pip install pypots 
from pypots.data import mcar, fill_nan_with_mask
from pypots.imputation import SAITS
from pypots.utils.metrics import cal_mae

# ❗️👀 Preprocess your data here to generate X, which should be a NumPy array
# X.shape should be [n_samples, T, D]

# hold out some observed values as ground truth
X_intact, X, missing_mask, indicating_mask = mcar(X, 0.1)
X = fill_nan_with_mask(X, missing_mask)

saits_base = SAITS(n_layers=2, d_model=256, d_inner=128, n_head=4, d_k=64, d_v=64, dropout=0.1, epochs=100)
saits_base.fit(X) # here use the whole set. You can also split X into train/val/test sets.
imputation = saits_base.impute(X)
mae = cal_mae(imputation, X_intact, indicating_mask)  # calculate mean absolute error on imputation

LOCF mentioned above is also available in PyPOTS. Try

from pypots.imputation import LOCF
imputed_X=LOCF().impute(X)

For more information, please refer to PyPOTS homepage. Please note that PyPOTS is still under development and will integrate with more and more powerful algorithms for incomplete time series, and make them easy to use. If you like this handy tool, please give PyPOTS a star and recommend it to your colleagues in need. Any kind of your contribution to PyPOTS will help its growth and is much appreciated. Thank you!

[zerzerzerz]

Thank you for your detailed instruction, and I will recommend PyPOTS to others!