Question about imputation

[How-Will]

Hi!

In the example you provided, the following code is used to impute the originally-missing values and artificially-missing values

imputation = saits.impute(X)

After I converted imputation to DataFrame, I compared it with the original data and found that there was a big difference. Did I make a mistake? What should I do?

I used the following code to compare the original data and imputation ：

data['X'].head(5)

pd.DataFrame(imputation.reshape(-1, 37)).head(5)

[WenjieDu]

Hi there 👋,

Thank you so much for your attention to PyPOTS! If you find PyPOTS helpful to your work, please star⭐️ this repository. Your star is your recognition, which can help more people notice PyPOTS and grow PyPOTS community. It matters and is definitely a kind of contribution to the community.

I have received your message and will respond ASAP. Thank you for your patience! 😃

Best,
Wenjie

[WenjieDu]

Hi, to make the gradient descend smoothly during training, the code normalizes the dataset with standard scaling. Therefore, after the model imputation, to get the imputed dataset in the original data space, you have to inverse the scaling. First, you have to keep the scaler, so you should

scaler = StandardScaler()
X = scaler.fit_transform(X.to_numpy())

To inverse the transformation, you can

X = X.reshape(num_samples*48,-1)
X = scaler.inverse_transform(X)

[How-Will]

Thank you very much, your reply help me a lot👍

[WenjieDu]

Ｍy pleasure! If you like SAITS and PyPOTS, please star ⭐️ them to let more people notice these useful tools. Thank you! 👍

[How-Will]

Your projects are very useful, I will recommand they to others. ❤️

In the SAITS project, how should I perform hyperparameter search? Should I use the following command to do so?

python run_models.py --config_path configs/AirQuality_SAITS_base.ini --param_searching_mode

Then, how can I check the hyperparameters obtained by the search?

Looking forward to your reply💕

[WenjieDu]

To run the hyperparameter search in the SAITS repo, you have to learn some basic things about NNI (Neural Network Intelligence) developed by Microsoft. You can read their docs to learn about it. After that, in the SAITS project, you should enter NNI_tuning/SAITS the NNI tuning config dir for SAITS, then run the command nnictl create --config SAITS_searching_config.yml to start the tuning experiment. Give it a try. If you have other questions, please raise an issue in the SAITS project. Thanks!

[WenjieDu]

Hi William, please close this issue if your questions have been solved. 😊

[How-Will]

Thank you, your suggestion is very helpful to me.

PyPots is so good 👍👍👍👍

[WenjieDu]

Thank you for your appreciation! You're welcome to join PyPOTS community if you have any comments, suggestions, or ideas to contribute, William!