KAN Experiment on an Unknown Function

[fbkoroglu]

Hello everyone,

I have been attempting to use KAN to derive a symbolic "explicit" function for a dataset obtained through finite element analysis results. Despite my efforts, I have not observed any improvement in the train/test losses. I have experimented with various depths and widths, but the results remain unchanged. As I am relatively new to KAN, like many others here, I would greatly appreciate any assistance or guidance on this matter. I am sharing my code and dataset in here as well as one of the models that I trained and its corresponding loss history.

Thank you

seedDataset.xlsx

import matplotlib.pyplot as plt
import pandas as pd
from kan import *
import torch
from sklearn.model_selection import train_test_split

# torch.set_default_tensor_type('torch.cuda.FloatTensor')
# =============================================================================
# LOAD DATA AND CONVERT IT TO TENSOR FIELD
# =============================================================================
dataset = 'seedDataset.xlsx' # Data file name
dataset = pd.read_excel(dataset, sheet_name='Sheet1') # Read data
dataset = dataset.to_numpy()


second_size = 0.20
TrainData, TestData = train_test_split(dataset, test_size=second_size)
TrainData = torch.from_numpy(TrainData)
TestData = torch.from_numpy(TestData)

dataset = {}
dataset['train_input'] = TrainData[:,0:3]
dataset['test_input']  = TestData[:,0:3]
dataset['train_label'] = TrainData[:,3]
dataset['test_label']  = TestData[:,3]

# =============================================================================
# CREATE KAN MODEL
# =============================================================================
model = KAN(width=[3,3,1], grid=3, k=3, seed=0)

# Plot KAN at Initialization
model(dataset['train_input']);
model.plot(beta=100)

# Train the Model
model.train(dataset, opt="LBFGS", steps=50, lamb=0.0, update_grid=False);

# Loop Over Grids
grids = [3,5,10]

train_losses = []
test_losses = []

for i in range(len(grids)):
    model = KAN(width=[3,3,1], grid=grids[i], k=3, seed=0,
                ).initialize_from_another_model(model, dataset['train_input'])
    results = model.train(dataset, opt="LBFGS", steps=50, lamb=0.0, update_grid=False);
    train_losses += results['train_loss']
    test_losses += results['test_loss']

# Visualize the Model
model.plot(beta=100)

mode = "auto"
# mode = "manual"

if mode == "manual":
    # manual mode
    model.fix_symbolic(0,0,0,'sin');
    model.fix_symbolic(0,1,0,'x^2');
    model.fix_symbolic(1,0,0,'exp');
elif mode == "auto":
    # automatic mode
    lib = ['x','x^2','x^3','x^4','exp','log','sqrt','tanh','sin','abs']
    # lib = ['x','x^2','x^3', 'exp','sqrt','abs']
    model.auto_symbolic(lib=lib)

# Get Formulation
model.symbolic_formula(floating_digit=4)[0]

# Plot RMSE History
cm = 0.393701 # cm to inches
plt.figure(999, figsize = (12*cm,10*cm))
plt.plot(train_losses)
plt.plot(test_losses)
plt.legend(['train', 'test'])
plt.ylabel('RMSE')
plt.xlabel('step')
# plt.yscale('log')

[KindXiaoming]

Hi, the shape of label looks a bit suspicious. it should be (N,1) instead of (N,).

dataset['train_label'] = TrainData[:,3]
dataset['test_label']  = TestData[:,3]

to

dataset['train_label'] = TrainData[:,[3]]
dataset['test_label']  = TestData[:,[3]]

[fbkoroglu]

Hi,

I do see a real improvement! Thanks a lot for the help! For the ones who are new in KAN, I am going to share the log-scale improved loss history for them.

As a probable silly question, do I need to normalize the inputs like we do in MLP?

[KindXiaoming]

Glad it works! The grid_update_from_sample function (by default included in train) can handle non-normalized data, so in principle there is no need to normalize your inputs. However, it's still advisable to normalize your inputs because that may help optimization landscape in subtle ways in my experience.

[fbkoroglu]

Thanks a lot for the help! I will continue to experiment on KANs, they are promising!

[BKS00]

Did you find an approximation of the unknown function you were looking for? I would also like to start with KANs, I would like to have your feedback if possible

[fbkoroglu]

Hi @BKS00

Sorry for my late response I've been out of my country because of a conference. Yes, it worked! The algorithm found a formulation for my case.

[BKS00]

Glad to know, is it possible that you can send your results. I find it really interesting.
Have you tested the equation that was obtained, to verify its accuracy? Thanks in advance for the response.

[fbkoroglu]

Hi @BKS00

I used the script and data that I ve shared here, nothing new actually. I ve been able to derive a formulation but the problem is that "seed" argument is not working eventhough I am not prunning the model. Therefore, I am having different formulation from some of the runs. I am gonna do some more work to solve the problem.

[KindXiaoming]

@fbkoroglu Sometimes I find adding this line in the beginning helps reproducibility. Looks like torch LBFGS is using some non-deterministic operations which are non-deterministic even with seeding.

torch.use_deterministic_algorithms(True)

[fbkoroglu]

Hi @KindXiaoming

I tried to use torch.use_deterministic_algorithms(True) however it did not work well in my case. Anyway, I have a bigger problem than this one.

R^2 for the trained model is very well and I tested the KAN model by using Monte Carlo Simulation (MCS) and compared the result that I obtained by using FEM-MCS. They were almost the same indicating that everything was good. However, when I used the derived symbolic function, the result that I got was quite wrong. I evaluated the symbolic function by using the same inputs in the model training dataset to calculate the outputs manually and compared them with FEM results. I realized that there is a difference of up to 10 times and the average is around 3 times.

Do you have an idea to solve this issue? I am adding two plots here to illustrate the situation. By the way, thank you a lot for your suggestions up to now.

@BKS00 you might be also interested in this issue.

[KindXiaoming]

@fbkoroglu how big is your KAN and is there any activation whose symbolic fitting is quite off (low R^2)? Would be great if you can include what's been printed after you call auto_symbolic().

[fbkoroglu]

Dear @KindXiaoming, thank you for your fast response!

My KAN is [3,3,1] it is not that big and I am trying grids for
grids = [3,5,10]

I have the following lines for grid trials and training:

for i in range(len(grids)):
    model = KAN(width=[3,3,1], grid=grids[i], k=3, seed=0,
                ).initialize_from_another_model(model, dataset['train_input'])
    results = model.train(dataset, opt="LBFGS", steps=100, lamb=0.0, update_grid=False)

I am using "default" library for symbolic as lib = ['x','x^2','x^3','x^4','exp','log','sqrt','tanh','sin','abs']

As you can see I tried to construct the KAN as simple as possible. I did no pruning because all of my inputs are needed in the final model. Consequently, I have the following R^2 for symbolic fitting which seems okay:

fixing (0,0,0) with exp, r2=0.99991643005362
fixing (0,0,1) with x^3, r2=0.9999803834577455
fixing (0,0,2) with x^2, r2=0.9999384586149089
fixing (0,1,0) with x^4, r2=0.9999782011737739
fixing (0,1,1) with x^3, r2=0.9999919262912189
fixing (0,1,2) with x^3, r2=0.9999248052283842
fixing (0,2,0) with abs, r2=0.9999999997727396
fixing (0,2,1) with abs, r2=0.9999999997815957
fixing (0,2,2) with abs, r2=0.9999999997385964
fixing (1,0,0) with sin, r2=0.9971181178166976
fixing (1,1,0) with sin, r2=0.9990668493482523
fixing (1,2,0) with sin, r2=0.9993178252452747

[KindXiaoming]

After auto_symbolic, did you do further training? There are a few affine transformations to be fixed via further training. What's the loss before auto_symbolic, right after auto_symbolic, and after further training?

[fbkoroglu]

Hi @KindXiaoming

I needed to update KAN version since I missed out the major improvements when I was abroad. However, now I managed to come up with a solid symbolic function based on your suggestions. Thank you for your advices! I am very happy with the losses and estimation capacity of KANs as well as the illustrations!

I have two final questions before closing this issue.

The first one, sometimes I am coming across in cases where one of my inputs is diminishing. However, by the engineering judgement I know that the input must appear in the final equation. As far as I understand from the other issues, "lock" is no more supported. So, what should I do for such cases?

The second question is again about reproducebility. I just read @issue318 and checked hellokan.ipynb but I could not see any difference. The line you mentioning in that issue is torch.use_deterministic_algorithms(True) or something else?

Thanks in advance!

[fbkoroglu]

Hello everyone,

I solved the reproducibility issue and I realized that it was my own mistake. I just forgot to put a seed to train/test sets split. Therefore, now I am able to get reproducible results.

Since my first question covers a rare case, I think it is not that important. Therefore, I am closing this issue.

Thank you so much for your help @KindXiaoming