rom.optimal_mu can't always improve accuracy

[jacktang]

Hello there

My model problem is still the 2d mixing tee problem. And I reduced the parameters by fixing inlet1 temperature and inlet2 temperature to 40.0, inlet1 and inlet velocity values are sampled in [0.1, 1.5] uniform distribution.

And I've solved projection errors on training dataset, now the max error around 1e-14. And I created the workflow and hoped to improve rom accuracy automatically. The workflow code list below:

def build_rom():
        mus, snapshots = load_snapshots_with_mus(...)   
        pod = POD('svd', rank=-1)
        gpr = GPR()
        db = Database(mus, snapshots)
        rom = ROM(db, pod, gpr)
        rom.fit()
        return rom

def optimize():
    for i in range(5000):
            rom = build_rom()
            errors = rom.kfold_cv_error(n_splits = 10)
            avg_err = errors.mean()

           if avg_err > 1e-4:
                next_mus = rom.optimal_mu(k=5)
                snapshots = generate_snapshots(next_mus, [40, 40]) # fixed params
                update_mus_snapshots(....)
           else:
                print("You've got a GOOD rom!")
                return

And here are what I found:

1. POD on 600+ snapshots is very slow
2. The value of avg_err is from 0.0030 to 0.0045

Is something wrong in the workflow? Appreciate your time!

[mtezzele]

To speed up the computations you can try using other interpolators such as Linear or RBF. Gaussian process regression does not scale well with the number of samples.

[mtezzele]

Be aware that you are computing the average relative error. So asking it to be below 1e-4 is quite demanding.

[siedan17]

I think an error from your code could be, that you are setting n_splits = 10.

This means that you are always leaving 10 cases out if you have 100 snapshots, or 50 cases if you have 500 snapshots. So I don't know if you can expect that this particular error metric will decrease with more snapshots.

For me it would be more reasonable to always using as many splits as you have snapshots (= loo_error). Or your n_splits parameter should also scale with the number of snapshots you have. E.g. n_splits = 10 with 100 snapshots and n_splits = 50 with 500 snapshots. In this way you ensure that you always leave out groups of 10 cases.

Let me know if this helps.

[jacktang]

@siedan17 the point is the mu values suggested by rom.optimal_mu method did not improve the final rom accuracy. And rom.optimal_mu has no relationship with avg_err indicator. Or I missed something?

[siedan17]

@jacktang you are right that rom.optimal_mu has no relationship with avg_err. However what do you mean with "final rom accuracy"? Is that the last avg_err or something else?

[jacktang]

@siedan17 I supposed the mu suggested by optimal_mu could improve the new trained rom a little bit, and if I've collected a lot of mus and corresponding snapshots, the last rom should be much more accurate than the first one. The avg_err is just the threshold value I wanted to selected the rom I wanted.

[siedan17]

@jacktang so I think that your reasoning "the mu suggested by optioma_mu can improve the overall model" is right and in most cases it will do that.
My point is that your avg_err as it is defined above, makes it hard for the model to improve on this evaluation.
As described above, if you fix n_splits = 10 your model will always partition the number of snapshots you have (500 --> 50; 100 --> 10). So even though your model may be better (500 snapshots) the avg_err is still high (or even higher) because you left out 10% of the whole training data and tried to predict them.

[jacktang]

OK, I got your point! I'll try another error measure later. Thanks a lot!

[jacktang]

@mtezzele Thanks for you kindly suggestions!
I've update the workflow and let it keep running. Meanwhile I think the accuracy of ROM can be improved by more training dataset. I'd like to compare both later.

[ndem0]

Dear @jacktang, also consider a couple of things:

• optimal_mu uses by default loo_error, which means that you are creating and training N different ROMs (where N is the number of snapshots you have). It's very demanding!
• All error measures are actually estimations rather than bounds. It is not said it reduces global error, especially using RBF/GPR/ANN (using Linear I guess you will have a more stable decay of the error).
• I suggest avoiding the rank=-1 in POD (and I suspect it's also the biggest source of error in your code), since it's very prone to overfitting (training error almost 1e-14, but test error much bigger). Use rank=0 to use the automatic selection.

[jacktang]

Thanks for your explanation! @ndem0

• I think I may misunderstand optimal_mu. From the statement These function can be used to achieve the wanted (estimated) accuracy (in tutorial 1) and I thought the ROM could be improve step by step. Training model is cheap but generating snapshots is expensive in my cases. That's why I am interested in above workflow. And can you share the case when optimal_mu should be used?
• I also tried rank=0, the value of avg_err goes up to 0.0045. From my experiences util now, generating proper amount of samples can improve the ROM. and I will investigate the modes in every iteration later.

Thanks again!