Merging models when using different parameters for modules

[Ceglowa]

In the example shown here: https://maartengr.github.io/BERTopic/getting_started/merge/merge.html#example, 3 models are being merged together using the BERTopic.merge_models method.

Each of the models uses the same UMAP model. I have a few questions connected to this:

1. If the instance of the UMAP model is initialized once, does this mean that for each of the three fit calls, this UMAP instance is being fitted each time from scratch? Or is the UMAP model being fitted with new data on the same instance?
2. What would happen if each BERTopic model had a different UMAP model with different parameters? For example, topic_model_1 has a UMAP model with n_neighbors=15, topic_model_2 has a UMAP model with n_neighbors=30 and topic_model_3 has a UMAP model with n_neighbors=45. From what I tested, it seems that the code will be working like this. However, is this wrong conceptually?
3. And also the same question for the clustering model. Here, I am asking about the HDBSCAN model. If each model has different parameters for the HDBSCAN model, then is that correct approach?

[MaartenGr]

If the instance of the UMAP model is initialized once, does this mean that for each of the three fit calls, this UMAP instance is being fitted each time from scratch? Or is the UMAP model being fitted with new data on the same instance?

Nope, UMAP is not merged at all and is completely removed from the merged model. Both HDBSCAN and UMAP are removed and only the topic representations themselves are being saved. The result is a similar representation as when you save a model with .safetensors.

What would happen if each BERTopic model had a different UMAP model with different parameters? For example, topic_model_1 has a UMAP model with n_neighbors=15, topic_model_2 has a UMAP model with n_neighbors=30 and topic_model_3 has a UMAP model with n_neighbors=45. From what I tested, it seems that the code will be working like this. However, is this wrong conceptually?

This is not wrong conceptually since the fitted UMAP models are not merged but ignored and removed from the final topic model.

And also the same question for the clustering model. Here, I am asking about the HDBSCAN model. If each model has different parameters for the HDBSCAN model, then is that correct approach?

Same as above.

[Ceglowa]

This is not wrong conceptually since the fitted UMAP models are not merged but ignored and removed from the final topic model.

I am thinking about the process of the training. Let's assume we have two BERTopic models, each with its individual UMAP model. And also let's assume that we are training on the same set of data. Through the training process, each of the models is being fitted. Isn't it the case that the reduced embeddings of the documents in the first model and the second model can look different? The UMAP will behave differently so that means that the reduced embeddings can also be different, right?

If what I am writing is true, then that would also apply to the HDBSCAN model. I understand that the HDBSCAN and UMAP models are removed in the merged model. I am just curious if the fact that the training model will look differently for two models would affect the embeddings of the topics that are then being used in the merging process.

And I have an additional question about the lack of HDBSCAN and UMAP models in the merged model (and also the lack of them when you save a model with .safetensors). I checked the code and I see that dimensionality reduction doesn't happen at all and topic assignment is done through calculating the cosine similarity. Doesn't it mean that the predictions when using transform will be of worse quality?

[MaartenGr]

If what I am writing is true, then that would also apply to the HDBSCAN model. I understand that the HDBSCAN and UMAP models are removed in the merged model. I am just curious if the fact that the training model will look differently for two models would affect the embeddings of the topics that are then being used in the merging process.

The reduced embeddings are not used in the merged model, only the original embeddings. This means that there is no effect here with respect to different parameters of HDBSCAN/UMAP aside from the topic representations (the actual labels/keywords) that are being generated.

And I have an additional question about the lack of HDBSCAN and UMAP models in the merged model (and also the lack of them when you save a model with .safetensors). I checked the code and I see that dimensionality reduction doesn't happen at all and topic assignment is done through calculating the cosine similarity. Doesn't it mean that the predictions when using transform will be of worse quality?

I don't believe so since the original embeddings hold more information than the reduced embeddings. They are reduced as a way to more easily handle clustering in high dimensional space.

[Ceglowa]

Great, thanks a lot @MaartenGr. I understand better how the prediction works in BERTopic. By the way, I wanted to say that I appreciate a lot your response time on this repository. It is great that you are so willing to help people here.

[MaartenGr]

Thank you for your kind words! Glad I could be of help 😄