Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ChatGLM Model Support #516

Merged
merged 3 commits into from
Jun 11, 2024
Merged

Add ChatGLM Model Support #516

merged 3 commits into from
Jun 11, 2024

Conversation

Qubitium
Copy link
Contributor

@Qubitium Qubitium commented Jun 8, 2024

Add ChatGLM Model loading. Code adapted from vllm main.

Notable Changes:

  1. model_config.get_num_kv_heads() and .get_total_num_kv_heads() ported from vllm to correctly caculate kv heads from model config json. Found that chatglm with sglang main code retrieved wrong kv heads from config causing kv cache to contain the wrong shape.
  2. New EntryClassRemapping property added to model entry definition to help with future compat. Chatglm has ChatGLMModel set in config.json when model loader needs ChatGLMForCausalLM.

Remapping code:

	    # compat: some models such as chatglm has incorrect class set in config.json
            # usage: [ tuple("From_Entry_Class_Name": EntryClass), ]
            if hasattr(module, "EntryClassRemapping") and isinstance(module.EntryClassRemapping, list):
                for remap in module.EntryClassRemapping:
                    if isinstance(remap, tuple) and len(remap) == 2:
                        model_arch_name_to_cls[remap[0]] = remap[1]

Usage:

EntryClass = ChatGLMForCausalLM
# compat: glm model.config class == ChatGLMModel
EntryClassRemapping = [("ChatGLMModel", ChatGLMForCausalLM)]

TESTS:

  • PASSED ChatGLM
  • PASSED Non-ChatGLM: regression test for new get_num_kv_heads
  • PASSED TP=2
  • PASSED DP=2

@Qubitium Qubitium marked this pull request as ready for review June 11, 2024 03:11
@Qubitium Qubitium changed the title WIP: Add ChatGLM Model Support Add ChatGLM Model Support Jun 11, 2024
@merrymercy merrymercy merged commit a8c787d into sgl-project:main Jun 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants