ggml : fix YARN + add tests + add asserts #7617

ggerganov · 2024-05-29T11:33:26Z

There might be some issues with RoPE, will try to address here

ggerganov/ggml#839
#4093
#7519 (comment)

github-actions · 2024-05-29T14:18:08Z

📈 llama.cpp server for bench-server-baseline on Standard_NC4as_T4_v3 for phi-2-q4_0: 536 iterations 🚀

Expand details for performance related PR only

Concurrent users: 8, duration: 10m
HTTP request : avg=8743.5ms p(95)=21549.19ms fails=, finish reason: stop=476 truncated=60
Prompt processing (pp): avg=104.95tk/s p(95)=516.49tk/s
Token generation (tg): avg=33.81tk/s p(95)=46.93tk/s
ggml-org/models/phi-2/ggml-model-q4_0.gguf parallel=8 ctx-size=16384 ngl=33 batch-size=2048 ubatch-size=256 pp=1024 pp+tg=2048 branch=gg/rope-tests commit=aa15e7d246ebf1de30148f29805111714b3cfa35

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 536 iterations"
    y-axis "llamacpp:prompt_tokens_seconds"
    x-axis "llamacpp:prompt_tokens_seconds" 1717000522 --> 1717001150
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 416.17, 416.17, 416.17, 416.17, 416.17, 982.09, 982.09, 982.09, 982.09, 982.09, 955.89, 955.89, 955.89, 955.89, 955.89, 958.24, 958.24, 958.24, 958.24, 958.24, 999.79, 999.79, 999.79, 999.79, 999.79, 1003.31, 1003.31, 1003.31, 1003.31, 1003.31, 998.87, 998.87, 998.87, 998.87, 998.87, 1012.07, 1012.07, 1012.07, 1012.07, 1012.07, 996.55, 996.55, 996.55, 996.55, 996.55, 1006.4, 1006.4, 1006.4, 1006.4, 1006.4, 1021.61, 1021.61, 1021.61, 1021.61, 1021.61, 982.66, 982.66, 982.66, 982.66, 982.66, 960.88, 960.88, 960.88, 960.88, 960.88, 970.38, 970.38, 970.38, 970.38, 970.38, 971.74, 971.74, 971.74, 971.74, 971.74, 967.5, 967.5, 967.5, 967.5, 967.5, 968.15, 968.15, 968.15, 968.15, 968.15, 962.84, 962.84, 962.84, 962.84, 962.84, 976.62, 976.62, 976.62, 976.62, 976.62, 975.89, 975.89, 975.89, 975.89, 975.89, 980.13, 980.13, 980.13, 980.13, 980.13, 976.99, 976.99, 976.99, 976.99, 976.99, 978.53, 978.53, 978.53, 978.53, 978.53, 983.1, 983.1, 983.1, 983.1, 983.1, 983.75, 983.75, 983.75, 983.75, 983.75, 983.04, 983.04, 983.04, 983.04, 983.04, 960.93, 960.93, 960.93, 960.93, 960.93, 957.15, 957.15, 957.15, 957.15, 957.15, 953.69, 953.69, 953.69, 953.69, 953.69, 952.38, 952.38, 952.38, 952.38, 952.38, 954.74, 954.74, 954.74, 954.74, 954.74, 952.99, 952.99, 952.99, 952.99, 952.99, 948.72, 948.72, 948.72, 948.72, 948.72, 950.7, 950.7, 950.7, 950.7, 950.7, 944.28, 944.28, 944.28, 944.28, 944.28, 945.8, 945.8, 945.8, 945.8, 945.8, 946.08, 946.08, 946.08, 946.08, 946.08, 943.84, 943.84, 943.84, 943.84, 943.84, 940.77, 940.77, 940.77, 940.77, 940.77, 937.42, 937.42, 937.42, 937.42, 937.42, 938.91, 938.91, 938.91, 938.91, 938.91, 937.4, 937.4, 937.4, 937.4, 937.4, 936.86, 936.86, 936.86, 936.86, 936.86, 932.23, 932.23, 932.23, 932.23, 932.23, 927.46, 927.46, 927.46, 927.46, 927.46, 925.04, 925.04, 925.04, 925.04, 925.04, 922.51, 922.51, 922.51, 922.51, 922.51, 920.42, 920.42, 920.42, 920.42, 920.42, 924.08, 924.08, 924.08, 924.08, 924.08, 922.45, 922.45, 922.45, 922.45, 922.45, 927.27, 927.27, 927.27, 927.27, 927.27, 925.98, 925.98, 925.98, 925.98, 925.98, 927.82, 927.82, 927.82, 927.82, 927.82, 905.92, 905.92, 905.92, 905.92, 905.92, 904.76, 904.76, 904.76, 904.76, 904.76, 903.43, 903.43, 903.43, 903.43, 903.43, 901.51, 901.51, 901.51, 901.51, 901.51, 900.92, 900.92, 900.92, 900.92, 900.92, 901.28, 901.28, 901.28, 901.28, 901.28, 900.21, 900.21, 900.21, 900.21, 900.21, 900.75, 900.75, 900.75, 900.75, 900.75]

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 536 iterations"
    y-axis "llamacpp:predicted_tokens_seconds"
    x-axis "llamacpp:predicted_tokens_seconds" 1717000522 --> 1717001150
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 42.4, 42.4, 42.4, 42.4, 42.4, 29.41, 29.41, 29.41, 29.41, 29.41, 27.18, 27.18, 27.18, 27.18, 27.18, 29.64, 29.64, 29.64, 29.64, 29.64, 29.79, 29.79, 29.79, 29.79, 29.79, 31.38, 31.38, 31.38, 31.38, 31.38, 33.16, 33.16, 33.16, 33.16, 33.16, 34.0, 34.0, 34.0, 34.0, 34.0, 34.22, 34.22, 34.22, 34.22, 34.22, 34.16, 34.16, 34.16, 34.16, 34.16, 34.13, 34.13, 34.13, 34.13, 34.13, 34.33, 34.33, 34.33, 34.33, 34.33, 33.97, 33.97, 33.97, 33.97, 33.97, 33.63, 33.63, 33.63, 33.63, 33.63, 31.94, 31.94, 31.94, 31.94, 31.94, 29.61, 29.61, 29.61, 29.61, 29.61, 29.37, 29.37, 29.37, 29.37, 29.37, 29.47, 29.47, 29.47, 29.47, 29.47, 29.58, 29.58, 29.58, 29.58, 29.58, 29.4, 29.4, 29.4, 29.4, 29.4, 29.52, 29.52, 29.52, 29.52, 29.52, 29.74, 29.74, 29.74, 29.74, 29.74, 30.14, 30.14, 30.14, 30.14, 30.14, 30.02, 30.02, 30.02, 30.02, 30.02, 30.14, 30.14, 30.14, 30.14, 30.14, 30.27, 30.27, 30.27, 30.27, 30.27, 30.33, 30.33, 30.33, 30.33, 30.33, 30.24, 30.24, 30.24, 30.24, 30.24, 30.18, 30.18, 30.18, 30.18, 30.18, 30.45, 30.45, 30.45, 30.45, 30.45, 30.51, 30.51, 30.51, 30.51, 30.51, 30.53, 30.53, 30.53, 30.53, 30.53, 30.58, 30.58, 30.58, 30.58, 30.58, 30.71, 30.71, 30.71, 30.71, 30.71, 30.72, 30.72, 30.72, 30.72, 30.72, 30.66, 30.66, 30.66, 30.66, 30.66, 30.19, 30.19, 30.19, 30.19, 30.19, 29.84, 29.84, 29.84, 29.84, 29.84, 29.8, 29.8, 29.8, 29.8, 29.8, 29.68, 29.68, 29.68, 29.68, 29.68, 29.84, 29.84, 29.84, 29.84, 29.84, 29.96, 29.96, 29.96, 29.96, 29.96, 30.05, 30.05, 30.05, 30.05, 30.05, 30.03, 30.03, 30.03, 30.03, 30.03, 29.7, 29.7, 29.7, 29.7, 29.7, 29.31, 29.31, 29.31, 29.31, 29.31, 28.79, 28.79, 28.79, 28.79, 28.79, 28.31, 28.31, 28.31, 28.31, 28.31, 28.23, 28.23, 28.23, 28.23, 28.23, 28.32, 28.32, 28.32, 28.32, 28.32, 28.37, 28.37, 28.37, 28.37, 28.37, 28.41, 28.41, 28.41, 28.41, 28.41, 28.46, 28.46, 28.46, 28.46, 28.46, 28.5, 28.5, 28.5, 28.5, 28.5, 28.5, 28.5, 28.5, 28.5, 28.5, 28.55, 28.55, 28.55, 28.55, 28.55, 28.55, 28.55, 28.55, 28.55, 28.55, 28.69, 28.69, 28.69, 28.69, 28.69, 28.8, 28.8, 28.8, 28.8, 28.8, 28.92, 28.92, 28.92, 28.92, 28.92, 28.98, 28.98, 28.98, 28.98, 28.98]

Details

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 536 iterations"
    y-axis "llamacpp:kv_cache_usage_ratio"
    x-axis "llamacpp:kv_cache_usage_ratio" 1717000522 --> 1717001150
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.12, 0.12, 0.12, 0.12, 0.12, 0.4, 0.4, 0.4, 0.4, 0.4, 0.15, 0.15, 0.15, 0.15, 0.15, 0.1, 0.1, 0.1, 0.1, 0.1, 0.19, 0.19, 0.19, 0.19, 0.19, 0.1, 0.1, 0.1, 0.1, 0.1, 0.11, 0.11, 0.11, 0.11, 0.11, 0.18, 0.18, 0.18, 0.18, 0.18, 0.16, 0.16, 0.16, 0.16, 0.16, 0.13, 0.13, 0.13, 0.13, 0.13, 0.2, 0.2, 0.2, 0.2, 0.2, 0.25, 0.25, 0.25, 0.25, 0.25, 0.3, 0.3, 0.3, 0.3, 0.3, 0.41, 0.41, 0.41, 0.41, 0.41, 0.41, 0.41, 0.41, 0.41, 0.41, 0.27, 0.27, 0.27, 0.27, 0.27, 0.28, 0.28, 0.28, 0.28, 0.28, 0.19, 0.19, 0.19, 0.19, 0.19, 0.31, 0.31, 0.31, 0.31, 0.31, 0.13, 0.13, 0.13, 0.13, 0.13, 0.2, 0.2, 0.2, 0.2, 0.2, 0.12, 0.12, 0.12, 0.12, 0.12, 0.13, 0.13, 0.13, 0.13, 0.13, 0.08, 0.08, 0.08, 0.08, 0.08, 0.14, 0.14, 0.14, 0.14, 0.14, 0.16, 0.16, 0.16, 0.16, 0.16, 0.26, 0.26, 0.26, 0.26, 0.26, 0.2, 0.2, 0.2, 0.2, 0.2, 0.13, 0.13, 0.13, 0.13, 0.13, 0.18, 0.18, 0.18, 0.18, 0.18, 0.16, 0.16, 0.16, 0.16, 0.16, 0.21, 0.21, 0.21, 0.21, 0.21, 0.17, 0.17, 0.17, 0.17, 0.17, 0.15, 0.15, 0.15, 0.15, 0.15, 0.27, 0.27, 0.27, 0.27, 0.27, 0.35, 0.35, 0.35, 0.35, 0.35, 0.41, 0.41, 0.41, 0.41, 0.41, 0.35, 0.35, 0.35, 0.35, 0.35, 0.29, 0.29, 0.29, 0.29, 0.29, 0.14, 0.14, 0.14, 0.14, 0.14, 0.12, 0.12, 0.12, 0.12, 0.12, 0.11, 0.11, 0.11, 0.11, 0.11, 0.13, 0.13, 0.13, 0.13, 0.13, 0.53, 0.53, 0.53, 0.53, 0.53, 0.63, 0.63, 0.63, 0.63, 0.63, 0.55, 0.55, 0.55, 0.55, 0.55, 0.43, 0.43, 0.43, 0.43, 0.43, 0.17, 0.17, 0.17, 0.17, 0.17, 0.23, 0.23, 0.23, 0.23, 0.23, 0.2, 0.2, 0.2, 0.2, 0.2, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.23, 0.23, 0.23, 0.23, 0.23, 0.26, 0.26, 0.26, 0.26, 0.26, 0.14, 0.14, 0.14, 0.14, 0.14, 0.24, 0.24, 0.24, 0.24, 0.24, 0.14, 0.14, 0.14, 0.14, 0.14, 0.11, 0.11, 0.11, 0.11, 0.11, 0.18, 0.18, 0.18, 0.18, 0.18, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14]

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 536 iterations"
    y-axis "llamacpp:requests_processing"
    x-axis "llamacpp:requests_processing" 1717000522 --> 1717001150
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 2.0, 2.0, 2.0, 2.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 1.0, 1.0, 1.0, 1.0, 1.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 1.0, 1.0, 1.0, 1.0, 1.0, 3.0, 3.0, 3.0, 3.0, 3.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 3.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 3.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 6.0, 6.0, 6.0, 6.0, 6.0]

ggml-ci

slaren · 2024-05-29T17:21:03Z

ggml-metal.m

+ ggml_is_contiguous_2(src0);
+ ggml_is_contiguous_2(src1);


Missing GGML_ASSERT?

Thanks - fixing

slaren · 2024-05-29T17:21:49Z

ggml-kompute.cpp

+ ggml_is_contiguous_2(src0);
+ ggml_is_contiguous_2(src1);


mofosyne added the Review Complexity : Medium Generally require more time to grok but manageable by beginner to medium expertise level label May 29, 2024

ggerganov force-pushed the gg/rope-tests branch from 13bd89d to 20d70c6 Compare May 29, 2024 11:57

github-actions bot added testing Everything test related Nvidia GPU Issues specific to Nvidia GPUs Vulkan Issues specific to the Vulkan backend python python script changes ggml changes relating to the ggml tensor library for machine learning labels May 29, 2024

ggerganov force-pushed the gg/rope-tests branch from 20d70c6 to 3b0df03 Compare May 29, 2024 12:09

This was referenced May 29, 2024

ggml : add rope tests + fix neox ggerganov/ggml#844

Closed

Add support for DeepseekV2ForCausalLM #7519

Merged

ggerganov added 3 commits May 29, 2024 17:18

tests : add rope tests

9d5605f

ggml-ci

ggml : fixes (hopefully)

b822605

ggml-ci

tests : add non-cont tests

1e41f2f

ggml-ci

ggerganov force-pushed the gg/rope-tests branch from 4fa3b23 to 036007e Compare May 29, 2024 14:46

cuda : add asserts for rope/norm + fix DS2

5db268c

ggml-ci

ggerganov force-pushed the gg/rope-tests branch from 036007e to 5db268c Compare May 29, 2024 14:46

ggerganov changed the title ~~tests : add rope tests~~ ggml : fix YARN + add tests + add asserts May 29, 2024

ggml : assert contiguousness

a69935b

github-actions bot added SYCL https://en.wikipedia.org/wiki/SYCL - GPU programming language Kompute https://github.com/KomputeProject/kompute/ labels May 29, 2024

tests : reduce RoPE tests

aa15e7d

ggml-ci

ggerganov marked this pull request as ready for review May 29, 2024 16:02

ggerganov merged commit fb76ec3 into master May 29, 2024
71 checks passed

ggerganov deleted the gg/rope-tests branch May 29, 2024 17:17

This was referenced May 29, 2024

YaRN : correction to GPT-NeoX implementation #4093

Closed

fix YaRN for NeoX type ggerganov/ggml#839

Closed

slaren reviewed May 29, 2024

View reviewed changes

ggerganov added a commit that referenced this pull request May 29, 2024

metal : add missing asserts (#7617)

975ec63

ggerganov added a commit that referenced this pull request May 29, 2024

metal : remove invalid asserts (#7617)

55d6226

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ggml : fix YARN + add tests + add asserts #7617

ggml : fix YARN + add tests + add asserts #7617

ggerganov commented May 29, 2024

github-actions bot commented May 29, 2024 •

edited

Loading

slaren May 29, 2024

ggerganov May 29, 2024

slaren May 29, 2024

ggml : fix YARN + add tests + add asserts #7617

ggml : fix YARN + add tests + add asserts #7617

Conversation

ggerganov commented May 29, 2024

github-actions bot commented May 29, 2024 • edited Loading

slaren May 29, 2024

Choose a reason for hiding this comment

ggerganov May 29, 2024

Choose a reason for hiding this comment

slaren May 29, 2024

Choose a reason for hiding this comment

github-actions bot commented May 29, 2024 •

edited

Loading