Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

merge inference into BertMLM_fix #3

Open
wants to merge 381 commits into
base: bert_fix1
Choose a base branch
from
Open

merge inference into BertMLM_fix #3

wants to merge 381 commits into from

Conversation

xinhaoc
Copy link
Owner

@xinhaoc xinhaoc commented May 17, 2024

Description of changes:

Related Issues:

Linked Issues:

  • Issue #

Issues closed by this PR:

  • Closes #

Before merging:

  • Did you update the flexflow-third-party repo, if modifying any of the Cmake files, the build configs, or the submodules?

jiazhihao and others added 30 commits May 10, 2023 17:57
@jiazhihao
Tree verify bug fix (flexflow#719)
0bf4fa9 
* Support multiple FFModels in a single top_level_task

* [TreeVerifyMHA] bug fixes
@xinhaoc
[Inference] opt model (flexflow#717)
b5b1375 
* init

* fix

* code

* clean up

* fix

* fix, add md

* format

* hip_roc

* add comment
@jiazhihao @xinhaoc
TreeIncMHA and SpecIncMHA bug fixes (flexflow#720)
b2d6d9a 
* Support multiple FFModels in a single top_level_task

* [TreeVerifyMHA] bug fixes

* bug fixes

* TreeIncMHA and SpecIncMHA bug fixes

* fomat.

---------

Co-authored-by: xinhaoc <[email protected]>
@xinhaoc
[inference] serving opt pipeline (flexflow#722)
f779d89 
* serving opt pipeline

* format
@goliaro @jiazhihao
add opt tokenizer functionality in C++ tokenizer (flexflow#727)
c08b1a0 
Co-authored-by: Zhihao Jia <[email protected]>
@xinhaoc @jiazhihao
code (flexflow#726)
ebb5f8e 
Co-authored-by: Zhihao Jia <[email protected]>
@goliaro
fix CI
86ec73a
@xinhaoc @jiazhihao
Kernel bug fix (flexflow#728)
8423610 
* complex into metadata

* topk

* format

---------

Co-authored-by: Zhihao Jia <[email protected]>
@goliaro
add decoder for gpt tokenizer
c9b2c5d
@jiazhihao @xinhaoc @goliaro
Spec infer demo (flexflow#724)
555aa33 
* Support multiple FFModels in a single top_level_task

* [TreeVerifyMHA] bug fixes

* bug fixes

* TreeIncMHA and SpecIncMHA bug fixes

* fomat.

* .

* add sentence piece tokenizer

* format

* prepare spec_infer demo

* prettier prints

* make the llama model work

* add small model config

* enable speculative inference for spec_infer

* fix

* rename

* fix one of the bugs

* fix

* del

* attempt to fix ci

* integrated gpt/opt tokenizer

* integrate opt tokenizer with pipeline

* .

* format

* move files

* Update README.md

* add an overview figure

* update images

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* add tokenizer in readme

* fix

* fix

* fix

* Update README.md

* Update README.md

* add gif

* add weights to readme, clean some print

* Update README.md

* update demo

* Update README.md

* Update README.md

* remove outdate file

* Update README.md

* Update README.md

* .

---------

Co-authored-by: xinhaoc <[email protected]>
Co-authored-by: Gabriele Oliaro <[email protected]>
Co-authored-by: xinhaoc <[email protected]>
@jiazhihao
Update README.md
16a5d02
@jiazhihao @xinhaoc @goliaro
Uses data and pipeline parallel by default. (flexflow#729)
b9fd233 
* Support multiple FFModels in a single top_level_task

* [TreeVerifyMHA] bug fixes

* bug fixes

* TreeIncMHA and SpecIncMHA bug fixes

* fomat.

* .

* add sentence piece tokenizer

* format

* prepare spec_infer demo

* prettier prints

* make the llama model work

* add small model config

* enable speculative inference for spec_infer

* fix

* rename

* fix one of the bugs

* fix

* del

* attempt to fix ci

* integrated gpt/opt tokenizer

* integrate opt tokenizer with pipeline

* .

* format

* move files

* Update README.md

* add an overview figure

* update images

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* add tokenizer in readme

* fix

* fix

* fix

* Update README.md

* Update README.md

* add gif

* add weights to readme, clean some print

* Update README.md

* update demo

* Update README.md

* Update README.md

* remove outdate file

* Update README.md

* Update README.md

* .

* use data parallel by default

---------

Co-authored-by: xinhaoc <[email protected]>
Co-authored-by: Gabriele Oliaro <[email protected]>
Co-authored-by: xinhaoc <[email protected]>
@jiazhihao
Update README.md
b8e5586
@goliaro
Merge branch 'master' into inference
07cb9f0
@goliaro
fix make build, edit cmake
0aabf34
@goliaro
update std version in makefile
427d602
@xinhaoc
file path adapt (flexflow#730)
d87197d 
* file path adapt

* fix

* fix

* fix
@goliaro
Update README.md
b9fddec
@goliaro
Update README.md
dc6dcf8
@goliaro
Update README.md
1193b51
@goliaro
[Inference][CI] - Fix GPU-CI and hip_rocmbuild tests (flexflow#731)
155989a 
* fix hip_rocm build with sentencepiece

* shellcheck 1

* shellcheck 2

* shellecheck 3

* fix install script

* .github/workflows/helpers/install_dependencies.sh

* fix

* shellcheck

* restore unnecessary changes

* fix build

* removed outdated test from c++ tests

* update link in readme
@goliaro
[Inference] - Cleanup/refactor (flexflow#732)
f0604b3 
* implemented file-based configs, remove spec_pipeline folder

* fix

* add inference test, script to downlaod weights

* update readme

* update ci scripts

* newlines

* fix gpu-ci

* fix

* fix

* update test file

* added incr decoding program, moved LLAMA folder from examples

* linting

* add incremental decoding to test

* update readme

* add script to download opt weights

* fix support for opt, move code to root inference folder

* linting

* update test file

* fix

* bug fix

* update test
@jiazhihao @xinhaoc
Merging duplicate functions in IncMHA, SpecIncMHA, and TreeIncNHA (fl…
28b31cd 
…exflow#736)

* making TreeIncMultiHeadSelfAttentionMeta a subclass of IncMultiHeadSelfAttentionMeta

* make BeamSearchIncMultiHeadAttentionMeta a subclass of IncMultiHeadAttentionMeta

* format

* merging kernel functions

* merge more functions

* merge compute_qkv_kernel

* format

* fix config

---------

Co-authored-by: xinhaoc <[email protected]>
@goliaro
[Inference] - Alignment fixes (flexflow#740)
b0a5b9c 
* fix alignment bugs (part 1)

* add missing file
@goliaro
Update README.md (flexflow#741)
1ab3d80
@jiazhihao @xinhaoc
Supporting mixed-precision (Spec/Tree/Normal) Incremental MultiHead A…
9f5bf94 
…ttention (flexflow#737)

* making TreeIncMultiHeadSelfAttentionMeta a subclass of IncMultiHeadSelfAttentionMeta

* make BeamSearchIncMultiHeadAttentionMeta a subclass of IncMultiHeadAttentionMeta

---------

Co-authored-by: xinhaoc <[email protected]>
@goliaro
[Inference] - Alignment tests (flexflow#742)
d7dd6bb 
* save output to file

* add alignment tests

* fix

* change conflicting name, add comments

* fix typo

* formatting

* more comments and clean dead code

* formatting

* fixed issue with length mismatch

* fix ci skip

* update inf test

* add precision selection support in incr decoding
@xinhaoc
Update README.md (flexflow#744)
6c13936 
* Update README.md

* update readme

* fix
@xinhaoc
fix
d8072ab
@goliaro
[Inference] - Add half precision & HuggingFace alignment tests + Spee…
9f2688d 
…d tests (flexflow#749)

* add support for downloading mixed precision llama/opt weights

* fix

* update test script to also run half precision tests

* disable workflow for inference PRs

* add verbose option

* linting

* copy opt weights in download weights script

* add alignment tests with huggingface (llama)

* fix, add diff to test script

* fix

* add opt tests

* comment out tests not passing

* add e2e latency to output files

* add speed tests

* shellcheck

* shellcheck

* fix

* fix

* linting

* fix
goliaro and others added 28 commits January 9, 2024 06:56
@goliaro
fix
ba4af39
@goliaro
Fuse bias and relu in OPT (flexflow#1265)
9c85a4f
@jiazhihao
fix spec decoding
197e308
@jiazhihao
Revert "fix spec decoding"
ed4dbd8 
This reverts commit 197e308.
@jiazhihao @goliaro
@zwang86 @xinhaoc
Add a background server for RequestManager (flexflow#1223)
12fdbac 
* add a background server for RequestManager

* .

* make incr_decoding work

* make spec_infer work

* format

* update python inference

* fix python issues

* bug fix

* add a Legion future to capture the termination of the background server

* Add thread safety for background server.

* Simplify backend server design.

* resolve conflict.

* Add server task timeout.

* register callbacks to terminate background worker at exit or termination

* [Python] enable decoding multiple requests

* update README.md and default configuration

* [Python] no need to use the llm context environment to start/stop the background server

* require at least four cpu cores

* [Python] add back explict start_server()/stop_server().

* fix

* fix python chatgpt.json

---------

Co-authored-by: Gabriele Oliaro <[email protected]>
Co-authored-by: zwang86 <[email protected]>
Co-authored-by: Zeyu Wang <[email protected]>
Co-authored-by: xinhaoc <[email protected]>
@jiazhihao
Update README.md
18cd485
@goliaro
Better debugging/logging tools for alignment checks (flexflow#1275)
75edadc 
* only stop server if rm is initialized

* fix

* better logging

* pass layer names to ops

* add debugging functionality to hf script

* fix

* fixes

* fix

* fix

---------

Co-authored-by: Ubuntu <[email protected]>
@FelixBrakel @goliaro
Fix incorrect innode being checked (flexflow#1273)
57d1883 
Co-authored-by: Gabriele Oliaro <[email protected]>
@jiazhihao @goliaro
Bug fixes and update Legion version (flexflow#1259)
317cffd 
* bug fixes and update Legion version

* fix

* bug fix

* update legion

* fix arithmetic error due to num_devices uninitialized

* update legion version

* update ci

* fix

* debugging ci

* Revert "debugging ci"

This reverts commit 0b3148e.

---------

Co-authored-by: Gabriele Oliaro <[email protected]>
@goliaro
Revert "Bug fixes and update Legion version" (flexflow#1286)
d73bba1
@april-yyt @jiazhihao
@goliaro @zwang86 @xinhaoc
Chatbot with Gradio, FastApi Endpoint, Langchain Integration (flexflo…
abf9fb8 
…w#1246)

* add a background server for RequestManager

* .

* make incr_decoding work

* make spec_infer work

* format

* update python inference

* fix python issues

* bug fix

* add a Legion future to capture the termination of the background server

* gradio finished

* chatbot gradio version 2

* chainlit1

* chainlit2

* fastapi done

* fastapi incr_decoding

* langchain example & wrapper class

* langchain example & wrapper class1

* added documentation

* entrypoint

* del apikey

* delete extra files

* rag search fixed some bugs

* fixed rag search issues

* updates before rebase

* minor changes

* reorganize files

* Add thread safety for background server.

* Simplify backend server design.

* resolve conflict.

* specinfer usecases with issues labeled

* specinfer usecases with issues labeled 2

* fixed issues with prompt template

* fix issues with rag specinfer

* Add server task timeout.

* register callbacks to terminate background worker at exit or termination

* [Python] enable decoding multiple requests

* update README.md and default configuration

* fix issues with gradio and prompt template

* fix issues with rag

* adjusted fastapi entrypoint

* update documentation

* resole conflicts

* issues fix

* adjustments on usecases and api entrypoints

* remove redundent changes

* testing CI

* Enable backtrace

* restore newlines

* version

* add back misdeleted line

* legion verion

---------

Co-authored-by: Zhihao Jia <[email protected]>
Co-authored-by: Gabriele Oliaro <[email protected]>
Co-authored-by: zwang86 <[email protected]>
Co-authored-by: Zeyu Wang <[email protected]>
Co-authored-by: xinhaoc <[email protected]>
@goliaro @jiazhihao
Bug fixes and update Legion version (flexflow#1287)
d21ed66 
* bug fixes and update Legion version

* fix

* bug fix

* update legion

* fix arithmetic error due to num_devices uninitialized

* update legion version

* update ci

* fix

* debugging ci

* Revert "debugging ci"

This reverts commit 0b3148e.

* update mapper interface

* add ncclFinalize

* Only delete nccl communications for training jobs

---------

Co-authored-by: Zhihao Jia <[email protected]>
@april-yyt @jiazhihao
Docs Modification for Python Usecases (flexflow#1291)
be28d71 
* modify README

* fix link issues

* update legion version

---------

Co-authored-by: Zhihao Jia <[email protected]>
@goliaro
Add support for docker machines with cuda 12.1 and cuda 12.2 (flexflo…
e24eb03 
…w#1308)
@goliaro
Fix NCCL tear down issue, update docker pre-build cuda version list (f…
0d75c10 
…lexflow#1318)
@goliaro
add expansion config param in specinfer
ea31426
@goliaro
parametrize max_spec_tree_token_num
e03dec0
@goliaro
fix
c856680
@goliaro
fix
8d82c91
@goliaro
fix
0479a64
@goliaro
run CI per commit only on inference branch
5bd7123
@goliaro
fix
e0a6e4f
@chenzhuofu
fix: 'model_configs' AttributeError (flexflow#1358)
1210256
@goliaro
Changes to support Perlmutter environment (flexflow#1360)
b4a639c 
* .

* remove deadcode

* add benchmarking mode, initializing weights randomly

* better logging when running out of memory

* update

---------

Co-authored-by: Gabriele Oliaro <[email protected]>
@goliaro
update workflow to build rocm docker images
7da197e
@goliaro
downgrade to python 3.11 for now
002fdf0
@chenzhuofu @goliaro
doc: fix c++ serving example (flexflow#1372)
d54e4b6 
Co-authored-by: Gabriele Oliaro <[email protected]>
@xinhaoc
Merge remote-tracking branch 'xinhao/xinhao_candle' into xinhao_infer…
024d188 
…ence
@xinhaoc xinhaoc changed the title Xinhao inference merge inference into BertMLM_fix May 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone